Skip to content

[SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode #28258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
akshatb1 wants to merge 15 commits into from
Closed

[SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode #28258

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
13ea149
Implementation of spark.submit.waitForCompletion flag which will cont…
Apr 19, 2020
68d76d0
Fixing to adher to JavaDoc and Scalastyle guidelines
akshatb1 Apr 19, 2020
34c7d26
Using conf.getBoolean for the config
akshatb1 Apr 20, 2020
a93ce76
Using Match Statement and updating config property location in docume…
akshatb1 May 1, 2020
d5eded1
Addressing documentation related review comments
akshatb1 May 2, 2020
8eef373
Fixing indentation
akshatb1 May 2, 2020
0918106
Implementing the flag using asyn message
akshatb1 May 10, 2020
e225495
Implementing the flag using async DriverStatusResponse message
akshatb1 May 10, 2020
20f1bd6
Merge branch 'master' of https://github.com/akshatb1/spark
akshatb1 May 10, 2020
9050a08
Using Thread Scheduler to Monitor Driver Status
akshatb1 May 16, 2020
45c9817
Refactoring to avoid multiple calls of pollAndReportStatus method
akshatb1 May 22, 2020
743d93d
Fixing indentation
akshatb1 Jun 2, 2020
fe142a8
Fixing Indentation for method parameters
akshatb1 Jun 2, 2020
27a81c9
Merging master
akshatb1 Jun 2, 2020
0e152f4
Merging master
akshatb1 Jun 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Using Match Statement and updating config property location in docume… 
…ntation
  • Loading branch information
@akshatb1
akshatb1 committed May 1, 2020
commit a93ce76d8205da6eb01ab41644c57504174d405d
24 changes: 13 additions & 11 deletions core/src/main/scala/org/apache/spark/deploy/Client.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@ private class ClientEndpoint(

private val lostMasters = new HashSet[RpcAddress]
private var activeMasterEndpoint: RpcEndpointRef = null
private val waitAppCompletion = conf.getBoolean("spark.submit.waitAppCompletion", false)
private val waitAppCompletion = conf.getBoolean("spark.standalone.submit.waitAppCompletion",
false)
private val REPORT_DRIVER_STATUS_INTERVAL = 1000


Expand Down Expand Up @@ -156,19 +157,20 @@ private class ClientEndpoint(
System.exit(-1)
case _ =>
if (!waitAppCompletion) {
logInfo(s"No exception found and waitAppCompletion is false, " +
s"exiting spark-submit JVM.")
System.exit(0)
} else if (statusResponse.state.get == DriverState.FINISHED ||
statusResponse.state.get == DriverState.FAILED ||
statusResponse.state.get == DriverState.ERROR ||
statusResponse.state.get == DriverState.KILLED) {
logInfo(s"waitAppCompletion is true, state is ${statusResponse.state.get}, " +
logInfo(s"spark-submit not configured to wait for completion, " +
s"exiting spark-submit JVM.")
System.exit(0)
} else {
logTrace(s"waitAppCompletion is true, state is ${statusResponse.state.get}," +
s"continue monitoring driver status.")
statusResponse.state.get match {
case DriverState.FINISHED | DriverState.FAILED |
DriverState.ERROR | DriverState.KILLED =>
logInfo(s"State of $driverId is ${statusResponse.state.get}, " +
s"exiting spark-submit JVM.")
System.exit(0)
case _ =>
logTrace(s"State of $driverId is ${statusResponse.state.get}," +
s"continue monitoring driver status.")
}
}
}
} else {
Expand Down
28 changes: 18 additions & 10 deletions docs/spark-standalone.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,16 +240,6 @@ SPARK_MASTER_OPTS supports the following system properties:
</td>
<td>1.6.3</td>
</tr>
<tr>
<td><code>spark.submit.waitAppCompletion</code></td>
<td><code>true</code></td>
<td>
In Standalone cluster mode, controls whether the client waits to exit until the application completes.
If set to <code>true</code>, the client process will stay alive reporting the application's status.
Otherwise, the client process will exit after submission.
</td>
<td>3.1.0</td>
</tr>
<tr>
<td><code>spark.worker.timeout</code></td>
<td>60</td>
Expand Down Expand Up @@ -384,6 +374,24 @@ To run an interactive Spark shell against the cluster, run the following command

You can also pass an option `--total-executor-cores <numCores>` to control the number of cores that spark-shell uses on the cluster.

#Spark Properties

Spark applications supports the following configuration properties specific to standalone Mode:
<table class="table">
<tr><th style="width:21%">Property Name</th><th>Default Value</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.standalone.submit.waitAppCompletion</code></td>
<td><code>false</code></td>
<td>
In Standalone cluster mode, controls whether the client waits to exit until the application completes.
Copy link
Member

@srowen srowen May 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Standalone -> standalone
reporting -> polling

Copy link
Contributor Author

@akshatb1 akshatb1 May 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in the latest commit.

If set to <code>true</code>, the client process will stay alive reporting the application's status.
Otherwise, the client process will exit after submission.
</td>
<td>3.1.0</td>
</tr>
</table>


# Launching Spark Applications

The [`spark-submit` script](submitting-applications.html) provides the most straightforward way to
Expand Down