Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Oct 14, 2019

What changes were proposed in this pull request?

This PR aims to add a new column Duration for running drivers in Apache Spark Standalone master web UI in order to improve UX. This help users like the other Duration columns in the Running and Completed application tables.

Why are the changes needed?

When we use --supervise, the drivers can survive longer.
Technically, the Duration column is not the same. (Please see the image below.)

Does this PR introduce any user-facing change?

Yes. The red box is added newly.

Screen Shot 2019-10-14 at 12 53 43 PM

How was this patch tested?

Manual since this is a UI column. After starting standalone cluster and jobs, kill the DriverWrapper and see the UI.

$ sbin/start-master.sh
$ sbin/start-slave.sh spark://$(hostname):7077
$ bin/spark-submit --master spark://(hostname):7077 --deploy-mode cluster --supervise --class org.apache.spark.examples.JavaSparkPi examples/target/scala-2.12/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar 1000
$ jps
41521 DriverWrapper
...
$ kill -9 41521   // kill the `DriverWrapper`.

@SparkQA
Copy link

SparkQA commented Oct 14, 2019

Test build #112056 has finished for PR 26113 at commit fab5ec4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

Retest this please

@SparkQA
Copy link

SparkQA commented Oct 16, 2019

Test build #112176 has finished for PR 26113 at commit fab5ec4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dbtsai
Copy link
Member

dbtsai commented Oct 18, 2019

LGTM. Thanks.

@dongjoon-hyun
Copy link
Member Author

Thank you for review and approval, @dbtsai . Merged to master!

@dongjoon-hyun dongjoon-hyun deleted the SPARK-29466 branch October 18, 2019 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants