Skip to content

[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server #19047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pgandhi999 wants to merge 12 commits into from
Closed

[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server #19047

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
172fc20
[SPARK-21503]: Fixed the issue
Jul 21, 2017
81422e0
[SPARK-21503][UI]: Adding changes as per comments
Jul 25, 2017
55c6c37
Merge pull request #1 from apache/master
pgandhi999 Jul 26, 2017
f454c89
[SPARK-21541]: Spark Logs show incorrect job status for a job that do…
Jul 26, 2017
6b7d5c6
Revert "[SPARK-21541]: Spark Logs show incorrect job status for a job…
Jul 26, 2017
bc41664
Merge pull request #2 from apache/master
pgandhi999 Jul 28, 2017
e46126f
[SPARK-21503]- Making Changes as per comments
Jul 28, 2017
9b3cebc
Merge pull request #4 from apache/master
pgandhi999 Aug 1, 2017
7f03341
[SPARK-21503]: Reverting Unit Test Code
Aug 1, 2017
2d01cab
Merge pull request #5 from apache/master
pgandhi999 Aug 24, 2017
eaf63e6
[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config…
Aug 24, 2017
e421a03
[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config…
Aug 25, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config… 
… for launching daemons like History Server

Adding new env variable SPARK_DAEMON_CLASSPATH to set classpath for launching daemons. Tested and verified for History Server and Standalone Mode.
  • Loading branch information
pgandhi committed Aug 24, 2017
commit eaf63e6bd4dddc726cf57fda080b9b5d6341e2f8
3 changes: 2 additions & 1 deletion launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,8 @@ List<String> buildClassPath(String appClassPath) throws IOException {

Set<String> cp = new LinkedHashSet<>();
addToClassPath(cp, appClassPath);

addToClassPath(cp, getenv("SPARK_DAEMON_CLASSPATH"));
Copy link
Contributor

@jerryshao jerryshao Aug 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to only support this env variable for daemon process like HistoryServer, ExternalShuffleService and others, like what we did for SPARK_DAEMON_JAVA_OPTS. Currently with your fix normal Spark application will also honor this env variable.

Copy link
Author

@pgandhi999 pgandhi999 Aug 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you on this. I have made the necessary changes.


addToClassPath(cp, getConfDir());

boolean prependClasses = !isEmpty(getenv("SPARK_PREPEND_CLASSES"));
Expand Down