-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-3167] Handle special driver configs in Windows #2129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
83ebe60
f97daa2
35caecc
eeb34a0
803218b
72004c2
afcffea
22b1acd
92e6047
881a8f0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -70,7 +70,7 @@ def preexec_func(): | |
| error_msg += "--------------------------------------------------------------\n" | ||
| raise Exception(error_msg) | ||
|
|
||
| # Ensure the Java child processes do not linger after python has exited in Windows. | ||
| # In Windows, ensure the Java child processes do not linger after Python has exited. | ||
| # In UNIX-based systems, the child process can kill itself on broken pipe (i.e. when | ||
| # the parent process' stdin sends an EOF). In Windows, however, this is not possible | ||
| # because java.lang.Process reads directly from the parent process' stdin, contending | ||
|
|
@@ -81,7 +81,7 @@ def preexec_func(): | |
| # (because the UNIX "exec" command is not available). This means we cannot simply | ||
| # call proc.kill(), which kills only the "spark-submit.cmd" process but not the | ||
| # JVMs. Instead, we use "taskkill" with the tree-kill option "/t" to terminate all | ||
| # child processes. | ||
| # child processes in the tree. | ||
| def killChild(): | ||
| Popen(["cmd", "/c", "taskkill", "/f", "/t", "/pid", str(proc.pid)]) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would link to wherever you found this - maybe here? Also, this is a bit scary to just issue kill commands (let's hope
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure. If we are to link it I'd rather provide a more official one, say http://technet.microsoft.com/en-us/library/bb491009.aspx |
||
| atexit.register(killChild) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there any public location where this behavior is specified (i.e. for a developer doc?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not that I'm aware of :/