Skip to content

[SPARK-21600][docs] The description of "this requires spark.shuffle.service.enabled to be set" for the spark.dynamicAllocation.enabled configuration item is not clear #18806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
guoxiaolongzte wants to merge 1 commit into from
Closed

[SPARK-21600][docs] The description of "this requires spark.shuffle.service.enabled to be set" for the spark.dynamicAllocation.enabled configuration item is not clear #18806

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
[SPARK-21600] The description of "this requires spark.shuffle.service… 
….enabled to be set" for the spark.dynamicAllocation.enabled configuration item is not clear. I am not sure how to set spark.shuffle.service.enabled is true or false, so that the user to guess, resulting in doubts. All i have changed here, stressed that must spark.shuffle.service.enabled to be set true.
  • Loading branch information
guoxiaolong committed Aug 2, 2017
commit 4c8de9a46ea4a09334eaa31328cb9eea1c2eb6b1
2 changes: 1 addition & 1 deletion docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1638,7 +1638,7 @@ Apart from these, the following properties are also available, and may be useful
For more detail, see the description
<a href="job-scheduling.html#dynamic-resource-allocation">here</a>.
<br><br>
This requires <code>spark.shuffle.service.enabled</code> to be set.
This requires <code>spark.shuffle.service.enabled</code> to be set true.
Copy link
Contributor

@jerryshao jerryshao Aug 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's no ambiguity here. Usually configuration with name "xxx.enabled" can only have two values "true" or "false". So "to be set" usually means to enable it (to set it to true).

Copy link
Author

@guoxiaolongzte guoxiaolongzte Aug 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, but usually is not very sure. Can not let the user to guess, the document needs to be accurately described. In addition, the spark project, several other are clearly described, spark.shuffle.service.enabled set true.
1
2

Copy link
Contributor

@jerryshao jerryshao Aug 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since other places are clearly defined the property, so there should be no ambiguity. Personally I'm not fond of this super nit fix...

Copy link
Author

@guoxiaolongzte guoxiaolongzte Aug 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your comments.
This requires spark.shuffle.service.enabled to be set true. It is very clearly. Only such an accurate description，there be no ambiguity.

The following configurations are also relevant:
<code>spark.dynamicAllocation.minExecutors</code>,
<code>spark.dynamicAllocation.maxExecutors</code>, and
Expand Down