Skip to content

[SPARK-25004][CORE] Add spark.executor.pyspark.memory limit. #21977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rdblue wants to merge 11 commits into from
Closed

[SPARK-25004][CORE] Add spark.executor.pyspark.memory limit. #21977

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a5004ba
SPARK-25004: Add spark.executor.pyspark.memory limit.
rdblue May 9, 2018
306538b
SPARK-25004: Fix problems from review.
rdblue Aug 3, 2018
9535a6b
SPARK-25004: Move configuration into PythonRunner.
rdblue Aug 3, 2018
5288f5b
SPARK-25004: Fix problems from review.
rdblue Aug 13, 2018
fbac4a5
SPARK-25004: Optionally capture stdout/stderr in YarnClusterSuite.
rdblue Aug 13, 2018
f11b3bb
SPARK-25004: Use SparkEnv instead of SparkSession.
rdblue Aug 15, 2018
ac7de4a
SPARK-25004: Fix remaining reference to conf on executors.
rdblue Aug 20, 2018
a38eac3
SPARK-25004: Remove renamining references to driver-side conf.
rdblue Aug 21, 2018
fcee94c
SPARK-25004: Access SparkEnv.get.conf in PythonRunner.
rdblue Aug 23, 2018
bb8fecb
SPARK-25004: Add docs for spark.executor.pyspark.memory setting.
rdblue Aug 24, 2018
0b275cf
SPARK-25004: Update configuration docs.
rdblue Aug 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SPARK-25004: Add docs for spark.executor.pyspark.memory setting.
  • Loading branch information
@rdblue
rdblue committed Aug 24, 2018
commit bb8fecb19e2a231061f95a327a162e48df48e9cd
9 changes: 9 additions & 0 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,15 @@ of the most common options to set are:
(e.g. <code>2g</code>, <code>8g</code>).
</td>
</tr>
<tr>
<td><code>spark.executor.pyspark.memory</code></td>
<td>Not set</td>
<td>
The amount of memory to be allocated to PySpark in each executor, in MiB
Copy link
Contributor

@holdenk holdenk Aug 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably mention that this is added to the executor memory request in Yarn mode.

Copy link
Contributor Author

@rdblue rdblue Aug 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added "When PySpark is run in YARN, this memory is added to executor resource requests."

unless otherwise specified. If set, PySpark memory for an executor will be
limited to this amount. If not set, Spark will not limit Python's memory use.
Copy link
Contributor

@holdenk holdenk Aug 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe mention that in this case (unset) it's up to the user to keep Python + system processes in the overhead %.

Copy link
Contributor Author

@rdblue rdblue Aug 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added "and it is up to the application to avoid exceeding the overhead memory space shared with other non-JVM processes."

</td>
</tr>
<tr>
<td><code>spark.executor.memoryOverhead</code></td>
<td>executorMemory * 0.10, with minimum of 384 </td>
Expand Down