Skip to content

[SPARK-25004][CORE] Add spark.executor.pyspark.memory limit. #21977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rdblue wants to merge 11 commits into from
Closed

[SPARK-25004][CORE] Add spark.executor.pyspark.memory limit. #21977

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a5004ba
SPARK-25004: Add spark.executor.pyspark.memory limit.
rdblue May 9, 2018
306538b
SPARK-25004: Fix problems from review.
rdblue Aug 3, 2018
9535a6b
SPARK-25004: Move configuration into PythonRunner.
rdblue Aug 3, 2018
5288f5b
SPARK-25004: Fix problems from review.
rdblue Aug 13, 2018
fbac4a5
SPARK-25004: Optionally capture stdout/stderr in YarnClusterSuite.
rdblue Aug 13, 2018
f11b3bb
SPARK-25004: Use SparkEnv instead of SparkSession.
rdblue Aug 15, 2018
ac7de4a
SPARK-25004: Fix remaining reference to conf on executors.
rdblue Aug 20, 2018
a38eac3
SPARK-25004: Remove renamining references to driver-side conf.
rdblue Aug 21, 2018
fcee94c
SPARK-25004: Access SparkEnv.get.conf in PythonRunner.
rdblue Aug 23, 2018
bb8fecb
SPARK-25004: Add docs for spark.executor.pyspark.memory setting.
rdblue Aug 24, 2018
0b275cf
SPARK-25004: Update configuration docs.
rdblue Aug 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SPARK-25004: Fix problems from review.
  • Loading branch information
@rdblue
rdblue committed Aug 23, 2018
commit 5288f5b1c5b5845f5bd6a2d62082176e3d4b0bda
17 changes: 9 additions & 8 deletions python/pyspark/worker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,17 +269,18 @@ def main(infile, outfile):
memory_limit_mb = int(os.environ.get('PYSPARK_EXECUTOR_MEMORY_MB', "-1"))
total_memory = resource.RLIMIT_AS
try:
(total_memory_limit, max_total_memory) = resource.getrlimit(total_memory)
msg = "Current mem: {0} of max {1}\n".format(total_memory_limit, max_total_memory)
print(msg, file=sys.stderr)
if memory_limit_mb > 0:
(soft_limit, hard_limit) = resource.getrlimit(total_memory)
msg = "Current mem limits: {0} of max {1}\n".format(soft_limit, hard_limit)
print(msg, file=sys.stderr)

if memory_limit_mb > 0 and total_memory_limit == resource.RLIM_INFINITY:
# convert to bytes
total_memory_limit = memory_limit_mb * 1024 * 1024
new_limit = memory_limit_mb * 1024 * 1024

msg = "Setting mem to {0} of max {1}\n".format(total_memory_limit, max_total_memory)
print(msg, file=sys.stderr)
resource.setrlimit(total_memory, (total_memory_limit, total_memory_limit))
if soft_limit == resource.RLIM_INFINITY or new_limit < soft_limit:
msg = "Setting mem limits to {0} of max {1}\n".format(new_limit, new_limit)
print(msg, file=sys.stderr)
resource.setrlimit(total_memory, (new_limit, new_limit))

except (resource.error, OSError, ValueError) as e:
# not all systems support resource limits, so warn instead of failing
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/python/AggregateInPandasExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ case class AggregateInPandasExec(
aggInputSchema,
sessionLocalTimeZone,
pythonRunnerConf,
sparkContext.conf).compute(projectedRowIter, context.partitionId(), context)
SparkEnv.get.conf).compute(projectedRowIter, context.partitionId(), context)

val joinedAttributes =
groupingExpressions.map(_.toAttribute) ++ udfExpressions.map(_.resultAttribute)
Expand Down