Skip to content

[SPARK-25004][CORE] Add spark.executor.pyspark.memory limit. #21977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rdblue wants to merge 11 commits into from
Closed

[SPARK-25004][CORE] Add spark.executor.pyspark.memory limit. #21977

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a5004ba
SPARK-25004: Add spark.executor.pyspark.memory limit.
rdblue May 9, 2018
306538b
SPARK-25004: Fix problems from review.
rdblue Aug 3, 2018
9535a6b
SPARK-25004: Move configuration into PythonRunner.
rdblue Aug 3, 2018
5288f5b
SPARK-25004: Fix problems from review.
rdblue Aug 13, 2018
fbac4a5
SPARK-25004: Optionally capture stdout/stderr in YarnClusterSuite.
rdblue Aug 13, 2018
f11b3bb
SPARK-25004: Use SparkEnv instead of SparkSession.
rdblue Aug 15, 2018
ac7de4a
SPARK-25004: Fix remaining reference to conf on executors.
rdblue Aug 20, 2018
a38eac3
SPARK-25004: Remove renamining references to driver-side conf.
rdblue Aug 21, 2018
fcee94c
SPARK-25004: Access SparkEnv.get.conf in PythonRunner.
rdblue Aug 23, 2018
bb8fecb
SPARK-25004: Add docs for spark.executor.pyspark.memory setting.
rdblue Aug 24, 2018
0b275cf
SPARK-25004: Update configuration docs.
rdblue Aug 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SPARK-25004: Use SparkEnv instead of SparkSession.
  • Loading branch information
@rdblue
rdblue committed Aug 23, 2018
commit f11b3bbc3e39c250835b75fa9fa59e46282ab288
4 changes: 2 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/python/BatchEvalPythonExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import scala.collection.JavaConverters._

import net.razorvine.pickle.{Pickler, Unpickler}

import org.apache.spark.TaskContext
import org.apache.spark.{SparkEnv, TaskContext}
import org.apache.spark.api.python.{ChainedPythonFunctions, PythonEvalType}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
Expand Down Expand Up @@ -67,7 +67,7 @@ case class BatchEvalPythonExec(udfs: Seq[PythonUDF], output: Seq[Attribute], chi

// Output iterator for results from Python.
val outputIterator = new PythonUDFRunner(
funcs, PythonEvalType.SQL_BATCHED_UDF, argOffsets, sparkContext.conf)
funcs, PythonEvalType.SQL_BATCHED_UDF, argOffsets, SparkEnv.get.conf)
.compute(inputIterator, context.partitionId(), context)

val unpickle = new Unpickler
Expand Down