Skip to content

SPARK-10611 Clone Configuration for each task for NewHadoopRDD #8763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mingyukim wants to merge 3 commits into from
Closed

SPARK-10611 Clone Configuration for each task for NewHadoopRDD #8763

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d516aae
SPARK-10611 Clone Configuration for each task for NewHadoopRDD
mingyukim Sep 15, 2015
6460266
Make getConf public
mingyukim Sep 15, 2015
4ba2acd
Fixing style issues
mingyukim Sep 16, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Fixing style issues
  • Loading branch information
@mingyukim
mingyukim committed Sep 16, 2015
commit 4ba2acd09e41aec97f049f57b68785cf419347f6
6 changes: 3 additions & 3 deletions core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,9 @@ class NewHadoopRDD[K, V](
// one job modifies a configuration while another reads it (SPARK-2546, SPARK-10611). This
// problem occurs somewhat rarely because most jobs treat the configuration as though it's
// immutable. One solution, implemented here, is to clone the Configuration object.
// Unfortunately, this clone can be very expensive. To avoid unexpected performance regressions
// for workloads and Hadoop versions that do not suffer from these thread-safety issues, this
// cloning is disabled by default.
// Unfortunately, this clone can be very expensive. To avoid unexpected performance
// regressions for workloads and Hadoop versions that do not suffer from these thread-safety
// issues, this cloning is disabled by default.
NewHadoopRDD.CONFIGURATION_INSTANTIATION_LOCK.synchronized {
logDebug("Cloning Hadoop Configuration")
new Configuration(conf)
Expand Down