-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-1103] [WIP] Automatic garbage collection of RDD, shuffle and broadcast data #126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
1e752f1
80dd977
e427a9e
8512612
a24fefc
cb0a5a6
ae9da88
e61daa0
a7260d3
892b952
e1fba5f
f2881fd
620eca3
a007307
d2f8b97
6c9dcf6
c7ccef1
ba52e00
d0edef3
544ac86
e95479c
f201a8d
c92e4d9
0d17060
34f436f
fbfeec8
88904a3
e442246
8557c12
7edbc98
634a097
7ed72fb
5016375
f0aabb1
762a4d8
a6460d4
c5b1d98
a2cc8bc
ada45f0
cd72d19
b27f8e8
a430f06
104a89a
6222697
41c9ece
2b95b5e
4d05314
cff023c
d25a86e
f489fdc
61b8d6e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -63,8 +63,11 @@ private[spark] class ContextCleaner(sc: SparkContext) extends Logging { | |
|
|
||
| private val cleaningThread = new Thread() { override def run() { keepCleaning() }} | ||
|
|
||
| /** Whether the cleaning thread will block on cleanup tasks */ | ||
| private val blockOnCleanupTasks = sc.conf.getBoolean("spark.cleaner.referenceTracking.blocking", false) | ||
| /** | ||
| * Whether the cleaning thread will block on cleanup tasks. | ||
| * This is set to true only for tests. */ | ||
| private val blockOnCleanupTasks = sc.conf.getBoolean( | ||
| "spark.cleaner.referenceTracking.blocking", false) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why is this a SparkConf instead of just having a utility function called
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That would require a "var", and I generally dont like having a "var" in a class field. |
||
|
|
||
| @volatile private var stopped = false | ||
|
|
||
|
|
@@ -170,7 +173,8 @@ private[spark] class ContextCleaner(sc: SparkContext) extends Logging { | |
| private def broadcastManager = sc.env.broadcastManager | ||
| private def mapOutputTrackerMaster = sc.env.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster] | ||
|
|
||
| // Used for testing, explicitly blocks until cleanup is completed | ||
| // Used for testing. These methods explicitly blocks until cleanup is completed | ||
| // to ensure that more reliable testing. | ||
|
|
||
| def cleanupRDD(rdd: RDD[_]) { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't get why these need to be exposed (each of these is only called once). Why not just call
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed. Dont really need private[spark] as the whole class is private[spark]. So just making them public is sufficient.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Will push with all updates. |
||
| doCleanupRDD(rdd.id, blocking = true) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This closing comment should be bumped down to the next line.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed. Will push with all changes.