[SPARK-22428][DOC] Add spark application garbage collector configurat… #19826
Conversation
|
Test build #3994 has finished for PR 19826 at commit
|
docs/configuration.md
Outdated
| <td> | ||
| Controls whether the cleaning thread should block on cleanup tasks (other than shuffle, which is controlled by | ||
| spark.cleaner.referenceTracking.blocking.shuffle Spark property).<br><br> | ||
| It is true as a workaround to <a href="https://issues.apache.org/jira/browse/SPARK-3015"> |
There was a problem hiding this comment.
I think you can remove comments like this, or leave them to the source code. This doesn't directly help the reader understand when or if to change it. Or, add context here about why one would set this.
docs/configuration.md
Outdated
| <td>true</td> | ||
| <td> | ||
| Controls whether the cleaning thread should block on cleanup tasks (other than shuffle, which is controlled by | ||
| spark.cleaner.referenceTracking.blocking.shuffle Spark property).<br><br> |
There was a problem hiding this comment.
You might surround prop names in <code> for extra clarity
docs/configuration.md
Outdated
| <td><code>spark.cleaner.periodicGC.interval</code></td> | ||
| <td>30min</td> | ||
| <td> | ||
| Controls how often to trigger a garbage collection. |
There was a problem hiding this comment.
You might give at least a sentence of explanation about what this has to do with the context cleaner.
There was a problem hiding this comment.
Given more detailed description.
docs/configuration.md
Outdated
| <td><code>spark.cleaner.referenceTracking</code></td> | ||
| <td>true</td> | ||
| <td> | ||
| Controls whether a ContextCleaner should be created when a SparkContext initializes. |
There was a problem hiding this comment.
I'd just say it controls whether context cleaning is enabled at all. This sort of suggests it controls whether it's created at a certain time as opposed to another time.
|
Test build #3995 has finished for PR 19826 at commit
|
docs/configuration.md
Outdated
| </tr> | ||
| </table> | ||
|
|
||
| ### Spark Application Garbage Collector |
There was a problem hiding this comment.
Pardon, I just noticed one more thing. This is really about the Context Cleaner, right? not garbage collection per se. I'd probably put these properties under Memory Management instead of a new section.
There was a problem hiding this comment.
Moved as asked.
|
retest this please |
| </tr> | ||
| </table> | ||
|
|
||
|
|
There was a problem hiding this comment.
Maybe I am insane but let's revert unneeded change
|
Test build #84343 has finished for PR 19826 at commit
|
|
retest this please |
|
Test build #84358 has finished for PR 19826 at commit
|
|
Merged to master |
4 participants
What changes were proposed in this pull request?
The spark properties for configuring the ContextCleaner are not documented in the official documentation at https://spark.apache.org/docs/latest/configuration.html#available-properties.
This PR adds the doc.
How was this patch tested?
Manual.