Skip to content

[SPARK-22428][DOC] Add spark application garbage collector configurat… #19826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gaborgsomogyi wants to merge 4 commits into from
Closed

[SPARK-22428][DOC] Add spark application garbage collector configurat… #19826

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cbd9d68
[SPARK-22428][DOC] Add spark application garbage collector configurat…
gaborgsomogyi Nov 27, 2017
bd6c2bb
Review fix
gaborgsomogyi Nov 28, 2017
d117c9f
Review fix
gaborgsomogyi Nov 29, 2017
241dbd0
Review fix
gaborgsomogyi Nov 30, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2306,6 +2306,51 @@ showDF(properties, numRows = 200, truncate = FALSE)
</tr>
</table>

### Spark Application Garbage Collector
Copy link
Member

@srowen srowen Nov 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pardon, I just noticed one more thing. This is really about the Context Cleaner, right? not garbage collection per se. I'd probably put these properties under Memory Management instead of a new section.

Copy link
Contributor Author

@gaborgsomogyi gaborgsomogyi Nov 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved as asked.


<table class="table">
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr>
<td><code>spark.cleaner.periodicGC.interval</code></td>
<td>30min</td>
<td>
Controls how often to trigger a garbage collection.<br><br>
This context cleaner triggers cleanups only when weak references are garbage collected.
In long-running applications with large driver JVMs, where there is little memory pressure
on the driver, this may happen very occasionally or not at all. Not cleaning at all may
lead to executors running out of disk space after a while.
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking</code></td>
<td>true</td>
<td>
Enables or disables context cleaning.
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking.blocking</code></td>
<td>true</td>
<td>
Controls whether the cleaning thread should block on cleanup tasks (other than shuffle, which is controlled by
<code>spark.cleaner.referenceTracking.blocking.shuffle</code> Spark property).
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking.blocking.shuffle</code></td>
<td>false</td>
<td>
Controls whether the cleaning thread should block on shuffle cleanup tasks.
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking.cleanCheckpoints</code></td>
<td>false</td>
<td>
Controls whether to clean checkpoint files if the reference is out of scope.
</td>
</tr>
</table>

Copy link
Member

@HyukjinKwon HyukjinKwon Nov 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I am insane but let's revert unneeded change

Copy link
Contributor Author

@gaborgsomogyi gaborgsomogyi Nov 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted.

### Cluster Managers

Expand Down