Skip to content

[SPARK-22428][DOC] Add spark application garbage collector configurat… #19826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gaborgsomogyi wants to merge 4 commits into from
Closed

[SPARK-22428][DOC] Add spark application garbage collector configurat… #19826

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cbd9d68
[SPARK-22428][DOC] Add spark application garbage collector configurat…
gaborgsomogyi Nov 27, 2017
bd6c2bb
Review fix
gaborgsomogyi Nov 28, 2017
d117c9f
Review fix
gaborgsomogyi Nov 29, 2017
241dbd0
Review fix
gaborgsomogyi Nov 30, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1132,6 +1132,46 @@ Apart from these, the following properties are also available, and may be useful
to get the replication level of the block to the initial number.
</td>
</tr>
<tr>
<td><code>spark.cleaner.periodicGC.interval</code></td>
<td>30min</td>
<td>
Controls how often to trigger a garbage collection.<br><br>
This context cleaner triggers cleanups only when weak references are garbage collected.
In long-running applications with large driver JVMs, where there is little memory pressure
on the driver, this may happen very occasionally or not at all. Not cleaning at all may
lead to executors running out of disk space after a while.
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking</code></td>
<td>true</td>
<td>
Enables or disables context cleaning.
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking.blocking</code></td>
<td>true</td>
<td>
Controls whether the cleaning thread should block on cleanup tasks (other than shuffle, which is controlled by
<code>spark.cleaner.referenceTracking.blocking.shuffle</code> Spark property).
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking.blocking.shuffle</code></td>
<td>false</td>
<td>
Controls whether the cleaning thread should block on shuffle cleanup tasks.
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking.cleanCheckpoints</code></td>
<td>false</td>
<td>
Controls whether to clean checkpoint files if the reference is out of scope.
</td>
</tr>
</table>

### Execution Behavior
Expand Down