Skip to content

[SPARK-22428][DOC] Add spark application garbage collector configurat… #19826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gaborgsomogyi wants to merge 4 commits into from
Closed

[SPARK-22428][DOC] Add spark application garbage collector configurat… #19826

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cbd9d68
[SPARK-22428][DOC] Add spark application garbage collector configurat…
gaborgsomogyi Nov 27, 2017
bd6c2bb
Review fix
gaborgsomogyi Nov 28, 2017
d117c9f
Review fix
gaborgsomogyi Nov 29, 2017
241dbd0
Review fix
gaborgsomogyi Nov 30, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2306,6 +2306,50 @@ showDF(properties, numRows = 200, truncate = FALSE)
</tr>
</table>

### Spark Application Garbage Collector
Copy link
Member

@srowen srowen Nov 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pardon, I just noticed one more thing. This is really about the Context Cleaner, right? not garbage collection per se. I'd probably put these properties under Memory Management instead of a new section.

Copy link
Contributor Author

@gaborgsomogyi gaborgsomogyi Nov 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved as asked.


<table class="table">
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr>
<td><code>spark.cleaner.periodicGC.interval</code></td>
<td>30min</td>
<td>
Controls how often to trigger a garbage collection.
Copy link
Member

@srowen srowen Nov 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might give at least a sentence of explanation about what this has to do with the context cleaner.

Copy link
Contributor Author

@gaborgsomogyi gaborgsomogyi Nov 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given more detailed description.

</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking</code></td>
<td>true</td>
<td>
Controls whether a ContextCleaner should be created when a SparkContext initializes.
Copy link
Member

@srowen srowen Nov 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just say it controls whether context cleaning is enabled at all. This sort of suggests it controls whether it's created at a certain time as opposed to another time.

Copy link
Contributor Author

@gaborgsomogyi gaborgsomogyi Nov 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rephrased.

</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking.blocking</code></td>
<td>true</td>
<td>
Controls whether the cleaning thread should block on cleanup tasks (other than shuffle, which is controlled by
spark.cleaner.referenceTracking.blocking.shuffle Spark property).<br><br>
Copy link
Member

@srowen srowen Nov 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might surround prop names in <code> for extra clarity

Copy link
Contributor Author

@gaborgsomogyi gaborgsomogyi Nov 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surrounded.

It is true as a workaround to <a href="https://issues.apache.org/jira/browse/SPARK-3015">
Copy link
Member

@srowen srowen Nov 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can remove comments like this, or leave them to the source code. This doesn't directly help the reader understand when or if to change it. Or, add context here about why one would set this.

Copy link
Contributor Author

@gaborgsomogyi gaborgsomogyi Nov 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

SPARK-3015 Removing broadcast in quick successions causes Akka timeout</a>.
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking.blocking.shuffle</code></td>
<td>false</td>
<td>
Controls whether the cleaning thread should block on shuffle cleanup tasks.<br><br>
It is false as a workaround to SPARK-3139 Akka timeouts from ContextCleaner when cleaning shuffles.
</td>
</tr>
<tr>
<td><code>spark.cleaner.referenceTracking.cleanCheckpoints</code></td>
<td>false</td>
<td>
Controls whether to clean checkpoint files if the reference is out of scope.
</td>
</tr>
</table>

Copy link
Member

@HyukjinKwon HyukjinKwon Nov 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I am insane but let's revert unneeded change

Copy link
Contributor Author

@gaborgsomogyi gaborgsomogyi Nov 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted.

### Cluster Managers

Expand Down