SPARK-1432: Make sure that all metadata fields are properly cleaned #338
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
Contributor
There was a problem hiding this comment.
Is there a reason why you removed this check? If you look where stageIdToDescription is populated, it may or may contain an an entry for the stage.
Contributor
There was a problem hiding this comment.
Ah I see - this is because we're using the Scala HashMap?
scala> import scala.collection.mutable.HashMap
import scala.collection.mutable.HashMap
scala> val x = new HashMap[String, String]()
x: scala.collection.mutable.HashMap[String,String] = Map()
scala> x.remove("nonexistent")
res0: Option[String] = None
Contributor
Author
There was a problem hiding this comment.
Yup! No need for that check. :)
Contributor
|
Okay this looks good to me. Do you mind creating a new JIRA for this and setting the title to something like: "stageIdToExecutorSummaries can lead to memory leak". And could you set the title of this to reflect the JIRA number? This is a bit of extra overhead, but may run into this leak and it would be good to have a reference for it. |
Contributor
Author
dgshep
changed the title
Contributor
|
Thanks - merged this into master and 0.9 branch. |
asfgit
pushed a commit
that referenced
this pull request
Apr 7, 2014
While working on spark-1337 with @pwendell, we noticed that not all of the metadata maps in JobProgessListener were being properly cleaned. This could lead to a (hypothetical) memory leak issue should a job run long enough. This patch aims to address the issue. Author: Davis Shepherd <[email protected]> Closes #338 from dgshep/master and squashes the following commits: a77b65c [Davis Shepherd] In the contex of SPARK-1337: Make sure that all metadata fields are properly cleaned (cherry picked from commit a3c51c6) Conflicts: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
andrewor14
pushed a commit
to andrewor14/spark
that referenced
this pull request
Apr 7, 2014
SPARK-1005 Ning upgrade
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
While working on spark-1337 with @pwendell, we noticed that not all of the metadata maps in JobProgessListener were being properly cleaned. This could lead to a (hypothetical) memory leak issue should a job run long enough. This patch aims to address the issue. Author: Davis Shepherd <[email protected]> Closes apache#338 from dgshep/master and squashes the following commits: a77b65c [Davis Shepherd] In the contex of SPARK-1337: Make sure that all metadata fields are properly cleaned
mccheah
pushed a commit
to mccheah/spark
that referenced
this pull request
Oct 3, 2018
bzhaoopenstack
pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
Run cluster-api-provider-openstack-image-build in sigs.k8s.io
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While working on spark-1337 with @pwendell, we noticed that not all of the metadata maps in JobProgessListener were being properly cleaned. This could lead to a (hypothetical) memory leak issue should a job run long enough. This patch aims to address the issue.