Skip to content

[Spark-1538] Fix SparkUI incorrectly hiding persisted RDDs #469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
andrewor14 wants to merge 6 commits into from
Closed

[Spark-1538] Fix SparkUI incorrectly hiding persisted RDDs #469

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bfe83f0
Backtrace RDD dependency tree to find all RDDs that belong to a Stage
andrewor14 Apr 21, 2014
2acb177
Move getNarrowAncestors to RDD.scala
andrewor14 Apr 21, 2014
d2bac8a
Deal with cycles in RDD dependency graph + add extensive tests
andrewor14 Apr 22, 2014
9d0e2b8
Hide details of getNarrowAncestors from outsiders
andrewor14 Apr 23, 2014
5d799fe
Add comment to justify testing of getNarrowAncestors with cycles
andrewor14 Apr 23, 2014
07fc7f0
Add back comment that was accidentally removed (minor)
andrewor14 Apr 23, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add comment to justify testing of getNarrowAncestors with cycles
  • Loading branch information
@andrewor14
andrewor14 committed Apr 23, 2014
commit 5d799fedbe7236095de49a780f1c6c98ce2de8ec
6 changes: 6 additions & 0 deletions core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -638,6 +638,12 @@ class RDDSuite extends FunSuite with SharedSparkContext {
assert(ancestors9.count(_.isInstanceOf[MappedValuesRDD[_, _, _]]) === 1)
}

/**
* This tests for the pathological condition in which the RDD dependency graph is cyclical.
*
* Since RDD is part of the public API, applications may actually implement RDDs that allow
* such graphs to be constructed. In such cases, getNarrowAncestor should not simply hang.
*/
test("getNarrowAncestors with cycles") {
val rdd1 = new CyclicalDependencyRDD[Int]
val rdd2 = new CyclicalDependencyRDD[Int]
Expand Down