Skip to content

[SPARK-1194] Fix the same-RDD rule for cache replacement #96

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liancheng wants to merge 4 commits into from
Closed

[SPARK-1194] Fix the same-RDD rule for cache replacement #96

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
62c92ac
Fixed SPARK-1194 https://spark-project.atlassian.net/browse/SPARK-1194
liancheng Mar 7, 2014
40cdcb2
Bug fix, and addressed PR comments from @mridulm
liancheng Mar 7, 2014
6e40c22
Remove redundant comments
liancheng Mar 7, 2014
2524ab9
Added regression test case for SPARK-1194
liancheng Mar 8, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove redundant comments
  • Loading branch information
@liancheng
liancheng committed Mar 7, 2014
commit 6e40c222f5b140ef3ba355ca3d655e19ee0039be
8 changes: 0 additions & 8 deletions core/src/main/scala/org/apache/spark/storage/MemoryStore.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,14 +236,6 @@ private class MemoryStore(blockManager: BlockManager, maxMemory: Long)
while (maxMemory - (currentMemory - selectedMemory) < space && iterator.hasNext) {
val pair = iterator.next()
val blockId = pair.getKey
// Apply the same-RDD rule for cache replacement. Quoted from the
// original RDD paper:
//
// When a new RDD partition is computed but there is not enough
// space to store it, we evict a partition from the least recently
// accessed RDD, unless this is the same RDD as the one with the
// new partition. In that case, we keep the old partition in memory
// to prevent cycling partitions from the same RDD in and out.
if (rddToAdd.isEmpty || rddToAdd != getRddId(blockId)) {
selectedBlocks += blockId
selectedMemory += pair.getValue.size
Expand Down