Skip to content

[SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel #1913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mengxr wants to merge 2 commits into from
Closed

[SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel #1913

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7550029
add ALS.setIntermediateDataStorageLevel
mengxr Aug 13, 2014
d942017
rename to setIntermediateRDDStorageLevel
mengxr Aug 14, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
rename to setIntermediateRDDStorageLevel
  • Loading branch information
@mengxr
mengxr committed Aug 14, 2014
commit d942017b36e3c5e7d829486b26eeabbfbf3e623c
10 changes: 5 additions & 5 deletions mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ class ALS private (
private var nonnegative = false

/** storage level for user/product in/out links */
private var intermediateDataStorageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK
private var intermediateRDDStorageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK

/**
* Set the number of blocks for both user blocks and product blocks to parallelize the computation
Expand Down Expand Up @@ -199,8 +199,8 @@ class ALS private (
* set `spark.rdd.compress` to `true` to reduce the space requirement, at the cost of speed.
*/
@DeveloperApi
def setIntermediateDataStorageLevel(storageLevel: StorageLevel): this.type = {
this.intermediateDataStorageLevel = storageLevel
def setIntermediateRDDStorageLevel(storageLevel: StorageLevel): this.type = {
this.intermediateRDDStorageLevel = storageLevel
this
}

Expand Down Expand Up @@ -456,8 +456,8 @@ class ALS private (
}, preservesPartitioning = true)
val inLinks = links.mapValues(_._1)
val outLinks = links.mapValues(_._2)
inLinks.persist(intermediateDataStorageLevel)
outLinks.persist(intermediateDataStorageLevel)
inLinks.persist(intermediateRDDStorageLevel)
outLinks.persist(intermediateRDDStorageLevel)
(inLinks, outLinks)
}

Expand Down