Skip to content

[SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel #1913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mengxr wants to merge 2 commits into from
Closed

[SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel #1913

mengxr wants to merge 2 commits into from

Conversation

@mengxr
Copy link
Contributor

@mengxr mengxr commented Aug 13, 2014

As mentioned in SPARK-2465, using MEMORY_AND_DISK_SER for user/product in/out links together with spark.rdd.compress=true can help reduce the space requirement by a lot, at the cost of speed. It might be useful to add this option so people can run ALS on much bigger datasets.

Another option for the method name is setIntermediateRDDStorageLevel.

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA tests have started for PR 1913. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18400/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA results for PR 1913:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18400/consoleFull

@brkyvz
Copy link
Contributor

brkyvz commented Aug 14, 2014

@mengxr: I would prefer setIntermediateRDDStorageLevel.

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA tests have started for PR 1913. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18522/consoleFull

@mengxr mengxr changed the title [SPARK-2995][MLLIB] add ALS.setIntermediateDataStorageLevel [SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel Aug 14, 2014
@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA results for PR 1913:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18522/consoleFull

asfgit pushed a commit that referenced this pull request Aug 14, 2014
@mengxr
[SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel
1baf06f 
As mentioned in SPARK-2465, using `MEMORY_AND_DISK_SER` for user/product in/out links together with `spark.rdd.compress=true` can help reduce the space requirement by a lot, at the cost of speed. It might be useful to add this option so people can run ALS on much bigger datasets.

Another option for the method name is `setIntermediateRDDStorageLevel`.

Author: Xiangrui Meng <[email protected]>

Closes #1913 from mengxr/als-storagelevel and squashes the following commits:

d942017 [Xiangrui Meng] rename to setIntermediateRDDStorageLevel
7550029 [Xiangrui Meng] add ALS.setIntermediateDataStorageLevel

(cherry picked from commit 69a57a1)
Signed-off-by: Xiangrui Meng <[email protected]>
@asfgit asfgit closed this in 69a57a1 Aug 14, 2014
@mengxr
Copy link
Contributor Author

mengxr commented Aug 14, 2014

Merged into both master and branch-1.1.

xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@mengxr @conviva-zz
[SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel
a3a226a 
As mentioned in SPARK-2465, using `MEMORY_AND_DISK_SER` for user/product in/out links together with `spark.rdd.compress=true` can help reduce the space requirement by a lot, at the cost of speed. It might be useful to add this option so people can run ALS on much bigger datasets.

Another option for the method name is `setIntermediateRDDStorageLevel`.

Author: Xiangrui Meng <[email protected]>

Closes apache#1913 from mengxr/als-storagelevel and squashes the following commits:

d942017 [Xiangrui Meng] rename to setIntermediateRDDStorageLevel
7550029 [Xiangrui Meng] add ALS.setIntermediateDataStorageLevel
snmvaughan pushed a commit to snmvaughan/spark that referenced this pull request Mar 26, 2024
@kazuyukitanimura
rdar://123462546 Update UC-Spark-Authz Plugin to 0.1.4 (apache#1913)
e688ccb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mengxr @SparkQA @brkyvz