Skip to content

[WIP] SPARK-1430: Support sparse data in Python MLlib #341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mateiz wants to merge 18 commits into from
Closed

[WIP] SPARK-1430: Support sparse data in Python MLlib #341

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
881fef7
Added a sparse vector in Python and made Java-Python format more compact
mateiz Apr 6, 2014
154f45d
Update docs, name some magic values
mateiz Apr 6, 2014
2abbb44
Further work to get linear models working with sparse data
mateiz Apr 7, 2014
eaee759
Update regression, classification and clustering models for sparse data
mateiz Apr 7, 2014
0e7a3d8
Keep vectors sparse in Java when reading LabeledPoints
mateiz Apr 7, 2014
a5d6426
Add linalg.py to run-tests script
mateiz Apr 7, 2014
ab244d1
Allow SparseVectors to be initialized using a dict
mateiz Apr 7, 2014
889dde8
Support scipy.sparse matrices in all our algorithms and models
mateiz Apr 10, 2014
74eefe7
Added LabeledPoint class in Python
mateiz Apr 11, 2014
a07ba10
Fix some typos and calculation of initial weights
mateiz Apr 11, 2014
c48e85a
Added some tests for passing lists as input, and added mllib/tests.py to
mateiz Apr 13, 2014
da0f27e
Added a MLlib K-means example and updated docs to discuss sparse data
mateiz Apr 13, 2014
37ab747
Fix some examples and docs due to changes in MLlib API
mateiz Apr 13, 2014
88bc01f
Clean up inheritance of LinearModel in Python, and expose its parametrs
mateiz Apr 13, 2014
1e1bd0f
Add MLlib logistic regression example in Python
mateiz Apr 13, 2014
b9f97a3
Fix test
mateiz Apr 13, 2014
ea5a25a
Fix remaining uses of copyto() after merge
mateiz Apr 15, 2014
d52e763
Remove no-longer-needed slice code and handle review comments
mateiz Apr 15, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add linalg.py to run-tests script
  • Loading branch information
@mateiz
mateiz committed Apr 15, 2014
commit a5d64261a8785f73b2cceec5740f99de350ffc39
3 changes: 2 additions & 1 deletion python/run-tests
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ rm -rf metastore warehouse
function run_test() {
SPARK_TESTING=0 $FWDIR/bin/pyspark $1 2>&1 | tee -a > unit-tests.log
FAILED=$((PIPESTATUS[0]||$FAILED))

# Fail and exit on the first test failure.
if [[ $FAILED != 0 ]]; then
cat unit-tests.log | grep -v "^[0-9][0-9]*" # filter all lines starting with a number.
Expand All @@ -57,6 +57,7 @@ run_test "pyspark/tests.py"
run_test "pyspark/mllib/_common.py"
run_test "pyspark/mllib/classification.py"
run_test "pyspark/mllib/clustering.py"
run_test "pyspark/mllib/linalg.py"
run_test "pyspark/mllib/recommendation.py"
run_test "pyspark/mllib/regression.py"

Expand Down