Skip to content

[SPARK-9750][MLlib] Improve equals on SparseMatrix and DenseMatrix #8042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
feynmanliang wants to merge 6 commits into from
Closed

[SPARK-9750][MLlib] Improve equals on SparseMatrix and DenseMatrix #8042

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6416fa0
Add failing sparse matrix equals tests
Aug 7, 2015
43d28fa
Fix failing test
Aug 7, 2015
78f9426
Add casts
Aug 7, 2015
22782df
Add equality based on matrix semantics, not representation
Aug 10, 2015
ab6f3c8
Sparse matrix compare for equals
Aug 11, 2015
bb70d5e
Breeze compare for dense matrices as well, in case other is sparse
Aug 11, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,8 +257,7 @@ class DenseMatrix(
this(numRows, numCols, values, false)

override def equals(o: Any): Boolean = o match {
case m: DenseMatrix =>
m.numRows == numRows && m.numCols == numCols && Arrays.equals(toArray, m.toArray)
case m: Matrix => toBreeze == m.toBreeze
case _ => false
}

Expand Down Expand Up @@ -519,6 +518,11 @@ class SparseMatrix(
rowIndices: Array[Int],
values: Array[Double]) = this(numRows, numCols, colPtrs, rowIndices, values, false)

override def equals(o: Any): Boolean = o match {
case m: Matrix => toBreeze == m.toBreeze
case _ => false
}

private[mllib] def toBreeze: BM[Double] = {
if (!isTransposed) {
new BSM[Double](values, numRows, numCols, colPtrs, rowIndices)
Expand Down
18 changes: 18 additions & 0 deletions mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,24 @@ class MatricesSuite extends SparkFunSuite {
}
}

test("equals") {
Copy link
Member

@jkbradley jkbradley Aug 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you expand this to compare dense with sparse?

Copy link
Contributor Author

@feynmanliang feynmanliang Aug 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

val dm1 = Matrices.dense(2, 2, Array(0.0, 1.0, 2.0, 3.0))
assert(dm1 === dm1)
assert(dm1 !== dm1.transpose)

val dm2 = Matrices.dense(2, 2, Array(0.0, 2.0, 1.0, 3.0))
assert(dm1 === dm2.transpose)

val sm1 = dm1.asInstanceOf[DenseMatrix].toSparse
assert(sm1 === sm1)
assert(sm1 === dm1)
assert(sm1 !== sm1.transpose)

val sm2 = dm2.asInstanceOf[DenseMatrix].toSparse
assert(sm1 === sm2.transpose)
assert(sm1 === dm2.transpose)
}

test("matrix copies are deep copies") {
val m = 3
val n = 2
Expand Down