Skip to content

[SPARK-4581][MLlib] Refactorize StandardScaler to improve the transformation performance #3435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dbtsai wants to merge 6 commits into from
Closed

[SPARK-4581][MLlib] Refactorize StandardScaler to improve the transformation performance #3435

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5bffd3d
first commit
Nov 24, 2014
fc795e4
update
Nov 25, 2014
9c51eef
style
Nov 25, 2014
cdb5cef
small change
Nov 25, 2014
daf2b06
Address the feedback
Nov 25, 2014
85885a9
revert to have lazy in shift array.
Nov 25, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
small change
  • Loading branch information
DB Tsai committed Nov 25, 2014
commit cdb5cefc89548e6d040b06765cce7b684c7854a9
11 changes: 7 additions & 4 deletions mllib/src/main/scala/org/apache/spark/mllib/feature/StandardScaler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,15 +101,16 @@ class StandardScalerModel private[mllib] (
vector match {
case dv: DenseVector =>
val values = dv.values.clone()
val size = values.size
var i = 0
if (withStd) {
val localFactor = factor
while (i < values.length) {
while (i < size) {
Copy link
Contributor

@mengxr mengxr Nov 25, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we move var i = 0 inside each closure? It feels safer.

values(i) = (values(i) - localShift(i)) * localFactor(i)
i += 1
}
} else {
while (i < values.length) {
while (i < size) {
values(i) -= localShift(i)
i += 1
}
Expand All @@ -122,8 +123,9 @@ class StandardScalerModel private[mllib] (
vector match {
case dv: DenseVector =>
val values = dv.values.clone()
val size = values.size
var i = 0
while(i < values.length) {
while(i < size) {
values(i) *= localFactor(i)
i += 1
}
Expand All @@ -133,8 +135,9 @@ class StandardScalerModel private[mllib] (
// so we can re-use it to save memory.
val indices = sv.indices
val values = sv.values.clone()
val size = values.size
Copy link
Contributor

@mengxr mengxr Nov 25, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

size -> nnz

var i = 0
while (i < indices.length) {
while (i < size) {
values(i) *= localFactor(indices(i))
i += 1
}
Expand Down