Skip to content

SPARK-11420 Updating Stddev support via Imperative Aggregate #9380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JihongMA wants to merge 14 commits into from
Closed

SPARK-11420 Updating Stddev support via Imperative Aggregate #9380

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
5094db0
SPARK-11420: stddev via Imperative Aggregate
JihongMA Oct 30, 2015
0113626
handle null
JihongMA Oct 30, 2015
24195fe
address comment
JihongMA Nov 3, 2015
c360f11
rebase upstream
JihongMA Nov 3, 2015
4ca8b19
minor fix
JihongMA Nov 3, 2015
747a911
rebase with upstream
JihongMA Nov 4, 2015
57eeeed
rebase with upstream to revert stddev as alias of stddev_samp
JihongMA Nov 4, 2015
402971c
style fix
JihongMA Nov 4, 2015
e3417aa
rebase with upstream
JihongMA Nov 5, 2015
b69d1e6
address comment
JihongMA Nov 6, 2015
eea699a
rebase with upstream & handle NaN
JihongMA Nov 11, 2015
dc0558b
fix tests
JihongMA Nov 11, 2015
ca407bc
fix R test
JihongMA Nov 12, 2015
7a239ec
fix test_sparkSQL.R
JihongMA Nov 12, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comment
  • Loading branch information
@JihongMA
JihongMA committed Nov 6, 2015
commit b69d1e68d52739df83a9ebe797c83f4b8c5ef0dd
7 changes: 4 additions & 3 deletions ...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Kurtosis.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,10 @@ case class Kurtosis(child: Expression,
s"$prettyName requires ${momentOrder + 1} central moments, received: ${moments.length}")
val m2 = moments(2)
val m4 = moments(4)
if (n == 0.0 || m2 == 0.0) {
null
} else {

if (n == 0.0) null
else if (m2 == 0.0) Double.NaN
else {
n * m4 / (m2 * m2) - 3.0
}
}
Expand Down
7 changes: 4 additions & 3 deletions ...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Skewness.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,10 @@ case class Skewness(child: Expression,
s"$prettyName requires ${momentOrder + 1} central moments, received: ${moments.length}")
val m2 = moments(2)
val m3 = moments(3)
if (n == 0.0 || m2 == 0.0) {
null
} else {

if (n == 0.0) null
else if (m2 == 0.0) Double.NaN
else {
math.sqrt(n) * m3 / math.sqrt(m2 * m2 * m2)
}
}
Expand Down
4 changes: 3 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Stddev.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ case class StddevSamp(child: Expression,
require(moments.length == momentOrder + 1,
s"$prettyName requires ${momentOrder + 1} central moments, received: ${moments.length}")

if (n == 0.0 || n == 1.0) null else math.sqrt(moments(2) / (n - 1.0))
if (n == 0.0) null
else if (n == 1.0) Double.NaN
else math.sqrt(moments(2) / (n - 1.0))
}
}
4 changes: 3 additions & 1 deletion ...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Variance.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@ case class VarianceSamp(child: Expression,
require(moments.length == momentOrder + 1,
s"$prettyName requires ${momentOrder + 1} central moment, received: ${moments.length}")

if (n == 0.0 || n == 1.0) null else moments(2) / (n - 1.0)
if (n == 0.0) null
else if (n== 1.0) Double.NaN
else moments(2) / (n - 1.0)
}
}

Expand Down
4 changes: 2 additions & 2 deletions sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ class DataFrameAggregateSuite extends QueryTest with SharedSQLContext {
val input = Seq((1, 2)).toDF("a", "b")
checkAnswer(
input.agg(variance('a), var_samp('a), var_pop('a), skewness('a), kurtosis('a)),
Row(null, null, 0.0, null, null))
Row(Double.NaN, Double.NaN, 0.0, Double.NaN, Double.NaN))

checkAnswer(
input.agg(
Expand All @@ -229,7 +229,7 @@ class DataFrameAggregateSuite extends QueryTest with SharedSQLContext {
expr("var_pop(a)"),
expr("skewness(a)"),
expr("kurtosis(a)")),
Row(null, null, 0.0, null, null))
Row(Double.NaN, Double.NaN, 0.0, Double.NaN, Double.NaN))
}

test("null moments") {
Expand Down