Skip to content

[SPARK-11275] [SQL] Incorrect results when using rollup/cube #9815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
aray wants to merge 2 commits into from
Closed

[SPARK-11275] [SQL] Incorrect results when using rollup/cube #9815

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
12914fa
Fixes bug with grouping sets (including cube/rollup) where aggregates…
aray Nov 18, 2015
2162b6c
additional comments
aray Nov 19, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
additional comments
  • Loading branch information
@aray
aray committed Nov 19, 2015
commit 2162b6cac8d72959f4fe57dcfbdfe13ca286a615
6 changes: 5 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,8 +222,12 @@ class Analyzer(
}
Copy link
Contributor

@yhuai yhuai Nov 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. We need to distinguish the real grouping expression values and those nulls added by expand. Otherwise, we generate wrong results.


val aggregations: Seq[NamedExpression] = x.aggregations.map {
// Don't transform AggregateExpressions see SPARK-11275
// If an expression is an aggregate (contains a AggregateExpression) then we dont change
// it so that the aggregation is computed on the unmodified value of its argument
// expressions.
case expr if expr.find(_.isInstanceOf[AggregateExpression]).nonEmpty => expr
// If not then its a grouping expression and we need to use the modified (with nulls from
// Expand) value of the expression.
case expr => expr.transformDown {
case e => groupByAliases.find(_.child.semanticEquals(e)).map(_.toAttribute).getOrElse(e)
}.asInstanceOf[NamedExpression]
Expand Down