Skip to content

[SPARK-13404] [SQL] Create variables for input row when it's actually used #11274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davies wants to merge 14 commits into from
Closed

[SPARK-13404] [SQL] Create variables for input row when it's actually used #11274

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c92e457
improve codegen
Feb 19, 2016
e58c8a6
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 22, 2016
f6139e6
Defer the evaluation of expresssions in Project
Feb 22, 2016
4faf5f9
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 23, 2016
4fb0bc8
fix broadcast hash join
Feb 24, 2016
ef9e8f3
fix bug
Feb 25, 2016
705da3f
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 25, 2016
ca8fe0f
fix aggregate
Feb 25, 2016
853502e
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 26, 2016
1a1452e
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 29, 2016
62682b2
Merge branch 'master' of github.com:apache/spark into gen_defer
Mar 4, 2016
76ca6c6
Merge branch 'master' of github.com:apache/spark into gen_defer
Mar 4, 2016
ffc9d8c
fix tests
Mar 5, 2016
f431170
improve docs
Mar 7, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix aggregate
  • Loading branch information
Davies Liu committed Feb 25, 2016
commit ca8fe0f5f55cabb1bb5903c3e85c150b31eaa7c7
1 change: 0 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,6 @@ private[sql] case class PhysicalRDD(
| while ($input.hasNext()) {
| InternalRow $row = (InternalRow) $input.next();
| $numOutputRows.add(1);
| ${columns.map(_.code).mkString("\n").trim}
| ${consume(ctx, columns).trim}
| if (shouldStop()) {
| return;
Expand Down
5 changes: 3 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,14 +175,15 @@ case class TungstenAggregate(
val aggResults = functions.map(_.evaluateExpression).map { e =>
BindReferences.bindReference(e, aggregateBufferAttributes).gen(ctx)
}
val evaluateAggResults = evaluateVariables(aggResults)
// evaluate result expressions
ctx.currentVars = aggResults
val resultVars = resultExpressions.map { e =>
BindReferences.bindReference(e, aggregateAttributes).gen(ctx)
}
(resultVars, s"""
| ${evaluateVariables(aggResults)}
| ${evaluateVariables(resultVars)}
|$evaluateAggResults
|${evaluateVariables(resultVars)}
""".stripMargin)
} else if (modes.contains(Partial) || modes.contains(PartialMerge)) {
// output the aggregate buffer directly
Expand Down