Skip to content

[SPARK-13404] [SQL] Create variables for input row when it's actually used #11274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davies wants to merge 14 commits into from
Closed

[SPARK-13404] [SQL] Create variables for input row when it's actually used #11274

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c92e457
improve codegen
Feb 19, 2016
e58c8a6
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 22, 2016
f6139e6
Defer the evaluation of expresssions in Project
Feb 22, 2016
4faf5f9
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 23, 2016
4fb0bc8
fix broadcast hash join
Feb 24, 2016
ef9e8f3
fix bug
Feb 25, 2016
705da3f
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 25, 2016
ca8fe0f
fix aggregate
Feb 25, 2016
853502e
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 26, 2016
1a1452e
Merge branch 'master' of github.com:apache/spark into gen_defer
Feb 29, 2016
62682b2
Merge branch 'master' of github.com:apache/spark into gen_defer
Mar 4, 2016
76ca6c6
Merge branch 'master' of github.com:apache/spark into gen_defer
Mar 4, 2016
ffc9d8c
fix tests
Mar 5, 2016
f431170
improve docs
Mar 7, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix bug
  • Loading branch information
Davies Liu committed Feb 25, 2016
commit ef9e8f3558b20f6a4a7c94eb7c46a8f435694ad4
15 changes: 9 additions & 6 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoin.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,11 +193,14 @@ case class BroadcastHashJoin(
val numOutput = metricTerm(ctx, "numOutputRows")

val checkCondition = if (condition.isDefined) {
val expr = condition.get
// evaluate the variables from build side that used by condition
val eval = evaluateRequiredVariables(buildPlan.output, buildVars, expr.references)
// filter the output via condition
ctx.currentVars = input ++ buildVars
val ev = BindReferences.bindReference(
condition.get, streamedPlan.output ++ buildPlan.output).gen(ctx)
val ev = BindReferences.bindReference(expr, streamedPlan.output ++ buildPlan.output).gen(ctx)
s"""
|$eval
|${ev.code}
|if (${ev.isNull} || !${ev.value}) continue;
""".stripMargin
Expand Down Expand Up @@ -257,10 +260,11 @@ case class BroadcastHashJoin(
// filter the output via condition
val conditionPassed = ctx.freshName("conditionPassed")
val checkCondition = if (condition.isDefined) {
val eval = evaluateRequiredVariables(buildPlan.output, buildVars, condition.get.references)
val expr = condition.get
// evaluate the variables from build side that used by condition
val eval = evaluateRequiredVariables(buildPlan.output, buildVars, expr.references)
ctx.currentVars = input ++ buildVars
val ev = BindReferences.bindReference(condition.get,
streamedPlan.output ++ buildPlan.output).gen(ctx)
val ev = BindReferences.bindReference(expr, streamedPlan.output ++ buildPlan.output).gen(ctx)
s"""
|boolean $conditionPassed = true;
|${eval.trim}
Expand All @@ -285,7 +289,6 @@ case class BroadcastHashJoin(
|UnsafeRow $matched = $anyNull ? null: (UnsafeRow)$relationTerm.getValue(${keyEv.value});
|${checkCondition.trim}
|if (!$conditionPassed) {
| // reset to null
| $matched = null;
| // reset the variables those are already evaluated.
Copy link
Contributor

@nongli nongli Mar 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this required?

Copy link
Contributor Author

@davies davies Mar 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we find a row on build side, but the condition do not pass, the variables for build side still have the value (from build side), but the result should be empty (outer join).

| ${buildVars.filter(_.code == "").map(v => s"${v.isNull} = true;").mkString("\n")}
Expand Down