Skip to content

[SPARK-42551][SQL] Support more subexpression elimination cases #41119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wankunde wants to merge 10 commits into from
Closed

[SPARK-42551][SQL] Support more subexpression elimination cases #41119

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ffe4ca8
[SPARK-42551][SQL] Support more subexpression elimination cases
wankunde Feb 24, 2023
db77ea2
Support whole stage subexpressions elimination
wankunde Apr 8, 2023
e5ee05e
Bug fix for whole stage subexpression elimination
wankunde Apr 13, 2023
f152b2a
Bug fix
wankunde Apr 13, 2023
9a7cdc6
Do not support CSE in produce method
wankunde Apr 14, 2023
076792f
Clear stale commonExpressions when reoptimize plan
wankunde Apr 14, 2023
f524ef5
Remove whole-stage expression elimination
wankunde May 9, 2023
670446a
Merge code into origin CSE
wankunde May 10, 2023
fbfa00b
bug fix
wankunde May 10, 2023
06238be
Consider common expression local input variables.
wankunde May 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Bug fix
  • Loading branch information
@wankunde
wankunde committed May 9, 2023
commit f152b2aaa64cd2f0803cbd8f27a2190414314678
46 changes: 26 additions & 20 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,30 +196,36 @@ abstract class Expression extends TreeNode[Expression] {
}.getOrElse {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val exprKey = ExpressionEquals(this)
val eval = ctx.commonExpressions.get(exprKey) match {
case Some(stats) =>
// We should reuse the currentVar references which code is not empty
val nonEmptyRefs = this.exists {
case BoundReference(ordinal, _, _) =>
ctx.currentVars != null && ctx.currentVars(ordinal) != null &&
ctx.currentVars(ordinal).code != EmptyBlock
case _ => false
}
val eval = doGenCode(ctx, ExprCode(
JavaCode.isNullVariable(isNull),
JavaCode.variable(value, dataType)))
if (eval.code != EmptyBlock && !nonEmptyRefs) {
ctx.genReusedCode(stats, eval)
} else {
eval
}
val eval =
if (EquivalentExpressions.supportedExpression(this)) {
ctx.commonExpressions.get(ExpressionEquals(this)) match {
case Some(stats) =>
// We should reuse the currentVar references which code is not empty
val nonEmptyRefs = this.exists {
case BoundReference(ordinal, _, _) =>
ctx.currentVars != null && ctx.currentVars(ordinal) != null &&
ctx.currentVars(ordinal).code != EmptyBlock
case _ => false
}
val eval = doGenCode(ctx, ExprCode(
JavaCode.isNullVariable(isNull),
JavaCode.variable(value, dataType)))
if (eval.code != EmptyBlock && !nonEmptyRefs) {
ctx.genReusedCode(stats, eval)
} else {
eval
}

case None =>
case None =>
doGenCode(ctx, ExprCode(
JavaCode.isNullVariable(isNull),
JavaCode.variable(value, dataType)))
}
} else {
doGenCode(ctx, ExprCode(
JavaCode.isNullVariable(isNull),
JavaCode.variable(value, dataType)))
}
}
reduceCodeSize(ctx, eval)
if (eval.code.toString.nonEmpty) {
// Add `this` in the comment.
Expand Down
4 changes: 2 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -680,12 +680,12 @@ case class WholeStageCodegenExec(child: SparkPlan)(val codegenStageId: Int)
// If the input attributes changed, collect current common expressions and clear
// equivalentExpressions
if (attributeSeq.size != newAttributeSeq.size ||
attributeSeq.zip(newAttributeSeq).exists(tup => !tup._1.equals(tup._2))) {
attributeSeq.zip(newAttributeSeq).exists { case (left, right) => left != right }) {
equivalence = new EquivalentExpressions
}
if (newReusableExpressions.nonEmpty) {
val bondExpressions =
BindReferences.bindReferences(newReusableExpressions, newAttributeSeq.toSeq)
BindReferences.bindReferences(newReusableExpressions, newAttributeSeq)
executeSeq += ((c, bondExpressions, equivalence))
ctx.wholeStageSubexpressionElimination(bondExpressions, equivalence)
}
Expand Down