Skip to content

[SPARK-33861][SQL] Simplify conditional in predicate #30865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 8 commits into from
Closed

[SPARK-33861][SQL] Simplify conditional in predicate #30865

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f786f18
Simplify conditional in predicate
wangyum Dec 20, 2020
45fd7a5
Merge remote-tracking branch 'upstream/master' into SPARK-33861
wangyum Dec 22, 2020
448faf0
fix
wangyum Dec 22, 2020
94551ea
Merge remote-tracking branch 'upstream/master' into SPARK-33861
wangyum Dec 23, 2020
c90227a
fix
wangyum Dec 23, 2020
cccbaf1
fix
wangyum Dec 24, 2020
8bd9ef9
further optimize
wangyum Dec 24, 2020
878beb0
fix
wangyum Dec 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ abstract class Optimizer(catalogManager: CatalogManager)
RemoveDispensableExpressions,
SimplifyBinaryComparison,
ReplaceNullWithFalseInPredicate,
SimplifyConditionalInPredicate,
PruneFilters,
SimplifyCasts,
SimplifyCaseConversionExpressions,
Expand Down
28 changes: 28 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.util.Utils

/*
* Optimization rules defined in this file should not affect the structure of the logical plan.
Expand Down Expand Up @@ -570,6 +571,33 @@ object PushFoldableIntoBranches extends Rule[LogicalPlan] with PredicateHelper {
}


object SimplifyConditionalInPredicate extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case f @ Filter(cond, _) => f.copy(condition = simplifyConditional(cond))
}

private def simplifyConditional(e: Expression): Expression = e match {
case cw @ CaseWhen(branches, elseValue) if cw.dataType == BooleanType && branches.size == 1 &&
elseValue.forall(_.semanticEquals(FalseLiteral)) =>
val (whenVal, thenVal) = branches.head
And(whenVal, thenVal)
case i @ If(pred, trueVal, FalseLiteral) if i.dataType == BooleanType =>
And(pred, trueVal)
case e if e.dataType == BooleanType =>
e
case e =>
val message = "Expected a Boolean type expression in simplifyConditional, " +
s"but got the type `${e.dataType.catalogString}` in `${e.sql}`."
if (Utils.isTesting) {
throw new IllegalArgumentException(message)
} else {
logWarning(message)
e
}
}
}


/**
* Simplifies LIKE expressions that do not need full regular expressions to evaluate the condition.
* For example, when the expression is just checking to see if a string starts with a given
Expand Down