Skip to content

[SPARK-33848][SQL][FOLLOWUP] Introduce allowList for push into (if / case) branches #30955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 4 commits into from
Closed

[SPARK-33848][SQL][FOLLOWUP] Introduce allowList for push into (if / case) branches #30955

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
814b3a4
fix
wangyum Dec 29, 2020
931c66c
fix
wangyum Dec 29, 2020
82a343c
fix
wangyum Dec 29, 2020
029fb53
fix
wangyum Dec 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix
  • Loading branch information
@wangyum
wangyum committed Dec 29, 2020
commit 931c66cd1cd771a7b797cc45651e276436c16c6c
25 changes: 18 additions & 7 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -549,18 +549,27 @@ object PushFoldableIntoBranches extends Rule[LogicalPlan] with PredicateHelper {
}

// Not all UnaryExpression support push into (if / case) branches, e.g. Alias.
Copy link
Contributor

@cloud-fan cloud-fan Dec 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not all UnaryExpression can be pushed into (if / case) branches, e.g. Alias.

private def supportedUnaryExpression(u: UnaryExpression): Boolean = u match {
case IsNull(_) | IsNotNull(_) => true
case _: UnaryMathExpression | Abs(_) | Bin(_) | Factorial(_) | Hex(_) => true
case _: String2StringExpression | Ascii(_) | Base64(_) | BitLength(_) | Chr(_) | Length(_) =>
private def supportedUnaryExpression(e: UnaryExpression): Boolean = e match {
case _: IsNull | _: IsNotNull => true
case _: UnaryMathExpression | _: Abs | _: Bin | _: Factorial | _: Hex => true
case _: String2StringExpression | _: Ascii | _: Base64 | _: BitLength | _: Chr | _: Length =>
true
case _: CastBase => true
case _: GetDateField | LastDay(_) => true
case _: GetDateField | _: LastDay => true
case _: ExtractIntervalPart => true
case _: ArraySetLike => true
case _ => false
Copy link
Contributor

@cloud-fan cloud-fan Dec 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's include ExtractValue as well, which is common with nested fields.

}

private def supportedBinaryExpression(e: BinaryExpression): Boolean = e match {
Copy link
Contributor

@cloud-fan cloud-fan Dec 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add comments as well.

case _: BinaryComparison | _: StringPredicate | _: StringRegexExpression => true
case _: BinaryArithmetic => true
case _: BinaryMathExpression => true
case _: AddMonths | _: DateAdd | _: DateAddInterval | _: DateDiff | _: DateSub => true
Copy link
Member

@MaxGekk MaxGekk Mar 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What the property should the expression have to be here? For example, can I add DateAddYMInterval, TimestampAddYMInterval and TimeAdd?

Copy link
Contributor

@cloud-fan cloud-fan Mar 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so

Copy link
Member

@MaxGekk MaxGekk Mar 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. I opened the JIRA for that: SPARK-34841

case _: FindInSet | _: RoundBase => true
case _ => false
}

def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case q: LogicalPlan => q transformExpressionsUp {
case u @ UnaryExpression(i @ If(_, trueValue, falseValue))
Expand All @@ -576,13 +585,15 @@ object PushFoldableIntoBranches extends Rule[LogicalPlan] with PredicateHelper {
elseValue.map(e => u.withNewChildren(Array(e))))

case b @ BinaryExpression(i @ If(_, trueValue, falseValue), right)
if right.foldable && atMostOneUnfoldable(Seq(trueValue, falseValue)) =>
if supportedBinaryExpression(b) && right.foldable &&
atMostOneUnfoldable(Seq(trueValue, falseValue)) =>
i.copy(
trueValue = b.withNewChildren(Array(trueValue, right)),
falseValue = b.withNewChildren(Array(falseValue, right)))

case b @ BinaryExpression(left, i @ If(_, trueValue, falseValue))
if left.foldable && atMostOneUnfoldable(Seq(trueValue, falseValue)) =>
if supportedBinaryExpression(b) && left.foldable &&
atMostOneUnfoldable(Seq(trueValue, falseValue)) =>
i.copy(
trueValue = b.withNewChildren(Array(left, trueValue)),
falseValue = b.withNewChildren(Array(left, falseValue)))
Expand Down