Skip to content

[SPARK-32721][SQL] Simplify if clauses with null and boolean #29567

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sunchao wants to merge 7 commits into from
Closed

[SPARK-32721][SQL] Simplify if clauses with null and boolean #29567

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f9e87f0
SPARK-32721
sunchao Aug 28, 2020
e4d929c
Fix bugs and merge into SimplifyConditionals
sunchao Aug 28, 2020
bb7c38c
Fix lint
sunchao Aug 28, 2020
cc66198
Fix test failure
sunchao Aug 28, 2020
60e0e92
Address comments
sunchao Aug 30, 2020
2b90c19
Test if evaluation with nullable condition
sunchao Aug 31, 2020
71bd033
Fix indentation
sunchao Aug 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix test failure
  • Loading branch information
@sunchao
sunchao committed Aug 28, 2020
commit cc661984f3ccbf59bbabb91c7d92b17524ab74d3
8 changes: 4 additions & 4 deletions ...t/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,14 +221,14 @@ class BooleanSimplificationSuite extends PlanTest with ExpressionEvalHelper with

test("Complementation Laws - null handling") {
checkCondition('e && !'e,
testRelationWithData.where(If('e.isNull, Literal.create(null, BooleanType), false)).analyze)
testRelationWithData.where(And(Literal(null, BooleanType), 'e.isNull)).analyze)
Copy link
Member

@viirya viirya Aug 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I think it is better to test BooleanSimplification like original, but BooleanSimplificationSuite's optimizer mixes SimplifyConditionals and BooleanSimplification.

Copy link
Member Author

@sunchao sunchao Aug 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. What is the recommended way for adding test suite for optimizers? should each test suite only use the corresponding optimizer rule, or it's OK to also add those relevant ones?

Copy link
Member

@viirya viirya Aug 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually we only test corresponding rule in each test suite. But some suites use multiple rules like this one. For this one, although we can still only test BooleanSimplification if we add another optimizer, but seems too much. Maybe currently it's good enough. Maybe add a comment.

checkCondition(!'e && 'e,
testRelationWithData.where(If('e.isNull, Literal.create(null, BooleanType), false)).analyze)
testRelationWithData.where(And(Literal(null, BooleanType), 'e.isNull)).analyze)

checkCondition('e || !'e,
testRelationWithData.where(If('e.isNull, Literal.create(null, BooleanType), true)).analyze)
testRelationWithData.where(Or('e.isNotNull, Literal(null, BooleanType))).analyze)
checkCondition(!'e || 'e,
testRelationWithData.where(If('e.isNull, Literal.create(null, BooleanType), true)).analyze)
testRelationWithData.where(Or('e.isNotNull, Literal(null, BooleanType))).analyze)
}

test("Complementation Laws - negative case") {
Expand Down