Skip to content

[SPARK-29231][SQL] Constraints should be inferred from cast equality constraint #27252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 10 commits into from
Closed

[SPARK-29231][SQL] Constraints should be inferred from cast equality constraint #27252

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
048a0ec
Constraints should be inferred from cast equality constraint
wangyum Jan 17, 2020
34c7001
Support filter at lower data type
wangyum Jan 19, 2020
7dcfe91
fix test error
wangyum Jan 20, 2020
2060190
Only support filter higher data type
wangyum Jan 31, 2020
420058b
IsNotNull should be constructed by `constructIsNotNullConstraints`.
wangyum Feb 4, 2020
8dcd076
Address comment
wangyum Feb 11, 2020
47eadf4
Address comment
wangyum Feb 12, 2020
d283028
constraints -> predicates
wangyum Feb 12, 2020
d055eba
Improve test coverage
wangyum Feb 12, 2020
4b14b3f
Improve test coverage
wangyum Feb 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Improve test coverage
  • Loading branch information
@wangyum
wangyum committed Feb 12, 2020
commit 4b14b3f619af56babc981e729f43e165369cad01
11 changes: 11 additions & 0 deletions ...test/scala/org/apache/spark/sql/catalyst/optimizer/InferFiltersFromConstraintsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,17 @@ class InferFiltersFromConstraintsSuite extends PlanTest {
Some("right.b".attr === "left.a".attr.cast(LongType))).foreach { condition =>
testConstraintsAfterJoin(originalLeft, originalRight, left, right, Inner, condition)
Copy link
Member

@maropu maropu Feb 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In terms of test coverage, its better to test both cases (left/right-side casts)? I have the same comment in the test below.

}
Copy link
Contributor

@cloud-fan cloud-fan Feb 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also test cast(int) here. The key is: we should test both left side cast and right side cast, as @maropu said.

"left.a".attr.cast(LongType) === "right.b".attr and "right.b".attr === "left.a".attr.cast(LongType) only test left side cast (join left side, not EqualTo left side).


Seq(Some("left.a".attr === "right.b".attr.cast(IntegerType)),
Some("right.b".attr.cast(IntegerType) === "left.a".attr)).foreach { condition =>
testConstraintsAfterJoin(
originalLeft,
originalRight,
testRelation1.where(IsNotNull('a)).subquery('left),
right,
Inner,
condition)
}
}

test("Constraints shouldn't be inferred from cast equality constraint(filter lower data type)") {
Expand Down