Skip to content

[SPARK-29231][SQL] Constraints should be inferred from cast equality constraint #27252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 10 commits into from
Closed

[SPARK-29231][SQL] Constraints should be inferred from cast equality constraint #27252

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
048a0ec
Constraints should be inferred from cast equality constraint
wangyum Jan 17, 2020
34c7001
Support filter at lower data type
wangyum Jan 19, 2020
7dcfe91
fix test error
wangyum Jan 20, 2020
2060190
Only support filter higher data type
wangyum Jan 31, 2020
420058b
IsNotNull should be constructed by `constructIsNotNullConstraints`.
wangyum Feb 4, 2020
8dcd076
Address comment
wangyum Feb 11, 2020
47eadf4
Address comment
wangyum Feb 12, 2020
d283028
constraints -> predicates
wangyum Feb 12, 2020
d055eba
Improve test coverage
wangyum Feb 12, 2020
4b14b3f
Improve test coverage
wangyum Feb 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address comment
  • Loading branch information
@wangyum
wangyum committed Feb 12, 2020
commit 47eadf4bbd35aa853175d28527278194c4f7225d
18 changes: 12 additions & 6 deletions ...test/scala/org/apache/spark/sql/catalyst/optimizer/InferFiltersFromConstraintsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,17 +283,23 @@ class InferFiltersFromConstraintsSuite extends PlanTest {
test("Constraints shouldn't be inferred from cast equality constraint(filter lower data type)") {
val testRelation1 = LocalRelation('a.int)
val testRelation2 = LocalRelation('b.long)
val originalLeft = testRelation1.where('a > 1).subquery('left)
val originalRight = testRelation2.where('b < 10).subquery('right)
val originalLeft = testRelation1.where('a === 1).subquery('left)
val originalRight = testRelation2.subquery('right)

val left = testRelation1.where(
IsNotNull('a) && 'a > 1 && 'a.cast(LongType) < Literal(10).cast(LongType)).subquery('left)
val right = testRelation2.where(
IsNotNull('b) && 'b < Literal(10).cast(LongType)).subquery('right)
val left = testRelation1.where(IsNotNull('a) && 'a === 1).subquery('left)
val right = testRelation2.where(IsNotNull('b)).subquery('right)

Seq(Some("left.a".attr.cast(LongType) === "right.b".attr),
Copy link
Contributor

@peter-toth peter-toth Feb 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might be wrong, but I find these test cases a bit confusing because left and right have equality filters. Eg. here 'a === 1 in left so actually it would be correct to infer 1.cast(LongType) === 'b for right. This PR doesn't do that obviously (#27518 will address that) but probably using inequalities ('a < 1) would be easier to follow.

Some("right.b".attr === "left.a".attr.cast(LongType))).foreach { condition =>
testConstraintsAfterJoin(originalLeft, originalRight, left, right, Inner, condition)
}

testConstraintsAfterJoin(
originalLeft,
originalRight,
left,
testRelation2.where(IsNotNull('b) && 'b.attr.cast(IntegerType) === 1).subquery('right),
Inner,
Some("left.a".attr === "right.b".attr.cast(IntegerType)))
}
}