Skip to content

[SPARK-34538][SQL] Hive Metastore support filter by not-in #31646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ulysses-you wants to merge 15 commits into from
Closed

[SPARK-34538][SQL] Hive Metastore support filter by not-in #31646

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
3f95df6
init
ulysses-you Feb 25, 2021
3cefafe
fix test
ulysses-you Feb 26, 2021
0dc1c4a
add null test
ulysses-you Feb 26, 2021
b9c7f44
avoid metastore stack over flow
ulysses-you Feb 26, 2021
cf3ba56
nit
ulysses-you Feb 26, 2021
0415448
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
ulysses-you Feb 26, 2021
113bd62
doc
ulysses-you Mar 10, 2021
01b7466
add null precheck
ulysses-you Mar 10, 2021
e37f5f9
test
ulysses-you Mar 10, 2021
93eedd3
pattern
ulysses-you Mar 10, 2021
e8c7b6c
move forward
ulysses-you Mar 10, 2021
062b1d4
doc
ulysses-you Mar 10, 2021
4704ec3
not in datetype suite
ulysses-you Mar 10, 2021
5acbf6a
HivePartitionFilteringSuite
ulysses-you Mar 11, 2021
17aea47
remove int test
ulysses-you Mar 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add null test
  • Loading branch information
@ulysses-you
ulysses-you committed Feb 26, 2021
commit 0dc1c4a7829ea7261688cebeabc1638245fd4ede
12 changes: 7 additions & 5 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/client/FiltersSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,25 +109,27 @@ class FiltersSuite extends SparkFunSuite with Logging with PlanTest {
"datecol != 2019-01-01")

filterTest("not-in int filter",
Copy link
Contributor

@c21 c21 Feb 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add a test case for NULL as while, e.g. NOT IN (1, 2, ..., NULL)? Given we have bug before.

Copy link
Contributor Author

@ulysses-you ulysses-you Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it should be, add the null value for test.

Copy link
Contributor Author

@ulysses-you ulysses-you Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems exists some issue about null value. Created #31659.

(Not(In(a("intcol", IntegerType), Seq(Literal(1), Literal(2))))) :: Nil,
(Not(In(a("intcol", IntegerType), Seq(Literal(1), Literal(2), Literal(null))))) :: Nil,
"(intcol != 1 and intcol != 2)")

filterTest("not-in string filter",
(Not(In(a("strcol", StringType), Seq(Literal("a"), Literal("b"))))) :: Nil,
(Not(In(a("strcol", StringType), Seq(Literal("a"), Literal("b"), Literal(null))))) :: Nil,
"""(strcol != "a" and strcol != "b")""")

filterTest("not-inset, int filter",
(Not(InSet(a("intcol", IntegerType), Set(1, 2)))) :: Nil,
(Not(InSet(a("intcol", IntegerType), Set(1, 2, null)))) :: Nil,
"(intcol != 1 and intcol != 2)")

filterTest("not-inset, string filter",
(Not(InSet(a("strcol", StringType), Set(Literal("a").eval(), Literal("b").eval())))) :: Nil,
(Not(InSet(a("strcol", StringType),
Set(Literal("a").eval(), Literal("b").eval(), Literal(null).eval())))) :: Nil,
"""(strcol != "a" and strcol != "b")""")

filterTest("not-inset, date filter",
(Not(InSet(a("datecol", DateType),
Set(Literal(Date.valueOf("2020-01-01")).eval(),
Literal(Date.valueOf("2020-01-02")).eval())))) :: Nil,
Literal(Date.valueOf("2020-01-02")).eval(),
Literal(null).eval())))) :: Nil,
"""(datecol != 2020-01-01 and datecol != 2020-01-02)""")

// Applying the predicate `x IN (NULL)` should return an empty set, but since this optimization
Expand Down