Skip to content

[SPARK-29864][SPARK-29920][SQL] Strict parsing of day-time strings to intervals #26473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 43 commits into from
+818 −439
Closed

[SPARK-29864][SPARK-29920][SQL] Strict parsing of day-time strings to intervals #26473

Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
800f6e5
Strict fromDayTimeString
MaxGekk Nov 11, 2019
8cf38db
Add tests
MaxGekk Nov 11, 2019
3a1a710
Update tests in IntervalUtilsSuite
MaxGekk Nov 11, 2019
4e051d6
Fix ExpressionParserSuite
MaxGekk Nov 11, 2019
a1ef591
Add tests for invalid input into literals.sql
MaxGekk Nov 11, 2019
e88c8e0
Regen interval.sql.out
MaxGekk Nov 11, 2019
c87cf8c
Add the config spark.sql.legacy.fromDayTimeString.enabled
MaxGekk Nov 12, 2019
3f66881
Restore previous version of fromDayTimeString
MaxGekk Nov 12, 2019
da9b801
Restore old tests in IntervalUtilsSuite
MaxGekk Nov 12, 2019
c8afd33
Enable legacy behavior for postgresql tests
MaxGekk Nov 12, 2019
988d532
Regen interval.sql.out
MaxGekk Nov 12, 2019
ce28745
Set LEGACY_FROM_DAYTIME_STRING to true in ThriftServerQueryTestSuite
MaxGekk Nov 12, 2019
58ba7b4
Address Wenchen's review comment about a test
MaxGekk Nov 12, 2019
60fe0c1
Add comments for parseDayTimeLegacy
MaxGekk Nov 12, 2019
e95068e
Avoid unnecessary catching of exceptions
MaxGekk Nov 12, 2019
880e7ed
Add comments for parseDayTime()
MaxGekk Nov 12, 2019
d00c95d
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 12, 2019
a37bad4
Regen literals.sql.out
MaxGekk Nov 12, 2019
8e733c1
Remove exact ops
MaxGekk Nov 12, 2019
6b5b7ef
Update the SQL migration guide
MaxGekk Nov 12, 2019
a2ce9ae
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 13, 2019
fc77452
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 15, 2019
6be5f4e
truncation -> truncated
MaxGekk Nov 15, 2019
d253094
Make the config internal
MaxGekk Nov 15, 2019
dfd0dce
Use the legacy method in the PostgreSQL dialect
MaxGekk Nov 15, 2019
d1145cd
Update the SQL migration guide
MaxGekk Nov 15, 2019
c94f1df
Reorganize tests in interval.sql
MaxGekk Nov 15, 2019
833c7b0
Regen interval.sql.out
MaxGekk Nov 15, 2019
5b26335
Check in the PostgreSQL dialect as well
MaxGekk Nov 17, 2019
f401bd2
Black list interval.sql in ThriftServerQueryTestSuite
MaxGekk Nov 17, 2019
ca46f44
Set settings explicitly in tests
MaxGekk Nov 19, 2019
eadaa92
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 19, 2019
8f10259
Regen interval.sql.out
MaxGekk Nov 19, 2019
d3d730a
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 20, 2019
32b4d2f
Regen interval.sql.out
MaxGekk Nov 20, 2019
e39ca52
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 22, 2019
e012f8b
Remove explicit set
MaxGekk Nov 22, 2019
73ef32f
Regen interval.sql.out
MaxGekk Nov 22, 2019
d27d434
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 22, 2019
9d8394e
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Dec 10, 2019
ef2cbe1
Remove usage of usePostgreSQLDialect
MaxGekk Dec 10, 2019
f9510e3
Regen interval.sql.out
MaxGekk Dec 10, 2019
c16f2a7
Replace 999999999 by 123456789 in seconds fractions in tests
MaxGekk Dec 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Replace 999999999 by 123456789 in seconds fractions in tests
  • Loading branch information
@MaxGekk
MaxGekk committed Dec 11, 2019
commit c16f2a7e1919aaafc7b3de35ad6fba3dba2c2e13
12 changes: 6 additions & 6 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,14 +416,14 @@ class IntervalUtilsSuite extends SparkFunSuite with SQLHelper {
checkFail("5 12:40", HOUR, MINUTE, "must match day-time format")

check("12:40:30.999999999", HOUR, SECOND, "12 hours 40 minutes 30.999999 seconds")
Copy link
Contributor

@cloud-fan cloud-fan Nov 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we test 12:40:30.0123456789 as well?

check("+12:40:30.999999999", HOUR, SECOND, "12 hours 40 minutes 30.999999 seconds")
check("-12:40:30.999999999", HOUR, SECOND, "-12 hours -40 minutes -30.999999 seconds")
check("+12:40:30.123456789", HOUR, SECOND, "12 hours 40 minutes 30.123456 seconds")
check("-12:40:30.123456789", HOUR, SECOND, "-12 hours -40 minutes -30.123456 seconds")
checkFail("5 12:40:30", HOUR, SECOND, "must match day-time format")
checkFail("12:40:30.0123456789", HOUR, SECOND, "must match day-time format")

check("40:30.999999999", MINUTE, SECOND, "40 minutes 30.999999 seconds")
check("+40:30.999999999", MINUTE, SECOND, "40 minutes 30.999999 seconds")
check("-40:30.999999999", MINUTE, SECOND, "-40 minutes -30.999999 seconds")
check("40:30.123456789", MINUTE, SECOND, "40 minutes 30.123456 seconds")
check("+40:30.123456789", MINUTE, SECOND, "40 minutes 30.123456 seconds")
check("-40:30.123456789", MINUTE, SECOND, "-40 minutes -30.123456 seconds")
checkFail("12:40:30", MINUTE, SECOND, "must match day-time format")

check("5 12", DAY, HOUR, "5 days 12 hours")
Expand All @@ -438,7 +438,7 @@ class IntervalUtilsSuite extends SparkFunSuite with SQLHelper {

check("5 12:40:30.123", DAY, SECOND, "5 days 12 hours 40 minutes 30.123 seconds")
check("+5 12:40:30.123456", DAY, SECOND, "5 days 12 hours 40 minutes 30.123456 seconds")
check("-5 12:40:30.999999999", DAY, SECOND, "-5 days -12 hours -40 minutes -30.999999 seconds")
check("-5 12:40:30.123456789", DAY, SECOND, "-5 days -12 hours -40 minutes -30.123456 seconds")
checkFail("5 12", DAY, SECOND, "must match day-time format")

checkFail("5 30:12:20", DAY, SECOND, "hour 30 outside range")
Expand Down