Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
800f6e5
Strict fromDayTimeString
MaxGekk Nov 11, 2019
8cf38db
Add tests
MaxGekk Nov 11, 2019
3a1a710
Update tests in IntervalUtilsSuite
MaxGekk Nov 11, 2019
4e051d6
Fix ExpressionParserSuite
MaxGekk Nov 11, 2019
a1ef591
Add tests for invalid input into literals.sql
MaxGekk Nov 11, 2019
e88c8e0
Regen interval.sql.out
MaxGekk Nov 11, 2019
c87cf8c
Add the config spark.sql.legacy.fromDayTimeString.enabled
MaxGekk Nov 12, 2019
3f66881
Restore previous version of fromDayTimeString
MaxGekk Nov 12, 2019
da9b801
Restore old tests in IntervalUtilsSuite
MaxGekk Nov 12, 2019
c8afd33
Enable legacy behavior for postgresql tests
MaxGekk Nov 12, 2019
988d532
Regen interval.sql.out
MaxGekk Nov 12, 2019
ce28745
Set LEGACY_FROM_DAYTIME_STRING to true in ThriftServerQueryTestSuite
MaxGekk Nov 12, 2019
58ba7b4
Address Wenchen's review comment about a test
MaxGekk Nov 12, 2019
60fe0c1
Add comments for parseDayTimeLegacy
MaxGekk Nov 12, 2019
e95068e
Avoid unnecessary catching of exceptions
MaxGekk Nov 12, 2019
880e7ed
Add comments for parseDayTime()
MaxGekk Nov 12, 2019
d00c95d
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 12, 2019
a37bad4
Regen literals.sql.out
MaxGekk Nov 12, 2019
8e733c1
Remove exact ops
MaxGekk Nov 12, 2019
6b5b7ef
Update the SQL migration guide
MaxGekk Nov 12, 2019
a2ce9ae
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 13, 2019
fc77452
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 15, 2019
6be5f4e
truncation -> truncated
MaxGekk Nov 15, 2019
d253094
Make the config internal
MaxGekk Nov 15, 2019
dfd0dce
Use the legacy method in the PostgreSQL dialect
MaxGekk Nov 15, 2019
d1145cd
Update the SQL migration guide
MaxGekk Nov 15, 2019
c94f1df
Reorganize tests in interval.sql
MaxGekk Nov 15, 2019
833c7b0
Regen interval.sql.out
MaxGekk Nov 15, 2019
5b26335
Check in the PostgreSQL dialect as well
MaxGekk Nov 17, 2019
f401bd2
Black list interval.sql in ThriftServerQueryTestSuite
MaxGekk Nov 17, 2019
ca46f44
Set settings explicitly in tests
MaxGekk Nov 19, 2019
eadaa92
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 19, 2019
8f10259
Regen interval.sql.out
MaxGekk Nov 19, 2019
d3d730a
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 20, 2019
32b4d2f
Regen interval.sql.out
MaxGekk Nov 20, 2019
e39ca52
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 22, 2019
e012f8b
Remove explicit set
MaxGekk Nov 22, 2019
73ef32f
Regen interval.sql.out
MaxGekk Nov 22, 2019
d27d434
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Nov 22, 2019
9d8394e
Merge remote-tracking branch 'remotes/origin/master' into strict-from…
MaxGekk Dec 10, 2019
ef2cbe1
Remove usage of usePostgreSQLDialect
MaxGekk Dec 10, 2019
f9510e3
Regen interval.sql.out
MaxGekk Dec 10, 2019
c16f2a7
Replace 999999999 by 123456789 in seconds fractions in tests
MaxGekk Dec 11, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove usage of usePostgreSQLDialect
  • Loading branch information
MaxGekk committed Dec 10, 2019
commit ef2cbe14965b2fd28513abf452d6da416aae1b78
2 changes: 1 addition & 1 deletion docs/sql-migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ license: |

- Since Spark 3.0, the unary arithmetic operator plus(`+`) only accepts string, numeric and interval type values as inputs. Besides, `+` with a integral string representation will be coerced to double value, e.g. `+'1'` results `1.0`. In Spark version 2.4 and earlier, this operator is ignored. There is no type checking for it, thus, all type values with a `+` prefix are valid, e.g. `+ array(1, 2)` is valid and results `[1, 2]`. Besides, there is no type coercion for it at all, e.g. in Spark 2.4, the result of `+'1'` is string `1`.

- Since Spark 3.0, day-time interval strings are converted to intervals with respect to the `from` and `to` bounds. If an input string does not match to the pattern defined by specified bounds, the `ParseException` exception is thrown. For example, `interval '2 10:20' hour to minute` raises the exception because the expected format is `[+|-]h[h]:[m]m`. In Spark version 2.4, the `from` bound was not taken into account, and the `to` bound was used to truncate the resulted interval. For instance, the day-time interval string from the showed example is converted to `interval 10 hours 20 minutes`. To restore the behavior before Spark 3.0, you can set `spark.sql.legacy.fromDayTimeString.enabled` to `true` or set `spark.sql.dialect` to `PostgreSQL`.
- Since Spark 3.0, day-time interval strings are converted to intervals with respect to the `from` and `to` bounds. If an input string does not match to the pattern defined by specified bounds, the `ParseException` exception is thrown. For example, `interval '2 10:20' hour to minute` raises the exception because the expected format is `[+|-]h[h]:[m]m`. In Spark version 2.4, the `from` bound was not taken into account, and the `to` bound was used to truncate the resulted interval. For instance, the day-time interval string from the showed example is converted to `interval 10 hours 20 minutes`. To restore the behavior before Spark 3.0, you can set `spark.sql.legacy.fromDayTimeString.enabled` to `true`.

## Upgrading from Spark SQL 2.4 to 2.4.1

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,6 @@ object IntervalUtils {
def fromDayTimeString(input: String, from: IntervalUnit, to: IntervalUnit): CalendarInterval = {
if (SQLConf.get.getConf(SQLConf.LEGACY_FROM_DAYTIME_STRING)) {
parseDayTimeLegacy(input, from, to)
} else if (SQLConf.get.usePostgreSQLDialect) {
parseDayTimeLegacy(input, from, to)
} else {
parseDayTime(input, from, to)
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -163,47 +163,43 @@ class IntervalUtilsSuite extends SparkFunSuite with SQLHelper {
}

test("from day-time string - legacy") {
Seq(
SQLConf.LEGACY_FROM_DAYTIME_STRING.key -> "true",
SQLConf.DIALECT.key -> "PostgreSQL").foreach { config =>
withSQLConf(config) {
assert(fromDayTimeString("5 12:40:30.999999999") ===
new CalendarInterval(
0,
5,
12 * MICROS_PER_HOUR +
40 * MICROS_PER_MINUTE +
30 * MICROS_PER_SECOND + 999999L))
assert(fromDayTimeString("10 0:12:0.888") ===
new CalendarInterval(
0,
10,
12 * MICROS_PER_MINUTE + 888 * MICROS_PER_MILLIS))
assert(fromDayTimeString("-3 0:0:0") === new CalendarInterval(0, -3, 0L))

try {
fromDayTimeString("5 30:12:20")
fail("Expected to throw an exception for the invalid input")
} catch {
case e: IllegalArgumentException =>
assert(e.getMessage.contains("hour 30 outside range"))
}

try {
fromDayTimeString("5 30-12")
fail("Expected to throw an exception for the invalid input")
} catch {
case e: IllegalArgumentException =>
assert(e.getMessage.contains("must match day-time format"))
}

try {
fromDayTimeString("5 1:12:20", HOUR, MICROSECOND)
fail("Expected to throw an exception for the invalid convention type")
} catch {
case e: IllegalArgumentException =>
assert(e.getMessage.contains("Cannot support (interval"))
}
withSQLConf(SQLConf.LEGACY_FROM_DAYTIME_STRING.key -> "true") {
assert(fromDayTimeString("5 12:40:30.999999999") ===
new CalendarInterval(
0,
5,
12 * MICROS_PER_HOUR +
40 * MICROS_PER_MINUTE +
30 * MICROS_PER_SECOND + 999999L))
assert(fromDayTimeString("10 0:12:0.888") ===
new CalendarInterval(
0,
10,
12 * MICROS_PER_MINUTE + 888 * MICROS_PER_MILLIS))
assert(fromDayTimeString("-3 0:0:0") === new CalendarInterval(0, -3, 0L))

try {
fromDayTimeString("5 30:12:20")
fail("Expected to throw an exception for the invalid input")
} catch {
case e: IllegalArgumentException =>
assert(e.getMessage.contains("hour 30 outside range"))
}

try {
fromDayTimeString("5 30-12")
fail("Expected to throw an exception for the invalid input")
} catch {
case e: IllegalArgumentException =>
assert(e.getMessage.contains("must match day-time format"))
}

try {
fromDayTimeString("5 1:12:20", HOUR, MICROSECOND)
fail("Expected to throw an exception for the invalid convention type")
} catch {
case e: IllegalArgumentException =>
assert(e.getMessage.contains("Cannot support (interval"))
}
}
}
Expand Down