Skip to content

[SPARK-47719][SQL] Change spark.sql.legacy.timeParserPolicy default to CORRECTED #45859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
srielau wants to merge 9 commits into from
Closed

[SPARK-47719][SQL] Change spark.sql.legacy.timeParserPolicy default to CORRECTED #45859

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
9fdf8df
[SPARK-47719] LEGACY_TIME_PARSER_POLICY_KEY default to CORRECTED
srielau Apr 3, 2024
8c51461
Add files
srielau Apr 3, 2024
ed0d9e2
Fix test mismatches,address comments
srielau Apr 4, 2024
33e81f0
Fix DefaultSqlApiConf
srielau Apr 4, 2024
7037125
Remove unneded new tests, fix other tests, fix docs
srielau Apr 5, 2024
8b786fe
Update DatetimeFormatterSuite.scala
cloud-fan Apr 5, 2024
e78e5eb
Fix scala style and another testcase
srielau Apr 5, 2024
8fba689
Merge branch 'SPARK-47719-parser-policy-default-to-corrected' of http…
srielau Apr 5, 2024
0792f3e
Fix spark connect testcase
srielau Apr 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4016,13 +4016,13 @@ object SQLConf {
.doc("When LEGACY, java.text.SimpleDateFormat is used for formatting and parsing " +
"dates/timestamps in a locale-sensitive manner, which is the approach before Spark 3.0. " +
"When set to CORRECTED, classes from java.time.* packages are used for the same purpose. " +
"The default value is EXCEPTION, RuntimeException is thrown when we will get different " +
"results.")
"When set to EXCEPTION, RuntimeException is thrown when we will get different " +
"results. The default is CORRECTED.")
.version("3.0.0")
.stringConf
.transform(_.toUpperCase(Locale.ROOT))
.checkValues(LegacyBehaviorPolicy.values.map(_.toString))
.createWithDefault(LegacyBehaviorPolicy.EXCEPTION.toString)
.createWithDefault(LegacyBehaviorPolicy.CORRECTED.toString)

val LEGACY_ARRAY_EXISTS_FOLLOWS_THREE_VALUED_LOGIC =
buildConf("spark.sql.legacy.followThreeValuedLogicInArrayExists")
Expand Down
Loading