Skip to content

[SPARK-39731][SQL] Fix issue in CSV and JSON data sources when parsing dates in "yyyyMMdd" format with CORRECTED time parser policy #37147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sadikovi wants to merge 20 commits into from
Closed

[SPARK-39731][SQL] Fix issue in CSV and JSON data sources when parsing dates in "yyyyMMdd" format with CORRECTED time parser policy #37147

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
1193ce7
fix issue
sadikovi Jul 11, 2022
b714b7f
update code
sadikovi Jul 12, 2022
8a10a68
fix issue in json
sadikovi Jul 14, 2022
9b65761
update code
sadikovi Jul 14, 2022
45011a0
add a config option to control legacy behavior
sadikovi Jul 18, 2022
40d07bd
add a config for json
sadikovi Jul 18, 2022
55c5579
update docs and comments
sadikovi Jul 18, 2022
ef91606
fix scalastyle
sadikovi Jul 18, 2022
15c07f7
trigger ci
sadikovi Jul 18, 2022
a83288b
update comments
sadikovi Jul 20, 2022
bf9351d
trigger ci
sadikovi Jul 21, 2022
ac63b63
Merge remote-tracking branch 'upstream/master' into fix-csv-date-infe…
sadikovi Jul 22, 2022
a447b08
fix tests for SPARK-39469
sadikovi Jul 22, 2022
8feb707
Merge remote-tracking branch 'upstream/master' into fix-csv-date-infe…
sadikovi Jul 22, 2022
8a01ece
Merge remote-tracking branch 'upstream/master' into fix-csv-date-infe…
sadikovi Jul 24, 2022
fbdf9d8
update comments
sadikovi Jul 24, 2022
2962cd9
Revert "fix tests for SPARK-39469"
sadikovi Jul 24, 2022
10ca4a4
update the priority order for SPARK-39469
sadikovi Jul 24, 2022
b2a3db2
trigger ci
sadikovi Jul 25, 2022
739e7db
Merge remote-tracking branch 'upstream/master' into fix-csv-date-infe…
sadikovi Jul 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update comments
  • Loading branch information
@sadikovi
sadikovi committed Jul 24, 2022
commit fbdf9d8e7eca3deda0e3fbbbe047980dc7fff2db
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ class UnivocityParser(
} catch {
case NonFatal(e) =>
// If fails to parse, then tries the way used in 2.0 and 1.x for backwards
// compatibility only if no custom pattern has been set or parser policy is legacy.
// compatibility if enabled.
if (!enableParsingFallbackForTimestampType) {
throw e
}
Expand All @@ -235,7 +235,7 @@ class UnivocityParser(
} catch {
case NonFatal(e) =>
// If fails to parse, then tries the way used in 2.0 and 1.x for backwards
// compatibility only if no custom pattern has been set or parser policy is legacy.
// compatibility if enabled.
if (!enableParsingFallbackForDateType) {
throw e
}
Expand Down
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,7 @@ class JacksonParser(
} catch {
case NonFatal(e) =>
// If fails to parse, then tries the way used in 2.0 and 1.x for backwards
// compatibility only if no custom pattern has been set or parser policy is legacy.
// compatibility if enabled.
if (!enableParsingFallbackForTimestampType) {
throw e
}
Expand All @@ -297,7 +297,7 @@ class JacksonParser(
} catch {
case NonFatal(e) =>
// If fails to parse, then tries the way used in 2.0 and 1.x for backwards
// compatibility only if no custom pattern has been set or parser policy is legacy.
// compatibility if enabled.
if (!enableParsingFallbackForDateType) {
throw e
}
Expand Down