Skip to content

[SPARK-31159][SQL] Rebase date/timestamp from/to Julian calendar in parquet #27915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 41 commits into from
Closed

[SPARK-31159][SQL] Rebase date/timestamp from/to Julian calendar in parquet #27915

Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
ae83fdc
Add rebaseGregorianToJulianMicros()
MaxGekk Mar 13, 2020
4c35ddb
Add rebaseJulianToGregorianMicros()
MaxGekk Mar 14, 2020
74774fc
Add a comment to rebaseJulianToGregorianMicros()
MaxGekk Mar 14, 2020
8d74214
Add comment about calendar
MaxGekk Mar 14, 2020
56ca744
Add round trip test for micros
MaxGekk Mar 14, 2020
0cfeed5
Rename to JULIAN_CUTOVER_MICROS
MaxGekk Mar 14, 2020
78cbd6c
Add rebaseJulianToGregorianDays and rebaseGregorianToJulianDays
MaxGekk Mar 14, 2020
13aad60
Minor fix code style
MaxGekk Mar 14, 2020
96573a9
Add comments for rebaseGregorianToJulianDays()
MaxGekk Mar 14, 2020
36c0400
Add the SQL config spark.sql.legacy.parquet.rebaseDateTime.enabled
MaxGekk Mar 14, 2020
f0a2df6
Perform rebase in write
MaxGekk Mar 14, 2020
9e3c201
Perform rebase dates in write
MaxGekk Mar 14, 2020
053861c
Perform rebase dates/timestamps in read
MaxGekk Mar 14, 2020
1624756
Rewrite days rebasing using Java 7 API
MaxGekk Mar 15, 2020
e3bbcb5
Rewrite micros rebasing using Java 7 API
MaxGekk Mar 15, 2020
d1e6d84
Extract common code
MaxGekk Mar 15, 2020
acd33f1
Revert "Extract common code"
MaxGekk Mar 16, 2020
41fc33f
Revert "Rewrite micros rebasing using Java 7 API"
MaxGekk Mar 16, 2020
fe9f130
Revert "Rewrite days rebasing using Java 7 API"
MaxGekk Mar 16, 2020
c2c53b8
Remove branching by cutover days in rebase functions
MaxGekk Mar 16, 2020
8e94359
Rebasing via system time zone
MaxGekk Mar 16, 2020
d6f7e6b
Rebase dates via UTC local date
MaxGekk Mar 16, 2020
81d342a
Check more time zones in days rebasing
MaxGekk Mar 16, 2020
63428ab
More dates/timestamps for testing
MaxGekk Mar 16, 2020
a34a9ce
Rename utcCal to cal
MaxGekk Mar 16, 2020
e590d36
Test multiple time zones in rebasing timestamps
MaxGekk Mar 16, 2020
262f744
Test reading parquet files written by Spark 2.4
MaxGekk Mar 16, 2020
8947298
Remove .asInstanceOf[DateType#InternalType]
MaxGekk Mar 16, 2020
276d159
Change SQL config description
MaxGekk Mar 16, 2020
167b463
Rebase timestamp INT96
MaxGekk Mar 16, 2020
bbc4a1a
Support rebasing in VectorizedColumnReader
MaxGekk Mar 16, 2020
a1b34cb
Bug fix in write
MaxGekk Mar 16, 2020
d7debb4
Add test for write
MaxGekk Mar 16, 2020
6bebf3b
Read SQL config in place
MaxGekk Mar 17, 2020
67cec02
Remove a gap
MaxGekk Mar 17, 2020
8fa19a6
Remove config settings from ParquetWriteBuilder
MaxGekk Mar 17, 2020
a061870
Initialize rebaseDateTime in default constructor in ParquetWriteSupport
MaxGekk Mar 18, 2020
ae49cc4
Check INT96 rebasing regardless of SQL config settings
MaxGekk Mar 18, 2020
a96392c
Add JIRA id
MaxGekk Mar 18, 2020
5b52735
Test INT96 w/ and w/o vectorized reader
MaxGekk Mar 18, 2020
184fcd8
Merge remote-tracking branch 'remotes/origin/master' into rebase-parq…
MaxGekk Mar 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Rebase dates via UTC local date
  • Loading branch information
@MaxGekk
MaxGekk committed Mar 16, 2020
commit d6f7e6b4f37b47d6ae7a0c2d5de60189ae851774
5 changes: 2 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,6 @@ object DateTimeUtils {
.atOffset(ZoneOffset.UTC)
.toInstant)
final val GREGORIAN_CUTOVER_MILLIS = microsToMillis(GREGORIAN_CUTOVER_MICROS)
final val JULIAN_CUTOVER_MICROS = rebaseGregorianToJulianMicros(GREGORIAN_CUTOVER_MICROS)
final val JULIAN_CUTOVER_DAY =
rebaseGregorianToJulianDays(DateTimeUtils.GREGORIAN_CUTOVER_DAY.toInt)

final val julianCommonEraStart = Timestamp.valueOf("0001-01-01 00:00:00")

Expand Down Expand Up @@ -1038,6 +1035,7 @@ object DateTimeUtils {
// `gregory` is a hybrid calendar that supports both
// the Julian and Gregorian calendar systems
.setCalendarType("gregory")
.setTimeZone(TimeZoneUTC)
Copy link
Member Author

@MaxGekk MaxGekk Mar 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use particular time zone here because the conversion of "logical" days is independent from time zone, actually. UTC is selected to avoid the problem of rounding micros to/from days because zone offset in UTC is 0.

.setInstant(Math.multiplyExact(days, MILLIS_PER_DAY))
.build()
val localDate = LocalDate.of(
Expand Down Expand Up @@ -1069,6 +1067,7 @@ object DateTimeUtils {
// `gregory` is a hybrid calendar that supports both
// the Julian and Gregorian calendar systems
.setCalendarType("gregory")
.setTimeZone(TimeZoneUTC)
.setDate(localDate.getYear, localDate.getMonthValue - 1, localDate.getDayOfMonth)
.build()
Math.toIntExact(Math.floorDiv(utcCal.getTimeInMillis, MILLIS_PER_DAY))
Expand Down