Skip to content

[SPARK-31221][SQL] Rebase any date-times in conversions to/from Java types #27980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 11 commits into from
Closed

[SPARK-31221][SQL] Rebase any date-times in conversions to/from Java types #27980

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
821ffb5
Rebase java date for whole range
MaxGekk Mar 22, 2020
0b2dd01
Use localDateToDays()
MaxGekk Mar 22, 2020
5e2148d
Rebase any date-time
MaxGekk Mar 22, 2020
b466cac
Fix tests
MaxGekk Mar 22, 2020
dd25238
Re-use rebase functions from DateTimeUtils in DaysWritable
MaxGekk Mar 22, 2020
577303f
Remove unused const from DateTimeUtils
MaxGekk Mar 22, 2020
0d4bc10
Add comments
MaxGekk Mar 22, 2020
a6420f2
Add comments to fromJavaDate and to toJavaDate
MaxGekk Mar 22, 2020
db89c92
Add comments to toJavaTimestamp and to fromJavaTimestamp
MaxGekk Mar 22, 2020
6e8dba0
Use -08:00 instead of PST in a test
MaxGekk Mar 23, 2020
46b6613
Use America/Los_Angeles instead of PST in a test
MaxGekk Mar 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Re-use rebase functions from DateTimeUtils in DaysWritable
  • Loading branch information
@MaxGekk
MaxGekk committed Mar 22, 2020
commit dd252384fa5484a621df0e4518c6f67cfb6a7dd4
39 changes: 4 additions & 35 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import java.sql.Date
import org.apache.hadoop.hive.serde2.io.DateWritable
import org.apache.hadoop.io.WritableUtils

import org.apache.spark.sql.catalyst.util.DateTimeUtils
import org.apache.spark.sql.catalyst.util.DateTimeUtils.{rebaseGregorianToJulianDays, rebaseJulianToGregorianDays}

/**
* The class accepts/returns days in Gregorian calendar and rebase them
Expand All @@ -41,13 +41,13 @@ private[hive] class DaysWritable(
extends DateWritable {

def this(gregorianDays: Int) =
this(gregorianDays, DaysWritable.rebaseGregorianToJulianDays(gregorianDays))
this(gregorianDays, rebaseGregorianToJulianDays(gregorianDays))
def this(dateWritable: DateWritable) = {
this(
gregorianDays = dateWritable match {
case daysWritable: DaysWritable => daysWritable.gregorianDays
case dateWritable: DateWritable =>
DaysWritable.rebaseJulianToGregorianDays(dateWritable.getDays)
rebaseJulianToGregorianDays(dateWritable.getDays)
},
julianDays = dateWritable.getDays)
}
Expand All @@ -63,37 +63,6 @@ private[hive] class DaysWritable(
@throws[IOException]
override def readFields(in: DataInput): Unit = {
julianDays = WritableUtils.readVInt(in)
gregorianDays = DaysWritable.rebaseJulianToGregorianDays(julianDays)
gregorianDays = rebaseJulianToGregorianDays(julianDays)
}
}

private[hive] object DaysWritable {
// Rebasing days since the epoch to store the same number of days
// as by Spark 2.4 and earlier versions. Spark 3.0 switched to
// Proleptic Gregorian calendar (see SPARK-26651), and as a consequence of that,
// this affects dates before 1582-10-15. Spark 2.4 and earlier versions use
// Julian calendar for dates before 1582-10-15. So, the same local date may
// be mapped to different number of days since the epoch in different calendars.
// For example:
// Proleptic Gregorian calendar: 1582-01-01 -> -141714
// Julian calendar: 1582-01-01 -> -141704
// The code below converts -141714 to -141704.
def rebaseGregorianToJulianDays(daysSinceEpoch: Int): Int = {
if (daysSinceEpoch < DateTimeUtils.GREGORIAN_CUTOVER_DAY) {
DateTimeUtils.rebaseGregorianToJulianDays(daysSinceEpoch)
} else {
daysSinceEpoch
}
}

def rebaseJulianToGregorianDays(daysSinceEpoch: Int): Int = {
if (daysSinceEpoch < JULIAN_CUTOVER_DAY) {
DateTimeUtils.rebaseJulianToGregorianDays(daysSinceEpoch)
} else {
daysSinceEpoch
}
}

final val JULIAN_CUTOVER_DAY =
rebaseGregorianToJulianDays(DateTimeUtils.GREGORIAN_CUTOVER_DAY.toInt)
}