[SPARK-35204][SQL] CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes #32312
Conversation
…nd new Java time classes
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
| case l: LocalDate => DateTimeUtils.localDateToDays(l) | ||
| case other => throw new IllegalArgumentException( | ||
| s"The value (${other.toString}) of the type (${other.getClass.getCanonicalName}) " | ||
| + s"cannot be converted to the date type") |
There was a problem hiding this comment.
to the date type
How about to call a method of DateType to render type name.
| case i: Instant => DateTimeUtils.instantToMicros(i) | ||
| case other => throw new IllegalArgumentException( | ||
| s"The value (${other.toString}) of the type (${other.getClass.getCanonicalName}) " | ||
| + s"cannot be converted to the timestamp type") |
| val i = Instant.ofEpochSecond(seconds) | ||
| val result1 = converter(i) | ||
|
|
||
| val t = new java.sql.Timestamp(seconds * 1000) |
|
Test build #137857 has finished for PR 32312 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
+1, LGTM. GA passed. Merging to master. |
|
Test build #137865 has finished for PR 32312 at commit
|
3 participants
What changes were proposed in this pull request?
CatalystTypeConvertersis useful when the type of the input data classes are not known statically (otherwise we can useExpressionEncoder). However, the currentCatalystTypeConvertersrequires you to know the datetime data class statically, which makes it hard to use.This PR improves the
CatalystTypeConvertersfor date/timestamp, to support the old and new Java time classes at the same time.Why are the changes needed?
Make
CatalystTypeConverterseasier to use.Does this PR introduce any user-facing change?
No.
How was this patch tested?
new test