[SPARK-34605][SQL] Support `java.time.Duration` as an external type of the day-time interval type #31729

MaxGekk · 2021-03-03T20:44:02Z

What changes were proposed in this pull request?

In the PR, I propose to extend Spark SQL API to accept java.time.Duration as an external type of recently added new Catalyst type - DayTimeIntervalType (see #31614). The Java class java.time.Duration has similar semantic to ANSI SQL day-time interval type, and it is the most suitable to be an external type for DayTimeIntervalType. In more details:

Added DurationConverter which converts java.time.Duration instances to/from internal representation of the Catalyst type DayTimeIntervalType (to Long type). The DurationConverter object uses new methods of IntervalUtils:
- durationToMicros() converts the input duration to the total length in microseconds. If this duration is too large to fit Long, the method throws the exception ArithmeticException. Note: the input duration has nanosecond precision, the method casts the nanos part to microseconds by dividing by 1000.
- microsToDuration() obtains a java.time.Duration representing a number of microseconds.
Support new type DayTimeIntervalType in RowEncoder via the methods createDeserializerForDuration() and createSerializerForJavaDuration().
Extended the Literal API to construct literals from java.time.Duration instances.

Why are the changes needed?

To allow users parallelization of java.time.Duration collections, and construct day-time interval columns. Also to collect such columns back to the driver side.
This will allow to write tests in other sub-tasks of SPARK-27790.

Does this PR introduce any user-facing change?

The PR extends existing functionality. So, users can parallelize instances of the java.time.Duration class and collect them back:

scala> val ds = Seq(java.time.Duration.ofDays(10)).toDS
ds: org.apache.spark.sql.Dataset[java.time.Duration] = [value: daytimeinterval]

scala> ds.collect
res0: Array[java.time.Duration] = Array(PT240H)

How was this patch tested?

Added a few tests to CatalystTypeConvertersSuite to check conversion from/to java.time.Duration.
Checking row encoding by new tests in RowEncoderSuite.
Making literals of DayTimeIntervalType are tested in LiteralExpressionSuite
Check collecting by DatasetSuite and JavaDatasetSuite.

SparkQA · 2021-03-03T21:54:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40303/

SparkQA · 2021-03-03T22:21:22Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40303/

SparkQA · 2021-03-04T01:48:31Z

Test build #135721 has finished for PR 31729 at commit b5b021e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2021-03-04T06:06:48Z

@cloud-fan @HyukjinKwon @dongjoon-hyun @viirya @maropu @yaooqinn Could you review this PR, please.

yaooqinn · 2021-03-04T06:15:35Z

I believe that Duration fits ANSI day-time interval well. But not related directly to this PR, what will we use for the year-month?

MaxGekk · 2021-03-04T06:19:32Z

what will we use for the year-month?

@yaooqinn I think of java.time.Period (https://docs.oracle.com/javase/8/docs/api/java/time/Period.html)

yaooqinn · 2021-03-04T06:27:11Z

what will we use for the year-month?

@yaooqinn I think of java.time.Period (https://docs.oracle.com/javase/8/docs/api/java/time/Period.html)

Does it mean that we only use the period with year and month filed and fail for the day part?

BTW, what is the behavior of Dataframe.show(), will it be something like PT1H2H3S

MaxGekk · 2021-03-04T06:38:14Z

Does it mean that we only use the period with year and month filed ...

Yes, we will use only year and month fields.

and fail for the day part?

Let's discuss this in https://issues.apache.org/jira/browse/SPARK-34615

BTW, what is the behavior of Dataframe.show(), will it be something like PT1H2H3S

No-no. It will be in the format that conforms to the ANSI SQL standard like:

INTERVAL ‘12 45:20:59.123’ DAY TO SECOND

BTW, the format PT1H2H3S is from the ISO-8601 standard.

cloud-fan · 2021-03-04T07:05:01Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala

+    Seq(
+      "PT20.123456S",
+      "P2DT3H4M",
+      "P2D").foreach { intervalStr =>


I think many people are not familiar with this format. Can we add comments or use a different way to create Duration?

Cam we add some negative cases?

@cloud-fan I created durations using another Duration methods. Also added negative tests.

cloud-fan · 2021-03-04T07:05:16Z

...talyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala

+  }
+
+  test("SPARK-34605: construct literals from arrays of java.time.Duration") {
+    val duration0 = Duration.parse("P2DT3H4M")


yaooqinn · 2021-03-04T07:40:44Z

It's better we update the sql-ref-datatypes.md within this PR or create a separate JIRA to do later.

MaxGekk · 2021-03-04T11:00:36Z

It's better we update the sql-ref-datatypes.md

@yaooqinn Let's update docs separately. I opened the sub-task SPARK-34619 for that.

SparkQA · 2021-03-04T11:48:59Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40335/

SparkQA · 2021-03-04T11:57:29Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40335/

SparkQA · 2021-03-04T15:54:19Z

Test build #135752 has finished for PR 31729 at commit b826afd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-03-04T16:58:25Z

thanks, merging to master!

Support java.time.Duration as an external type of DayTimeIntervalType

b5b021e

github-actions bot added the SQL label Mar 3, 2021

MaxGekk changed the title ~~[WIP][SPARK-34605][SQL] Support java.time.Duration as an external type of the day-time interval type~~ [SPARK-34605][SQL] Support java.time.Duration as an external type of the day-time interval type Mar 4, 2021

cloud-fan reviewed Mar 4, 2021

View reviewed changes

MaxGekk added 3 commits March 4, 2021 13:33

Add tests to IntervalUtilsSuite

737a55f

Add tests to CatalystTypeConvertersSuite

bb0f6cd

Add tests to LiteralExpressionSuite

b826afd

cloud-fan approved these changes Mar 4, 2021

View reviewed changes

cloud-fan closed this in 17601e0 Mar 4, 2021

MaxGekk mentioned this pull request Jun 8, 2021

[SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type #32814

Closed

[SPARK-34605][SQL] Support java.time.Duration as an external type of the day-time interval type #31729

[SPARK-34605][SQL] Support java.time.Duration as an external type of the day-time interval type #31729

Uh oh!

Conversation

MaxGekk commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Mar 3, 2021

Uh oh!

SparkQA commented Mar 3, 2021

Uh oh!

SparkQA commented Mar 4, 2021

Uh oh!

MaxGekk commented Mar 4, 2021

Uh oh!

yaooqinn commented Mar 4, 2021

Uh oh!

MaxGekk commented Mar 4, 2021

Uh oh!

yaooqinn commented Mar 4, 2021

Uh oh!

MaxGekk commented Mar 4, 2021

Uh oh!

cloud-fan Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

yaooqinn Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

MaxGekk Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

cloud-fan Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

yaooqinn commented Mar 4, 2021

Uh oh!

MaxGekk commented Mar 4, 2021

Uh oh!

SparkQA commented Mar 4, 2021

Uh oh!

SparkQA commented Mar 4, 2021

Uh oh!

SparkQA commented Mar 4, 2021

Uh oh!

cloud-fan commented Mar 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[SPARK-34605][SQL] Support `java.time.Duration` as an external type of the day-time interval type #31729

[SPARK-34605][SQL] Support `java.time.Duration` as an external type of the day-time interval type #31729

MaxGekk commented Mar 3, 2021 •

edited

Loading