Skip to content

[SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType #26092

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 19 commits into from
Closed

[SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType #26092

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
0c45b1e
Add DURATION to Encoders
MaxGekk Oct 11, 2019
55ea764
Add DurationConverter
MaxGekk Oct 11, 2019
7e8e455
Add createDeserializerForLocalTime
MaxGekk Oct 11, 2019
3cdef53
Update InternalRow.getWriter
MaxGekk Oct 11, 2019
d5618fc
Add createSerializerForJavaDuration
MaxGekk Oct 11, 2019
e6dd5c8
Update ScalaReflection
MaxGekk Oct 11, 2019
fc275f3
Update row encoder
MaxGekk Oct 11, 2019
6f231e5
Add the test "converting java.time.Duration to CalendarIntervalType"
MaxGekk Oct 11, 2019
eb05207
Add tests to CatalystTypeConvertersSuite
MaxGekk Oct 11, 2019
0476082
Add tests to RowEncoderSuite
MaxGekk Oct 11, 2019
ce397ce
Move the row encoder test out of flag
MaxGekk Oct 11, 2019
6198a84
Add a test to LiteralExpressionSuite
MaxGekk Oct 11, 2019
78b9e58
Add more tests to LiteralExpressionSuite
MaxGekk Oct 11, 2019
114bbc7
Implicit newDurationEncoder
MaxGekk Oct 11, 2019
728c801
Add the test "implicit encoder for Duration"
MaxGekk Oct 11, 2019
9385336
Add a test to JavaDatasetSuite
MaxGekk Oct 11, 2019
568a162
Fix default test
MaxGekk Oct 11, 2019
c188351
UDF test
MaxGekk Oct 11, 2019
4aeb9b0
Add a test to JavaUDFSuite
MaxGekk Oct 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
UDF test
  • Loading branch information
@MaxGekk
MaxGekk committed Oct 11, 2019
commit c188351c774f8d028c61b50e780b1bd0d8f188a0
8 changes: 8 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,14 @@ class UDFSuite extends QueryTest with SharedSparkSession {
}
}

test("Using java.time.Duration in UDF") {
val expected = java.time.Duration.ofMinutes(40).plusDays(1)
val plusDay = udf((i: java.time.Duration) => i.plusDays(1))
val df = spark.sql("SELECT interval 40 minutes as i")
.select(plusDay('i))
assert(df.collect().toSeq === Seq(Row(expected)))
}

test("SPARK-28321 0-args Java UDF should not be called only once") {
val nonDeterministicJavaUDF = udf(
new UDF0[Int] {
Expand Down