Skip to content

Conversation

@HyukjinKwon
Copy link
Member

What changes were proposed in this pull request?

This PR removes the legacy workaround for JDK 8 in #28736.

Why are the changes needed?

  • We still need the main code for completeness, and in case there are other diff in the future JDK versions so this PR only fixes the tests.
  • We dropped JDK 11/8 at SPARK-44112

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Fixed unittests.

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions github-actions bot added the SQL label Sep 25, 2023
@HyukjinKwon
Copy link
Member Author

cc @yaooqinn

Copy link
Member

@yaooqinn yaooqinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

val micros2 = formatter.parse("2009-12-12 12 am")
assert(micros2 === date(2009, 12, 12, 12))
}
intercept[DateTimeException](formatter.parse("2009-12-12 12 am"))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @wangyum from 412d86e

Copy link
Contributor

@LuciferYang LuciferYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you fix the compilation issue (due to the unused import), @HyukjinKwon ?

[error] /Users/dongjoon/APACHE/spark-merge/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimestampFormatterSuite.scala:22:34: Unused import
[error] import org.apache.commons.lang3.{JavaVersion, SystemUtils}
[error]                                  ^
[error] /Users/dongjoon/APACHE/spark-merge/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimestampFormatterSuite.scala:22:47: Unused import
[error] import org.apache.commons.lang3.{JavaVersion, SystemUtils}
[error]                                               ^

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM.

@dongjoon-hyun
Copy link
Member

I manually verified this. Merged to master.

$ build/sbt "catalyst/testOnly *.TimestampFormatterSuite"
...
[info] TimestampFormatterSuite:
[info] - explicitly forbidden datetime patterns (355 milliseconds)
[info] - SPARK-31939: Fix Parsing day of year when year field pattern is missing (50 milliseconds)
[info] - parsing timestamps using time zones (2 milliseconds)
[info] - format timestamps using time zones (6 milliseconds)
[info] - roundtrip micros -> timestamp -> micros using timezones (22 milliseconds)
[info] - roundtrip timestamp -> micros -> timestamp using timezones (8 milliseconds)
[info] - case insensitive parsing of am and pm (5 milliseconds)
[info] - format fraction of second (1 millisecond)
[info] - formatting negative years with default pattern (3 milliseconds)
[info] - parsing timestamp strings with various seconds fractions (8 milliseconds)
[info] - formatting timestamp strings up to microsecond precision (4 milliseconds)
[info] - SPARK-30958: parse timestamp with negative year (2 milliseconds)
[info] - SPARK-31557: rebasing in legacy formatters/parsers (10 milliseconds)
[info] - parsing hour with various patterns (3 milliseconds)
[info] - missing date fields (0 milliseconds)
[info] - missing year field with invalid date (1 millisecond)
[info] - missing am/pm field (1 millisecond)
[info] - missing time fields (1 millisecond)
[info] - missing hour field (0 milliseconds)
[info] - check result differences for datetime formatting (0 milliseconds)
[info] - SPARK-32424: avoid silent data change when timestamp overflows (1 millisecond)
[info] - SPARK-36418: default parsing w/o pattern (5 milliseconds)
[info] - SPARK-39193: support returning optional parse results in the default formatter (2 milliseconds)
[info] - SPARK-39280: support returning optional parse results in the iso8601 formatter (0 milliseconds)
[info] - SPARK-39281: support returning optional parse results in the legacy formatter (0 milliseconds)
[info] Run completed in 1 second, 193 milliseconds.
[info] Total number of tests run: 25
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 25, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 27 s, completed Sep 25, 2023, 8:14:13 PM

@HyukjinKwon HyukjinKwon deleted the SPARK-45300 branch January 15, 2024 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants