Skip to content

[SPARK-28389][SQL] Use Java 8 API in add_months #25153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 7 commits into from
Closed

[SPARK-28389][SQL] Use Java 8 API in add_months #25153

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
49061b9
Use Java 8 API to add months
MaxGekk Jul 11, 2019
93a16ae
Adjust tests
MaxGekk Jul 11, 2019
746de08
Remove unused methods
MaxGekk Jul 12, 2019
42d8e2d
Update the SQL migration guide
MaxGekk Jul 14, 2019
668608c
Merge remote-tracking branch 'origin/master' into add-months
MaxGekk Jul 14, 2019
5294204
Calc next elem as add_months(start, i * stepMonth, i * stepMicro)
MaxGekk Jul 14, 2019
f1a4241
Improve the SQL migration guide.
MaxGekk Jul 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update the SQL migration guide
  • Loading branch information
@MaxGekk
MaxGekk committed Jul 14, 2019
commit 42d8e2dd4324b1935cbb2aab066036043c57fd08
2 changes: 2 additions & 0 deletions docs/sql-migration-guide-upgrade.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,8 @@ license: |

- Since Spark 3.0, if files or subdirectories disappear during recursive directory listing (i.e. they appear in an intermediate listing but then cannot be read or listed during later phases of the recursive directory listing, due to either concurrent file deletions or object store consistency issues) then the listing will fail with an exception unless `spark.sql.files.ignoreMissingFiles` is `true` (default `false`). In previous versions, these missing files or subdirectories would be ignored. Note that this change of behavior only applies during initial table file listing (or during `REFRESH TABLE`), not during query execution: the net change is that `spark.sql.files.ignoreMissingFiles` is now obeyed during table file listing / query planning, not only at query execution time.

- Since Spark 3.0, the `add_months` function does not adjust the resulting date to a last day of month if the original date is a last day of month. The resulting date is adjust to a last day of month only if it is invalid. For example, `select add_months(DATE'2019-02-28', 1)` produces `2019-03-28` but `select add_months(DATE'2019-01-31', 1)` produces `2019-02-28`.
Copy link
Member

@srowen srowen Jul 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is adjust -> is adjusted.
So previously, adding a month to 2019-02-28 resulted in 2019-03-31? It might be worth clarifying that too in your example.

Copy link
Member Author

@MaxGekk MaxGekk Jul 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So previously, adding a month to 2019-02-28 resulted in 2019-03-31?

Yes, it does:

scala> spark.sql("select add_months(DATE'2019-02-28', 1)").show
+--------------------------------+
|add_months(DATE '2019-02-28', 1)|
+--------------------------------+
|                      2019-03-31|
+--------------------------------+

Copy link
Member Author

@MaxGekk MaxGekk Jul 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the SQL migration guide, and add the example.


## Upgrading from Spark SQL 2.4 to 2.4.1

- The value of `spark.executor.heartbeatInterval`, when specified without units like "30" rather than "30s", was
Expand Down