Skip to content

[SPARK-35105][SQL] Support multiple paths for ADD FILE/JAR/ARCHIVE commands #32205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sarutak wants to merge 7 commits into from
Closed

[SPARK-35105][SQL] Support multiple paths for ADD FILE/JAR/ARCHIVE commands #32205

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
627ba36
Add multiple arguments support for ADD FILES/JARS/ARCHIVES.
sarutak Apr 15, 2021
482b3c1
Update the migration guide.
sarutak Apr 16, 2021
8ef30c5
Update docs.
sarutak Apr 16, 2021
7cd0628
Add JIRA ID to the new tests.
sarutak Apr 16, 2021
002eaed
Merge branch 'master' of https://github.com/apache/spark into add-mul…
sarutak Apr 19, 2021
9fe3eee
Modify tests.
sarutak Apr 19, 2021
c7175de
Merge branch 'master' of https://github.com/apache/spark into add-mul…
sarutak Apr 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update the migration guide.
  • Loading branch information
@sarutak
sarutak committed Apr 16, 2021
commit 482b3c150a2bc1a560a6ae5709f2cc0d126eae25
2 changes: 2 additions & 0 deletions docs/sql-migration-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ license: |

## Upgrading from Spark SQL 3.1 to 3.2

- Since Spark 3.2, ADD FILE/JAR/ARCHIVE commands require each path to be enclosed by `"` or `'` if the path contains whitespaces.

- Since Spark 3.2, all the supported JDBC dialects use StringType for ROWID. In Spark 3.1 or earlier, Oracle dialect uses StringType and the other dialects use LongType.

- In Spark 3.2, PostgreSQL JDBC dialect uses StringType for MONEY and MONEY[] is not supported due to the JDBC driver for PostgreSQL can't handle those types properly. In Spark 3.1 or earlier, DoubleType and ArrayType of DoubleType are used respectively.
Expand Down