[SPARK-29174][SQL] Support LOCAL in INSERT OVERWRITE DIRECTORY to data source #27039

ajithme · 2019-12-29T07:23:53Z

What changes were proposed in this pull request?

INSERT OVERWRITE LOCAL DIRECTORY is supported with ensuring the provided path is always using file:// as scheme and removing the check which throws exception if we do insert overwrite by mentioning directory with LOCAL syntax

Why are the changes needed?

without the modification in PR, insert overwrite local directory <location> using

throws exception

Error: org.apache.spark.sql.catalyst.parser.ParseException:

LOCAL is not supported in INSERT OVERWRITE DIRECTORY to data source(line 1, pos 0)

which was introduced in #18975, but this restriction is not needed, hence dropping the same.
Keep behaviour consistent for local and remote file-system in INSERT OVERWRITE DIRECTORY

Does this PR introduce any user-facing change?

Yes, after this change INSERT OVERWRITE LOCAL DIRECTORY will not throw exception

How was this patch tested?

Added UT

ajithme · 2019-12-29T07:25:04Z

cc @srowen @dongjoon-hyun @gatorsmile @cloud-fan

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

HyukjinKwon · 2019-12-30T09:20:11Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

lol .. I didn't know there's such way.

HyukjinKwon · 2019-12-30T09:20:23Z

ok to test

SparkQA · 2019-12-30T11:58:00Z

Test build #115948 has finished for PR 27039 at commit 8917ef7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ajithme · 2019-12-30T14:02:42Z

retest this please

SparkQA · 2019-12-30T17:58:25Z

Test build #115960 has finished for PR 27039 at commit f7486a8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajithme · 2020-01-08T06:26:46Z

gentle ping @HyukjinKwon @dongjoon-hyun

dongjoon-hyun · 2020-01-10T20:10:28Z

Hi, @ajithme . Could you update your PR description? It seems that you always put some content before the first section, but it's not an Apache Spark's recommendation. You need to move all content before What changes were proposed in this pull request? into somewhere below.

ajithme · 2020-01-10T20:36:02Z

Hi, @ajithme . Could you update your PR description? It seems that you always put some content before the first section, but it's not an Apache Spark's recommendation. You need to move all content before What changes were proposed in this pull request? into somewhere below.

@dongjoon-hyun will ensure this, thanks for pointing out. I have updated the PR description

HyukjinKwon · 2020-01-17T03:54:59Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

@ajithme, Can you double check how Hive works if other scheme are given? The current code seems always forcing to local scheme, e.g., hdfs:/a/b/c will becomes file:/a/b/c. Does Hive works in this way too?

Sure, i checked in Hive 2.3.5, it throws a exception if any other scheme is used apart form file:

success scenario

0: jdbc:hive2://QWE:10000/> insert overwrite local directory 'file:/tmp/sample' select 1; No rows affected (1.436 seconds) 0: jdbc:hive2://QWE:10000/> insert overwrite local directory '/tmp/sample' select 1; No rows affected (1.474 seconds)

fail scenario

Below is the stack from hive server on client

0: jdbc:hive2://QWE:10000/> insert overwrite local directory 'hdfs:///tmp/sample' select 1; WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source file:/tmp/root/28572e2b-63d0-49fd-9938-85b9741c7413/hive_2020-01-17_14-32-02_389_528060968304222996-1/-mr-10000 to destination hdfs:/tmp/sample at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257) at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source file:/tmp/root/28572e2b-63d0-49fd-9938-85b9741c7413/hive_2020-01-17_14-32-02_389_528060968304222996-1/-mr-10000 to destination hdfs:/tmp/sample at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:104) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:259) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1232) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:255) ... 11 more Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs:/tmp/sample, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:665) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86) at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:630) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:861) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:625) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1437) at org.apache.hadoop.hive.ql.exec.MoveTask.moveFileFromDfsToLocal(MoveTask.java:143) at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:101) ... 20 more (state=08S01,code=1)

@HyukjinKwon Can i update the PR instead to follow Hive behavior i.e if a non local scheme is mentioned, the command fails.?

Updated PR to align behaviour with Hive

SparkQA · 2020-01-17T16:35:50Z

Test build #116943 has finished for PR 27039 at commit 6dfe0dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajithme · 2020-01-27T08:27:16Z

gentle ping @HyukjinKwon

HyukjinKwon · 2020-02-17T08:23:47Z

retest this please

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

SparkQA · 2020-02-17T11:39:20Z

Test build #118573 has finished for PR 27039 at commit 77f6b8e.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

ajithme · 2020-02-17T11:46:23Z

The doc generation error seems unrelated

HyukjinKwon · 2020-02-17T11:47:44Z

retest this please

SparkQA · 2020-02-17T12:11:54Z

Test build #118575 has finished for PR 27039 at commit 77f6b8e.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-17T13:13:06Z

Test build #118557 has finished for PR 27039 at commit 6dfe0dc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…directory

ajithme · 2020-02-17T13:14:56Z

Rebased to latest master. Retest this please

SparkQA · 2020-02-17T19:01:39Z

Test build #118584 has finished for PR 27039 at commit 6ddf84a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-02-18T00:41:54Z

Merged to master.

…a source ### What changes were proposed in this pull request? `INSERT OVERWRITE LOCAL DIRECTORY` is supported with ensuring the provided path is always using `file://` as scheme and removing the check which throws exception if we do insert overwrite by mentioning directory with `LOCAL` syntax ### Why are the changes needed? without the modification in PR, ``` insert overwrite local directory <location> using ``` throws exception ``` Error: org.apache.spark.sql.catalyst.parser.ParseException: LOCAL is not supported in INSERT OVERWRITE DIRECTORY to data source(line 1, pos 0) ``` which was introduced in apache#18975, but this restriction is not needed, hence dropping the same. Keep behaviour consistent for local and remote file-system in `INSERT OVERWRITE DIRECTORY` ### Does this PR introduce any user-facing change? Yes, after this change `INSERT OVERWRITE LOCAL DIRECTORY` will not throw exception ### How was this patch tested? Added UT Closes apache#27039 from ajithme/insertoverwrite2. Authored-by: Ajith <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

HyukjinKwon reviewed Dec 30, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Dec 30, 2019

View reviewed changes

ajithme requested a review from HyukjinKwon January 10, 2020 06:15

dongjoon-hyun added the SQL label Jan 10, 2020

dongjoon-hyun changed the title ~~[SPARK-29174][SQL] LOCAL is not supported in INSERT OVERWRITE DIRECTORY to data source~~ [SPARK-29174][SQL] Support LOCAL in INSERT OVERWRITE DIRECTORY to data source Jan 10, 2020

HyukjinKwon reviewed Jan 17, 2020

View reviewed changes

ajithme force-pushed the insertoverwrite2 branch from f7486a8 to 6dfe0dc Compare January 17, 2020 12:31

HyukjinKwon reviewed Feb 17, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala Outdated Show resolved Hide resolved

HyukjinKwon approved these changes Feb 17, 2020

View reviewed changes

ajithme added 5 commits February 17, 2020 18:43

Support LOCAL keyword in INSERT OVERWRITE DIRECTORY

aefcdf3

Force location scheme to be file:// incase of insert overwrite local …

37caeba

…directory

Fix testcase

a18da2e

Align with Hive behaviour

9e849d1

Update the comment

6ddf84a

ajithme force-pushed the insertoverwrite2 branch from 77f6b8e to 6ddf84a Compare February 17, 2020 13:14

HyukjinKwon closed this in 657d151 Feb 18, 2020

[SPARK-29174][SQL] Support LOCAL in INSERT OVERWRITE DIRECTORY to data source #27039

[SPARK-29174][SQL] Support LOCAL in INSERT OVERWRITE DIRECTORY to data source #27039

Uh oh!

Conversation

ajithme commented Dec 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

ajithme commented Dec 29, 2019

Uh oh!

Uh oh!

HyukjinKwon Dec 30, 2019

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Dec 30, 2019

Uh oh!

SparkQA commented Dec 30, 2019

Uh oh!

ajithme commented Dec 30, 2019

Uh oh!

SparkQA commented Dec 30, 2019

Uh oh!

ajithme commented Jan 8, 2020

Uh oh!

dongjoon-hyun commented Jan 10, 2020

Uh oh!

ajithme commented Jan 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon Jan 17, 2020

Choose a reason for hiding this comment

Uh oh!

ajithme Jan 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

success scenario

fail scenario

Uh oh!

ajithme Jan 17, 2020

Choose a reason for hiding this comment

Uh oh!

ajithme Jan 17, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 17, 2020

Uh oh!

ajithme commented Jan 27, 2020

Uh oh!

HyukjinKwon commented Feb 17, 2020

Uh oh!

Uh oh!

SparkQA commented Feb 17, 2020

Uh oh!

ajithme commented Feb 17, 2020

Uh oh!

HyukjinKwon commented Feb 17, 2020

Uh oh!

SparkQA commented Feb 17, 2020

Uh oh!

SparkQA commented Feb 17, 2020

Uh oh!

ajithme commented Feb 17, 2020

Uh oh!

SparkQA commented Feb 17, 2020

Uh oh!

HyukjinKwon commented Feb 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ajithme commented Dec 29, 2019 •

edited

Loading

ajithme commented Jan 10, 2020 •

edited

Loading

ajithme Jan 17, 2020 •

edited

Loading