Skip to content

[SPARK-4131] Support "Writing data into the filesystem from queries" #18975

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
janewangfb wants to merge 64 commits into from
Closed

[SPARK-4131] Support "Writing data into the filesystem from queries" #18975

Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
6ca7771
add insert overwrite local directory
janewangfb Aug 17, 2017
a975536
Add Unittests
janewangfb Aug 17, 2017
a15bf4e
fix local path
janewangfb Aug 17, 2017
9f596fd
Merge branch 'master' into port_local_directory
janewangfb Aug 17, 2017
b9db02e
fix style
janewangfb Aug 17, 2017
e516bec
Merge branch 'master' into port_local_directory
janewangfb Aug 18, 2017
e05624f
condense storage
janewangfb Aug 18, 2017
7f5664d
change InsertInto to InsertIntoTable
janewangfb Aug 18, 2017
d50b3a2
add InsertIntoDirectory
janewangfb Aug 19, 2017
61a18a2
update insertInto
janewangfb Aug 19, 2017
4c19aaf
SQLQuerySuite passed
janewangfb Aug 19, 2017
47fde8a
fix comments
janewangfb Aug 19, 2017
068662a
Merge branch 'master' into port_local_directory
janewangfb Aug 21, 2017
da7065b
Add tableProdier
janewangfb Aug 21, 2017
7f4b488
Add InsertIntoDataSourceDirCommand
janewangfb Aug 21, 2017
051018e
Merge branch 'master' into port_local_directory
janewangfb Aug 21, 2017
73f605e
fix style
janewangfb Aug 21, 2017
8261b39
fix style
janewangfb Aug 21, 2017
163c124
Merge branch 'master' into port_local_directory
janewangfb Aug 22, 2017
0882dd1
Address gatorsmile's comments
janewangfb Aug 22, 2017
c813ad8
Use withTempDir
janewangfb Aug 22, 2017
bc5424c
line length
janewangfb Aug 22, 2017
675ffd7
merge master
janewangfb Aug 23, 2017
f36e933
Address gatorsmile's comment
janewangfb Aug 23, 2017
64f37f4
fix typo
janewangfb Aug 23, 2017
b2068ce
Merge branch 'master' into port_local_directory
janewangfb Aug 31, 2017
8ebe5e2
Address gatorsmile's comments
janewangfb Aug 31, 2017
51f9a0a
fix comments and style
janewangfb Aug 31, 2017
e2db5e1
add more comments
janewangfb Aug 31, 2017
7bb5c85
Merge branch 'master' into port_local_directory
janewangfb Sep 1, 2017
2e7a29b
Merge branch 'master' into port_local_directory
janewangfb Sep 1, 2017
7ccbde4
Address gatorsmile's comments
janewangfb Sep 2, 2017
52350e8
fix style
janewangfb Sep 2, 2017
2ec9947
address Tejas' comment
janewangfb Sep 3, 2017
05d9d20
Merge branch 'master' into port_local_directory
janewangfb Sep 5, 2017
392593b
check point
janewangfb Sep 5, 2017
511cfc3
checkpoint
janewangfb Sep 5, 2017
77948bb
Merge insertIntoTable and insertIntoDirectory
janewangfb Sep 5, 2017
fd9322c
refactoring temp dir
janewangfb Sep 6, 2017
e590847
add TODO for tmpPath
janewangfb Sep 6, 2017
d1b4ec8
Merge branch 'master' into port_local_directory
janewangfb Sep 6, 2017
dd6a418
use tmpPath
janewangfb Sep 6, 2017
c2c693c
fix style
janewangfb Sep 6, 2017
e9c88b5
add exists check
janewangfb Sep 6, 2017
62370fd
Merge branch 'master' into port_local_directory
janewangfb Sep 6, 2017
b00acdc
Merge branch 'master' into port_local_directory
janewangfb Sep 6, 2017
b64520b
fix build
janewangfb Sep 6, 2017
3aaf6e8
fix build failure
janewangfb Sep 6, 2017
b461e00
fix build failure
janewangfb Sep 6, 2017
e9c24de
address's gatorsmile's comment
janewangfb Sep 6, 2017
28fcb39
Merge branch 'master' into port_local_directory
janewangfb Sep 6, 2017
0c03a2b
address gatorsmile's comment
janewangfb Sep 6, 2017
6c24b1b
remove unused import
janewangfb Sep 6, 2017
c4ab411
merge master
janewangfb Sep 7, 2017
0ec103e
address gatorsmile's comments
janewangfb Sep 7, 2017
160c0ec
fix style
janewangfb Sep 7, 2017
95ebfd3
add more unittest
janewangfb Sep 7, 2017
4a5ff29
add multi insert
janewangfb Sep 7, 2017
c99872b
Merge branch 'master' into port_local_directory
janewangfb Sep 9, 2017
449249e
address gatorsmile's comments
janewangfb Sep 9, 2017
7919041
address viirya's comment
janewangfb Sep 9, 2017
aeb5d5e
address viirya's comment
janewangfb Sep 9, 2017
81382df
Merge branch 'master' into port_local_directory
janewangfb Sep 9, 2017
f93d57a
address viirya's comment
janewangfb Sep 9, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address Tejas' comment
  • Loading branch information
@janewangfb
janewangfb committed Sep 3, 2017
commit 2ec9947af571691966e979e17aec12d3d683decf
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,9 +196,9 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
assert(ctx.children.size == 1)
Copy link
Contributor

@tejasapatil tejasapatil Sep 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious: what does this mean ?

Copy link
Contributor Author

@janewangfb janewangfb Sep 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it means for insert into can only specify one case, either
insert overwrite local directory path rowFormat....
or
insert overwrite local directiory path using....

Copy link
Member

@gatorsmile gatorsmile Sep 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a test case?

Copy link
Contributor

@tejasapatil tejasapatil Sep 3, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@janewangfb : thanks for the explanation. I got confused if this would have affected multi-inserts. ie.

FROM from_statement
INSERT OVERWRITE [LOCAL] DIRECTORY directory1 select_statement1
[INSERT OVERWRITE [LOCAL] DIRECTORY directory2 select_statement2] ...

One can even mix-in that with INSERT OVERWRITE TABLE ... in hive. You should have a test case for that.

Copy link
Member

@gatorsmile gatorsmile Sep 4, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be rewritten to

  private def withInsertInto(
      ctx: InsertIntoContext,
      query: LogicalPlan): LogicalPlan = withOrigin(ctx) {
    ctx match {
      case table: InsertIntoTableContext =>
        withInsertIntoTable(table, query)
      case dir: InsertOverwriteDirContext =>
        val (isLocal, storage, provider) = visitInsertOverwriteDir(dir)
        InsertIntoDir(isLocal, storage, provider, query, overwrite = true)
      case hiveDir: InsertOverwriteHiveDirContext =>
        val (isLocal, storage, provider) = visitInsertOverwriteHiveDir(hiveDir)
        InsertIntoDir(isLocal, storage, provider, query, overwrite = true)
    }
  }

Copy link
Contributor Author

@janewangfb janewangfb Sep 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gatorsmile, yes, that is more concise. Updated the code.
@tejasapatil added testcases.


ctx.getChild(0) match {
case c if c. isInstanceOf[InsertIntoTableContext] =>
case c : InsertIntoTableContext =>
withInsertIntoTable(c.asInstanceOf[InsertIntoTableContext], query)
case c if c.isInstanceOf[InsertOverwriteDirectoryContext] =>
case c : InsertOverwriteDirectoryContext =>
withInsertOverwriteDirectory(c.asInstanceOf[InsertOverwriteDirectoryContext], query)
}
}
Expand Down