[SPARK-3907][sql] add truncate table support #2770

wangxiaojing · 2014-10-11T16:14:57Z

JIRA issue: [SPARK-3907]https://issues.apache.org/jira/browse/SPARK-3907

Add turncate table support
TRUNCATE TABLE table_name [PARTITION partition_spec];
partition_spec:
: (partition_col = partition_col_value, partition_col = partiton_col_value, ...)
Removes all rows from a table or partition(s). Currently target table should be native/managed table or exception will be thrown. User can specify partial partition_spec for truncating multiple partitions at once and omitting partition_spec will truncate all partitions in the table.

AmplabJenkins · 2014-10-11T16:17:07Z

Can one of the admins verify this patch?

AmplabJenkins · 2014-10-11T16:17:11Z

Can one of the admins verify this patch?

rxin · 2014-10-12T19:57:01Z

Jenkins, test this please.

SparkQA · 2014-10-12T19:59:40Z

QA tests have started for PR 2770 at commit e483547.

This patch merges cleanly.

SparkQA · 2014-10-12T20:30:00Z

QA tests have finished for PR 2770 at commit e483547.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-12T20:30:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21673/
Test FAILed.

adrian-wang · 2014-10-14T05:47:11Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala

“tabel” is a typo. I think it's better not to include this part here, which would make the code not very clean. You can remove this part and fix truncate column in another PR, or fix that in this PR.

SparkQA · 2014-10-16T14:30:08Z

QA tests have started for PR 2770 at commit e483547.

This patch merges cleanly.

SparkQA · 2014-10-16T15:02:44Z

QA tests have finished for PR 2770 at commit e483547.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangxiaojing · 2014-10-17T04:52:46Z

@rxin

adrian-wang · 2014-10-17T05:00:35Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala

why is truncate table a no-explain-command?

@adrian-wang I think explain for truncate table is meaningless.

I think we should add that in nativeCommands

If add truncate table in nativeCommands,the sql truncate table test columns(i) will run hive mapreduce.At present, the columns way only support rcfile in hive.so ,we must put truncate table in nodeToPlan,and truncate table name columns(i)return anomalies.

Can you add a comment below explaining this?

SparkQA · 2014-10-20T00:25:53Z

QA tests have started for PR 2770 at commit f6e710e.

This patch merges cleanly.

SparkQA · 2014-10-20T01:00:45Z

QA tests have finished for PR 2770 at commit f6e710e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2014-10-24T01:14:41Z

test this please

SparkQA · 2014-10-24T01:19:57Z

QA tests have started for PR 2770 at commit f6e710e.

This patch merges cleanly.

SparkQA · 2014-10-24T01:55:40Z

QA tests have finished for PR 2770 at commit f6e710e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-24T01:55:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22102/
Test FAILed.

liancheng · 2014-10-24T02:08:04Z

retest this please

SparkQA · 2014-10-24T02:15:05Z

QA tests have started for PR 2770 at commit f6e710e.

This patch merges cleanly.

SparkQA · 2014-10-24T03:07:25Z

QA tests have finished for PR 2770 at commit f6e710e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-24T03:07:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22104/
Test PASSed.

SparkQA · 2014-10-25T08:09:00Z

Test build #442 has started for PR 2770 at commit f6e710e.

This patch merges cleanly.

SparkQA · 2014-10-25T08:13:46Z

Test build #442 has finished for PR 2770 at commit f6e710e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-10-26T22:35:41Z

Test build #468 has started for PR 2770 at commit 7a03707.

This patch merges cleanly.

SparkQA · 2014-10-26T22:37:06Z

Test build #468 has finished for PR 2770 at commit 7a03707.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-10-26T23:09:47Z

Test build #470 has started for PR 2770 at commit 7a03707.

This patch merges cleanly.

SparkQA · 2014-10-26T23:10:58Z

Test build #470 has finished for PR 2770 at commit 7a03707.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2014-10-26T23:34:49Z

It appears there are some style violations:

[marmbrus@michaels-mbp spark (pr/2770)]$ sbt hive/scalastyle
[info] Set current project to spark-parent (in build file:/Users/marmbrus/workspace/spark/)
error file=/Users/marmbrus/workspace/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala message=File line length exceeds 100 characters line=477
error file=/Users/marmbrus/workspace/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala message=insert.a.single.space.after.comment.start.and.before.end.message line=125 column=4
error file=/Users/marmbrus/workspace/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala message=insert.a.single.space.after.comment.start.and.before.end.message line=477 column=4

marmbrus · 2014-10-28T03:52:09Z

ok to test

marmbrus · 2014-10-28T03:53:13Z

Can you please make the title SPARK-XXXX so that the merge script recognizes the JIRA?

SparkQA · 2014-10-28T04:00:04Z

Test build #22331 has started for PR 2770 at commit 63dbd81.

This patch merges cleanly.

SparkQA · 2014-10-28T04:53:14Z

Test build #22331 has finished for PR 2770 at commit 63dbd81.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-28T04:53:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22331/
Test PASSed.

marmbrus · 2014-10-28T05:04:34Z

Thanks! Merged to master.

OopsOutOfMemory · 2014-11-04T16:10:02Z

@wangxiaojing @tianyi @marmbrus
Hi, wangxiaojing, I have an issues for this PR.
Currently the truncate table works fine in the situation that current table is in the corresponding database session. But it doesn't work in the scenario which the table is not in the corresponding database session.
What I mean is :
Assume we have two database:default, dw. A table named test_table in database dw
By default we login as default database session. So I run：

use dw;
truncate table test_table [partions......];  is OK.

If I just use the default database default to run:

use default;
truncate table dw.test_table;

It will throw exception

Failed to parse: truncate table dw.test_table.
 line 1:17 missing EOF at '.' near 'dw'

It's a bug when parsing the truncate table xxx
It would be nicer to fix it : )

marmbrus · 2014-11-04T19:38:01Z

@OopsOutOfMemory please file a JIRA.

wangxiaojing · 2014-11-05T01:48:04Z

@OopsOutOfMemory Thanks a lot!
This bug is also found in the hive.
eg: TRUNCATE TABLE default.test; FAILED: ParseException line 1:22 missing EOF at '.' near 'default'

OopsOutOfMemory · 2014-11-05T01:55:18Z

Yep, that is what I guess yesterday.
I think we should support all the commands which are similar to this.
The JIRA for this one locate in below:
https://issues.apache.org/jira/browse/SPARK-4232 .

add truncate table support

77b1f20

add golden file

e483547

add truncate can not support column err message

3b20007

adrian-wang reviewed Oct 14, 2014
View reviewed changes

wangxiaojing added 2 commits October 13, 2014 22:50

Correct spelling mistakes

a1f692c

change truncate table

f6e710e

adrian-wang reviewed Oct 17, 2014
View reviewed changes

add comment

7a03707

change hive scalastyle

63dbd81

wangxiaojing changed the title ~~[spark-3907][sql] add truncate table support~~ [SPARK-3907][sql] add truncate table support Oct 28, 2014

asfgit closed this in 0c34fa5 Oct 28, 2014

[SPARK-3907][sql] add truncate table support #2770

[SPARK-3907][sql] add truncate table support #2770

Uh oh!

Conversation

wangxiaojing commented Oct 11, 2014

Uh oh!

AmplabJenkins commented Oct 11, 2014

Uh oh!

AmplabJenkins commented Oct 11, 2014

Uh oh!

rxin commented Oct 12, 2014

Uh oh!

SparkQA commented Oct 12, 2014

Uh oh!

SparkQA commented Oct 12, 2014

Uh oh!

AmplabJenkins commented Oct 12, 2014

Uh oh!

adrian-wang Oct 14, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 16, 2014

Uh oh!

SparkQA commented Oct 16, 2014

Uh oh!

wangxiaojing commented Oct 17, 2014

Uh oh!

adrian-wang Oct 17, 2014

Choose a reason for hiding this comment

Uh oh!

tianyi Oct 17, 2014

Choose a reason for hiding this comment

Uh oh!

adrian-wang Oct 17, 2014

Choose a reason for hiding this comment

Uh oh!

wangxiaojing Oct 17, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus Oct 24, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 20, 2014

Uh oh!

SparkQA commented Oct 20, 2014

Uh oh!

liancheng commented Oct 24, 2014

Uh oh!

SparkQA commented Oct 24, 2014

Uh oh!

SparkQA commented Oct 24, 2014

Uh oh!

AmplabJenkins commented Oct 24, 2014

Uh oh!

liancheng commented Oct 24, 2014

Uh oh!

SparkQA commented Oct 24, 2014

Uh oh!

SparkQA commented Oct 24, 2014

Uh oh!

AmplabJenkins commented Oct 24, 2014

Uh oh!

SparkQA commented Oct 25, 2014

Uh oh!

SparkQA commented Oct 25, 2014

Uh oh!

SparkQA commented Oct 26, 2014

Uh oh!

SparkQA commented Oct 26, 2014

Uh oh!

SparkQA commented Oct 26, 2014

Uh oh!

SparkQA commented Oct 26, 2014

Uh oh!

marmbrus commented Oct 26, 2014

Uh oh!

marmbrus commented Oct 28, 2014

Uh oh!

marmbrus commented Oct 28, 2014

Uh oh!