Skip to content

[SPARK-19115] [SQL] Supporting Create External Table Like Location #16638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ouyangxiaochen wants to merge 7 commits into from
Closed

[SPARK-19115] [SQL] Supporting Create External Table Like Location #16638

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
adde008
spark-19115
Jan 19, 2017
713ca97
update test cases and comments
Jan 22, 2017
b80f8e6
1. add a comment for method visitCreateTableLike in SparkSqlParser.sc…
Jan 24, 2017
71f1d12
1.remove EXTERNAL key word in sqlbase.g4 file
Feb 8, 2017
9e59fb4
update test cases
Feb 8, 2017
bb3660a
resolve the conflict
Feb 8, 2017
5dd21b2
update test cases
Feb 8, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
spark-19115
  • Loading branch information
ouyangxiaochen committed Jan 19, 2017
commit adde008588cf8e05cf261c086201c27a8dd5584f
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ statement
rowFormat? createFileFormat? locationSpec?
(TBLPROPERTIES tablePropertyList)?
(AS? query)? #createHiveTable
| CREATE TABLE (IF NOT EXISTS)? target=tableIdentifier
LIKE source=tableIdentifier #createTableLike
| CREATE EXTERNAL? TABLE (IF NOT EXISTS)? target=tableIdentifier
Copy link
Contributor

@cloud-fan cloud-fan Jan 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since Spark 2.2, we wanna hide the manage/external concept from users. It looks reasonable to add a LOCATION statement in CREATE TABLE LIKE, but do we really need the EXTERNAL keyword? We don't need to be exactly same with hive.

Copy link
Member

@gatorsmile gatorsmile Jan 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine

Copy link
Contributor

@cloud-fan cloud-fan Jan 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok then let's simplify the logic: if location is specified, we create an external table internally. Else, create managed table.

LIKE source=tableIdentifier locationSpec? #createTableLike
| ANALYZE TABLE tableIdentifier partitionSpec? COMPUTE STATISTICS
(identifier | FOR COLUMNS identifierSeq)? #analyze
| ALTER (TABLE | VIEW) from=tableIdentifier
Expand Down
10 changes: 7 additions & 3 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1140,14 +1140,18 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
*
* For example:
* {{{
* CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
* LIKE [other_db_name.]existing_table_name
* CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
* LIKE [other_db_name.]existing_table_name [locationSpec]
* }}}
*/
override def visitCreateTableLike(ctx: CreateTableLikeContext): LogicalPlan = withOrigin(ctx) {
val targetTable = visitTableIdentifier(ctx.target)
val sourceTable = visitTableIdentifier(ctx.source)
CreateTableLikeCommand(targetTable, sourceTable, ctx.EXISTS != null)
val location = Option(ctx.locationSpec).map(visitLocationSpec)
if (ctx.EXTERNAL != null && location.isEmpty) {
Copy link
Member

@gatorsmile gatorsmile Jan 24, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment above this line:

    // If we are creating an EXTERNAL table, then the LOCATION field is required

Copy link
Author

@ouyangxiaochen ouyangxiaochen Jan 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I'll do it later, Thanks!

operationNotAllowed("CREATE EXTERNAL TABLE LIKE must be accompanied by LOCATION", ctx)
}
Copy link
Member

@gatorsmile gatorsmile Jan 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To the other reviewers, we are following what we did in visitCreateHiveTable

CreateTableLikeCommand(targetTable, sourceTable, location, ctx.EXISTS != null)
}

/**
Expand Down
11 changes: 9 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ import org.apache.spark.util.Utils
case class CreateTableLikeCommand(
Copy link
Member

@gatorsmile gatorsmile Jan 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the comment of this class.

Copy link
Author

@ouyangxiaochen ouyangxiaochen Jan 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok,i will update it later,Thanks!

targetTable: TableIdentifier,
sourceTable: TableIdentifier,
location: Option[String],
ifNotExists: Boolean) extends RunnableCommand {

override def run(sparkSession: SparkSession): Seq[Row] = {
Expand All @@ -70,12 +71,18 @@ case class CreateTableLikeCommand(
sourceTableDesc.provider
}

val tblType = if (location.isEmpty) {
CatalogTableType.MANAGED
} else {
CatalogTableType.EXTERNAL
}

val newTableDesc =
CatalogTable(
identifier = targetTable,
tableType = CatalogTableType.MANAGED,
tableType = tblType,
// We are creating a new managed table, which should not have custom table location.
storage = sourceTableDesc.storage.copy(locationUri = None),
storage = sourceTableDesc.storage.copy(locationUri = location),
schema = sourceTableDesc.schema,
provider = newProvider,
partitionColumnNames = sourceTableDesc.partitionColumnNames,
Expand Down
8 changes: 4 additions & 4 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -518,8 +518,8 @@ class HiveDDLCommandSuite extends PlanTest with SQLTestUtils with TestHiveSingle

test("create table like") {
val v1 = "CREATE TABLE table1 LIKE table2"
val (target, source, exists) = parser.parsePlan(v1).collect {
case CreateTableLikeCommand(t, s, allowExisting) => (t, s, allowExisting)
val (target, source, location, exists) = parser.parsePlan(v1).collect {
Copy link
Contributor

@cloud-fan cloud-fan Feb 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add an assert to check location is empty

case CreateTableLikeCommand(t, s, l, allowExisting) => (t, s, l, allowExisting)
}.head
assert(exists == false)
assert(target.database.isEmpty)
Expand All @@ -528,8 +528,8 @@ class HiveDDLCommandSuite extends PlanTest with SQLTestUtils with TestHiveSingle
assert(source.table == "table2")

val v2 = "CREATE TABLE IF NOT EXISTS table1 LIKE table2"
Copy link
Contributor

@cloud-fan cloud-fan Feb 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add one more test case to check CREATE TABLE LIKE with location

val (target2, source2, exists2) = parser.parsePlan(v2).collect {
case CreateTableLikeCommand(t, s, allowExisting) => (t, s, allowExisting)
val (target2, source2, location2, exists2) = parser.parsePlan(v2).collect {
case CreateTableLikeCommand(t, s, l, allowExisting) => (t, s, l, allowExisting)
}.head
assert(exists2)
assert(target2.database.isEmpty)
Expand Down