Skip to content

[SPARK-19261][SQL] Alter add columns for Hive serde and some datasource tables #16626

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
xwu0226 wants to merge 34 commits into from
Closed

[SPARK-19261][SQL] Alter add columns for Hive serde and some datasource tables #16626

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
52ca902
alter_add_col: initial changes
xwu0226 Nov 21, 2016
f498fa6
add testcases
xwu0226 Dec 1, 2016
522443e
negative testcases
xwu0226 Dec 1, 2016
1af2654
remove non-support testcase
xwu0226 Dec 5, 2016
ec57ee9
fix testcase
xwu0226 Dec 5, 2016
ec74849
update testcases
xwu0226 Dec 7, 2016
8fca889
update testcases
xwu0226 Dec 7, 2016
4a17529
update testcases
xwu0226 Jan 13, 2017
9699128
comments for command caseclass
xwu0226 Jan 20, 2017
9860e5c
udate comments based on review
xwu0226 Jan 21, 2017
dfff364
SPARK-19261: update to support datasource table and add new testcases
xwu0226 Feb 3, 2017
9f23254
remove workaournd for parquet issue since parquet-1.8.2 is now supported
xwu0226 Feb 4, 2017
180092f
SPARK-19261: using white list for datasource table types that support…
xwu0226 Feb 7, 2017
5a8aa80
fix code style
xwu0226 Feb 7, 2017
d3860e6
fix coding style
xwu0226 Feb 7, 2017
55577aa
update upon review
xwu0226 Feb 24, 2017
6fa913a
refactor code from alterTable function
xwu0226 Feb 25, 2017
7231efe
rebase and resolve conflict
xwu0226 Mar 6, 2017
e4e9ecf
resolve conflicts
xwu0226 Mar 9, 2017
75e7441
using ExternalCatalog.alterTableSchema
xwu0226 Mar 14, 2017
9847030
add InMemoryCatalog testcases
xwu0226 Mar 15, 2017
1a383bb
revert change in HiveExernalCatalog.scala
xwu0226 Mar 15, 2017
f994ce9
update upon review
xwu0226 Mar 16, 2017
5bf7360
add checking for duplicate column names
xwu0226 Mar 16, 2017
599c45e
add case sensativity for duplicate name checking and new testcases
xwu0226 Mar 16, 2017
b3edfea
typo
xwu0226 Mar 16, 2017
7d8a515
resolve conflicts and modify testcases
xwu0226 Mar 17, 2017
e895278
update testcases
xwu0226 Mar 17, 2017
e171ac4
move checkduplicate and schema arrangement to SessionCatalog.alterTab…
xwu0226 Mar 17, 2017
4391edd
change SessionCatalog.alterTableAddColumn back to alterTableSchema
xwu0226 Mar 18, 2017
a3fef12
update upon review comments
xwu0226 Mar 18, 2017
1eb7cd3
some minor updates upon review comments
xwu0226 Mar 19, 2017
04ce8f4
update based on review
xwu0226 Mar 21, 2017
7d8437d
update on minor comments
xwu0226 Mar 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
update on minor comments
  • Loading branch information
@xwu0226
xwu0226 committed Mar 21, 2017
commit 7d8437dbba76e1e88c9f230dc7ea895d343dceb8
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -344,10 +344,10 @@ class SessionCatalog(
}
Copy link
Member

@gatorsmile gatorsmile Mar 18, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let us output the column names.

    // not supporting dropping columns yet
    val nonExistentColumnNames = oldSchema.map(_.name).filterNot(isColumnNameResolved(newSchema, _))
    if (nonExistentColumnNames.nonEmpty) {
      throw new AnalysisException(
        s"""
          |Some existing schema fields (${nonExistentColumnNames.mkString("[", ",", "]")}) are
          |not present in the new schema. We don't support dropping columns yet.
         """.stripMargin)
    }


// assuming the newSchema has all partition columns at the end as required
externalCatalog.alterTableSchema(db, table, StructType(newSchema))
externalCatalog.alterTableSchema(db, table, newSchema)
}

def columnNameResolved(schema: StructType, colName: String): Boolean = {
private def columnNameResolved(schema: StructType, colName: String): Boolean = {
schema.fields.map(_.name).exists(conf.resolver(_, colName))
}
Copy link
Member

@gatorsmile gatorsmile Mar 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move these functions to Line 164


Expand Down
3 changes: 1 addition & 2 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,10 +455,9 @@ abstract class SessionCatalogSuite extends PlanTest {
withBasicCatalog { sessionCatalog =>
sessionCatalog.createTable(newTable("t1", "default"), ignoreIfExists = false)
val oldTab = sessionCatalog.externalCatalog.getTable("default", "t1")
val partitionSchema = oldTab.partitionSchema
sessionCatalog.alterTableSchema(
TableIdentifier("t1", Some("default")),
StructType(oldTab.dataSchema.add("c3", IntegerType) ++ partitionSchema))
StructType(oldTab.dataSchema.add("c3", IntegerType) ++ oldTab.partitionSchema))

val newTab = sessionCatalog.externalCatalog.getTable("default", "t1")
// construct the expected table schema
Expand Down