Skip to content

[SPARK-26388][SQL][WIP]_implement_hive_ddl_alter_table_replace_columns #24780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
emanuelebardelli wants to merge 13 commits into from
Closed

[SPARK-26388][SQL][WIP]_implement_hive_ddl_alter_table_replace_columns #24780

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a99e362
enable 'alter table replace columns' command in sql parser
manu-olx May 29, 2019
b20dc95
create abstract class for add/replace columns classes
manu-olx May 29, 2019
4f52b94
extract method verifying feasibility of add/replace columns operation…
manu-olx May 29, 2019
59332d0
add AlterTableReplaceColumnsCommand class
manu-olx May 29, 2019
01ce36c
fix failing tests
manu-olx May 31, 2019
8ba07c9
refactor basic alter table add/replace tests
manu-olx Jun 3, 2019
e680632
remove safety net preventing to drop columns
manu-olx Jun 3, 2019
d75509e
add alter table replace columns tests
manu-olx Jun 3, 2019
aa6026f
refactor verifyAlterTableAddReplaceColumn
manu-olx Jun 3, 2019
0c3bc55
rebase and implement #24776 changes
manu-olx Jun 4, 2019
8b6da23
prevent replacing existing column with a different data type
manu-olx Jun 5, 2019
5915c88
align replace columns tests to new changes
manu-olx Jun 5, 2019
30641eb
fix 2 hive tests failing
manu-olx Jun 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rebase and implement #24776 changes
  • Loading branch information
@manu-olx
manu-olx committed Jun 4, 2019
commit 0c3bc55320716dc2cc801c7384154d0af1b98f30
4 changes: 2 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,7 @@ case class AlterTableAddColumnsCommand(

// Text format doesn't need ADD COLUMNS as text datasource can have only one column
override val supportedDatasourceFormats = Seq("ParquetFileFormat", "OrcFileFormat",
"OrcDataSourceV2", "JsonDataSourceV2", "CSVDataSourceV2")
"OrcDataSourceV2", "JsonFileFormat", "JsonDataSourceV2", "CSVFileFormat", "CSVDataSourceV2")

override def run(sparkSession: SparkSession): Seq[Row] = {
val catalog = sparkSession.sessionState.catalog
Expand Down Expand Up @@ -296,7 +296,7 @@ case class AlterTableReplaceColumnsCommand(
// Csv format not supported by REPLACE COLUMNS as csv datasource is read positionally
// and a new replacement column would reference an old replaced column's data
override val supportedDatasourceFormats = Seq("ParquetFileFormat", "OrcFileFormat",
"OrcDataSourceV2", "JsonDataSourceV2", "TextDataSourceV2")
"OrcDataSourceV2", "JsonFileFormat", "JsonDataSourceV2", "TextFileFormat", "TextDataSourceV2")

override def run(sparkSession: SparkSession): Seq[Row] = {
val catalog = sparkSession.sessionState.catalog
Expand Down
13 changes: 10 additions & 3 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2678,13 +2678,20 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
}
}

val supportedNativeFileFormatsForAlterTableReplaceColumns = Seq("parquet", "orc", "json", "text")
val supportedNativeFileFormatsForAlterTableReplaceColumns = Seq("parquet", "orc", "json", "text",
"org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat",
"org.apache.spark.sql.execution.datasources.orc.OrcFileFormat",
"org.apache.spark.sql.execution.datasources.json.JsonFileFormat",
"org.apache.spark.sql.execution.datasources.text.TextFileFormat")
supportedNativeFileFormatsForAlterTableReplaceColumns.foreach { provider =>
test(s"alter datasource table replace columns - $provider") {
if (provider == "text") testReplaceColumnTextProvider() else testReplaceColumn(provider)
if (provider == "text" | provider.endsWith("TextFileFormat")) testReplaceColumnTextProvider()
else testReplaceColumn(provider)
}
test(s"alter datasource table replace columns - partitioned - $provider") {
if (provider == "text") testReplaceColumnPartitionedTextProvider()
if (provider == "text" | provider.endsWith("TextFileFormat")) {
testReplaceColumnPartitionedTextProvider()
}
else testReplaceColumn(provider)
}
}
Expand Down