Skip to content

[SPARK-20460][SQL] Make it more consistent to handle column name duplication #17758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
maropu wants to merge 27 commits into from
Closed

[SPARK-20460][SQL] Make it more consistent to handle column name duplication #17758

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
1e647ee
Share code to check column name duplication
maropu Apr 25, 2017
4467077
Apply reviews
maropu Jun 13, 2017
33ab217
Make code more consistent
maropu Jun 13, 2017
d8efb9d
Apply review comments
maropu Jun 16, 2017
11d1818
Apply xiao's reviews
maropu Jun 16, 2017
22e1e4f
Apply more xiao's reviews
maropu Jun 17, 2017
743a069
Replace map with foreach
maropu Jun 20, 2017
f6eab2d
Add tests for data schema + parititon schema
maropu Jun 20, 2017
09da8d6
Drop name dplication checks in HiveMetastoreCatalog.scala
maropu Jun 20, 2017
6d03f31
Modify exception messages
maropu Jun 20, 2017
a0b9b05
Revert logic to check name duplication
maropu Jun 20, 2017
91b6424
Add tests for write paths
maropu Jun 21, 2017
37ad3f3
Add tests for stream sink paths
maropu Jun 21, 2017
d0d9d3e
Burhs up code and adds more tests
maropu Jun 25, 2017
cbe9c71
Apply reviews
maropu Jun 26, 2017
c69270f
Apply more comments
maropu Jun 27, 2017
af959f6
Add more tests in create.sql
maropu Jun 27, 2017
8d3e10a
Move duplication checks in constructor
maropu Jun 29, 2017
9b386d5
Brush up code
maropu Jun 30, 2017
a878510
[WIP] Add DataSourceValidator trait to validate schema in write path
maropu Jul 3, 2017
be20127
Revert "Brush up code"
maropu Jul 3, 2017
f41bf80
Fix more issues
maropu Jul 4, 2017
0526391
Revert DataSourceValidator
maropu Jul 4, 2017
9e199bc
Add the check for external relation providers
maropu Jul 4, 2017
1ae132d
[WIP] Handle DataSource name duplication in one place
maropu Jul 5, 2017
5c29a75
Fix more
maropu Jul 6, 2017
5ed2c0d
Move some tests to DDLSuite
maropu Jul 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add more tests in create.sql
  • Loading branch information
@maropu
maropu committed Jul 6, 2017
commit af959f6a55937e25db52cd2d94fc65af7f3a40d6
14 changes: 11 additions & 3 deletions sql/core/src/test/resources/sql-tests/inputs/create.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,19 @@
-- Catch case-sensitive name duplication
SET spark.sql.caseSensitive=true;

CREATE TABLE t (c0 STRING, c1 INT, c1 DOUBLE, c0 INT) USING parquet;
CREATE TABLE t(c0 STRING, c1 INT, c1 DOUBLE, c0 INT) USING parquet;
Copy link
Contributor

@cloud-fan cloud-fan Jul 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We didn't have test cases for create table before?

Copy link
Member Author

@maropu maropu Jul 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In DDLSuite, we already have simple tests for duplicate columns. we better moving these tests there?

Copy link
Contributor

@cloud-fan cloud-fan Jul 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should keep them in one place. For now I think we still need to put them in DDLSuite because we need to run it with and without hive support. Can we pick some typical test cases here and move them to DDLSuite?

Copy link
Member Author

@maropu maropu Jul 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, will update

Copy link
Member Author

@maropu maropu Jul 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved some tests to DDLSuite and removed this file.


CREATE TABLE t(c0 INT, c1 INT) PARTITIONED BY (c1 INT) STORED AS parquet;

CREATE TABLE t(c0 INT) PARTITIONED BY (c1 INT, c1 INT) STORED AS parquet;

-- Catch case-insensitive name duplication
SET spark.sql.caseSensitive=false;

CREATE TABLE t (c0 STRING, c1 INT, c1 DOUBLE, c0 INT) USING parquet;
CREATE TABLE t(c0 STRING, c1 INT, c1 DOUBLE, c0 INT) USING parquet;

CREATE TABLE t(ab STRING, cd INT, ef DOUBLE, Ab INT) USING parquet;

CREATE TABLE t(ab INT, cd INT) PARTITIONED BY (Ab INT) STORED AS parquet;

CREATE TABLE t (ab STRING, cd INT, ef DOUBLE, Ab INT) USING parquet;
CREATE TABLE t(ab INT, cd INT) PARTITIONED BY (Ab INT, aB INT) STORED AS parquet;
54 changes: 45 additions & 9 deletions sql/core/src/test/resources/sql-tests/results/create.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 5
-- Number of queries: 9


-- !query 0
Expand All @@ -11,7 +11,7 @@ spark.sql.caseSensitive true


-- !query 1
CREATE TABLE t (c0 STRING, c1 INT, c1 DOUBLE, c0 INT) USING parquet
CREATE TABLE t(c0 STRING, c1 INT, c1 DOUBLE, c0 INT) USING parquet
-- !query 1 schema
struct<>
-- !query 1 output
Expand All @@ -20,26 +20,62 @@ Found duplicate column(s) in the table definition of `t`: `c1`, `c0`;


-- !query 2
SET spark.sql.caseSensitive=false
CREATE TABLE t(c0 INT, c1 INT) PARTITIONED BY (c1 INT) STORED AS parquet
-- !query 2 schema
struct<key:string,value:string>
struct<>
-- !query 2 output
spark.sql.caseSensitive false
org.apache.spark.sql.AnalysisException
Found duplicate column(s) in the table definition of `t`: `c1`;


-- !query 3
CREATE TABLE t (c0 STRING, c1 INT, c1 DOUBLE, c0 INT) USING parquet
CREATE TABLE t(c0 INT) PARTITIONED BY (c1 INT, c1 INT) STORED AS parquet
-- !query 3 schema
struct<>
-- !query 3 output
org.apache.spark.sql.AnalysisException
Found duplicate column(s) in the table definition of `t`: `c1`, `c0`;
Found duplicate column(s) in the table definition of `t`: `c1`;


-- !query 4
CREATE TABLE t (ab STRING, cd INT, ef DOUBLE, Ab INT) USING parquet
SET spark.sql.caseSensitive=false
-- !query 4 schema
struct<>
struct<key:string,value:string>
-- !query 4 output
spark.sql.caseSensitive false


-- !query 5
CREATE TABLE t(c0 STRING, c1 INT, c1 DOUBLE, c0 INT) USING parquet
-- !query 5 schema
struct<>
-- !query 5 output
org.apache.spark.sql.AnalysisException
Found duplicate column(s) in the table definition of `t`: `c1`, `c0`;


-- !query 6
CREATE TABLE t(ab STRING, cd INT, ef DOUBLE, Ab INT) USING parquet
-- !query 6 schema
struct<>
-- !query 6 output
org.apache.spark.sql.AnalysisException
Found duplicate column(s) in the table definition of `t`: `ab`;


-- !query 7
CREATE TABLE t(ab INT, cd INT) PARTITIONED BY (Ab INT) STORED AS parquet
-- !query 7 schema
struct<>
-- !query 7 output
org.apache.spark.sql.AnalysisException
Found duplicate column(s) in the table definition of `t`: `ab`;


-- !query 8
CREATE TABLE t(ab INT, cd INT) PARTITIONED BY (Ab INT, aB INT) STORED AS parquet
-- !query 8 schema
struct<>
-- !query 8 output
org.apache.spark.sql.AnalysisException
Found duplicate column(s) in the table definition of `t`: `ab`;