Skip to content

[SPARK-20460][SQL] Make it more consistent to handle column name duplication #17758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
maropu wants to merge 27 commits into from
Closed

[SPARK-20460][SQL] Make it more consistent to handle column name duplication #17758

Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
1e647ee
Share code to check column name duplication
maropu Apr 25, 2017
4467077
Apply reviews
maropu Jun 13, 2017
33ab217
Make code more consistent
maropu Jun 13, 2017
d8efb9d
Apply review comments
maropu Jun 16, 2017
11d1818
Apply xiao's reviews
maropu Jun 16, 2017
22e1e4f
Apply more xiao's reviews
maropu Jun 17, 2017
743a069
Replace map with foreach
maropu Jun 20, 2017
f6eab2d
Add tests for data schema + parititon schema
maropu Jun 20, 2017
09da8d6
Drop name dplication checks in HiveMetastoreCatalog.scala
maropu Jun 20, 2017
6d03f31
Modify exception messages
maropu Jun 20, 2017
a0b9b05
Revert logic to check name duplication
maropu Jun 20, 2017
91b6424
Add tests for write paths
maropu Jun 21, 2017
37ad3f3
Add tests for stream sink paths
maropu Jun 21, 2017
d0d9d3e
Burhs up code and adds more tests
maropu Jun 25, 2017
cbe9c71
Apply reviews
maropu Jun 26, 2017
c69270f
Apply more comments
maropu Jun 27, 2017
af959f6
Add more tests in create.sql
maropu Jun 27, 2017
8d3e10a
Move duplication checks in constructor
maropu Jun 29, 2017
9b386d5
Brush up code
maropu Jun 30, 2017
a878510
[WIP] Add DataSourceValidator trait to validate schema in write path
maropu Jul 3, 2017
be20127
Revert "Brush up code"
maropu Jul 3, 2017
f41bf80
Fix more issues
maropu Jul 4, 2017
0526391
Revert DataSourceValidator
maropu Jul 4, 2017
9e199bc
Add the check for external relation providers
maropu Jul 4, 2017
1ae132d
[WIP] Handle DataSource name duplication in one place
maropu Jul 5, 2017
5c29a75
Fix more
maropu Jul 6, 2017
5ed2c0d
Move some tests to DDLSuite
maropu Jul 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Replace map with foreach
  • Loading branch information
@maropu
maropu committed Jul 6, 2017
commit 743a069db250625f0b5c67160dc1096029e9eaa6
9 changes: 6 additions & 3 deletions sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -693,7 +693,8 @@ class DataFrameReaderWriterSuite extends QueryTest with SharedSQLContext with Be
withTempDir { src =>
// Check CSV format
Seq("1,1").toDF().coalesce(1).write.mode("overwrite").text(src.toString)
Seq((true, "a INT, a INT"), (false, "aA INT, Aa INT")).map { case (caseSensitive, schema) =>
Seq((true, "a INT, a INT"), (false, "aA INT, Aa INT"))
.foreach { case (caseSensitive, schema) =>
withSQLConf(SQLConf.CASE_SENSITIVE.key -> caseSensitive.toString) {
val e = intercept[AnalysisException] {
spark.read.format("csv").schema(schema).load(src.toString)
Expand All @@ -709,7 +710,8 @@ class DataFrameReaderWriterSuite extends QueryTest with SharedSQLContext with Be
checkAnswer(df, Row(1, 1))

// Check JSON format
Seq((true, ("a", "a")), (false, ("aA", "Aa"))).map { case (caseSensitive, (c0, c1)) =>
Seq((true, ("a", "a")), (false, ("aA", "Aa")))
.foreach { case (caseSensitive, (c0, c1)) =>
Seq(s"""{"$c0":1, "$c1":1}""").toDF().coalesce(1).write.mode("overwrite").text(src.toString)
withSQLConf(SQLConf.CASE_SENSITIVE.key -> caseSensitive.toString) {
val e1 = intercept[AnalysisException] {
Expand All @@ -726,7 +728,8 @@ class DataFrameReaderWriterSuite extends QueryTest with SharedSQLContext with Be

// Check Paruqet format
Seq((1, 1)).toDF("c0", "c1").coalesce(1).write.mode("overwrite").parquet(src.toString)
Seq((true, "a INT, a INT"), (false, "aA INT, Aa INT")).map { case (caseSensitive, schema) =>
Seq((true, "a INT, a INT"), (false, "aA INT, Aa INT"))
.foreach { case (caseSensitive, schema) =>
withSQLConf(SQLConf.CASE_SENSITIVE.key -> caseSensitive.toString) {
val e = intercept[AnalysisException] {
spark.read.format("parquet").schema(schema).load(src.toString)
Expand Down