[SPARK-34556][SQL][3.0] Checking duplicate static partition columns should respect case sensitive conf #31714
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… respect case sensitive conf This PR makes partition spec parsing respect case sensitive conf. When parsing the partition spec, Spark will call `org.apache.spark.sql.catalyst.parser.ParserUtils.checkDuplicateKeys` to check if there are duplicate partition column names in the list. But this method is always case sensitive and doesn't detect duplicate partition column names when using different cases. Yep. This prevents users from writing incorrect queries such as `INSERT OVERWRITE t PARTITION (c='2', C='3') VALUES (1)` when they don't enable case sensitive conf. The new added test will fail without this change. Closes apache#31669 from zsxwing/SPARK-34556. Authored-by: Shixiong Zhu <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
dongjoon-hyun
changed the title
dongjoon-hyun
approved these changes
Mar 3, 2021
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM (Pending CIs).
HyukjinKwon
approved these changes
Mar 3, 2021
Member
|
Merged to branch-3.0. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Mar 3, 2021
…hould respect case sensitive conf ### What changes were proposed in this pull request? Backport #31669 to branch-3.0. Add a new test in DDLParserSuite.scala since SQLInsertTestSuite.scala doesn't exist in branch-3.0. Closes #31714 from zsxwing/SPARK-34556-3.0. Authored-by: Shixiong Zhu <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
|
Test build #135679 has finished for PR 31714 at commit
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Backport #31669 to branch-3.0. Add a new test in DDLParserSuite.scala since SQLInsertTestSuite.scala doesn't exist in branch-3.0.