[SPARK-21150][SQL] Persistent view stored in Hive metastore should be case preserving #18360
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cloud-fan
commented
Jun 20, 2017
| val tableLocation = if (needDefaultTableLocation) { | ||
| Some(CatalogUtils.stringToURI(defaultTablePath(tableDefinition.identifier))) | ||
| } else { | ||
| // Ideally we should not create a managed table with location, but Hive serde table can |
Contributor
Author
There was a problem hiding this comment.
this change is actually small, just rearrange the if-else to make view and hive serde table fall into the same branch.
Contributor
Author
Member
|
LGTM |
Member
|
Oh. I just thought there might be an issue when doing The So once we replace an existing view, the schema properties will not be there anymore. |
|
Test build #78277 has finished for PR 18360 at commit
|
Contributor
Author
|
@viirya good catch! fixed |
|
Test build #78289 has finished for PR 18360 at commit
|
Member
|
Thanks! Merging to master/2.2 |
asfgit
pushed a commit
that referenced
this pull request
Jun 20, 2017
… case preserving ## What changes were proposed in this pull request? This is a regression in Spark 2.2. In Spark 2.2, we introduced a new way to resolve persisted view: https://issues.apache.org/jira/browse/SPARK-18209 , but this makes the persisted view non case-preserving because we store the schema in hive metastore directly. We should follow data source table and store schema in table properties. ## How was this patch tested? new regression test Author: Wenchen Fan <[email protected]> Closes #18360 from cloud-fan/view. (cherry picked from commit e862dc9) Signed-off-by: gatorsmile <[email protected]>
robert3005
pushed a commit
to palantir/spark
that referenced
this pull request
Jun 29, 2017
… case preserving ## What changes were proposed in this pull request? This is a regression in Spark 2.2. In Spark 2.2, we introduced a new way to resolve persisted view: https://issues.apache.org/jira/browse/SPARK-18209 , but this makes the persisted view non case-preserving because we store the schema in hive metastore directly. We should follow data source table and store schema in table properties. ## How was this patch tested? new regression test Author: Wenchen Fan <[email protected]> Closes apache#18360 from cloud-fan/view.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is a regression in Spark 2.2. In Spark 2.2, we introduced a new way to resolve persisted view: https://issues.apache.org/jira/browse/SPARK-18209 , but this makes the persisted view non case-preserving because we store the schema in hive metastore directly. We should follow data source table and store schema in table properties.
How was this patch tested?
new regression test