Skip to content

[SPARK-21617][SQL] Store correct table metadata when altering schema in Hive metastore. #18849

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vanzin wants to merge 14 commits into from
+171 −28
Closed

[SPARK-21617][SQL] Store correct table metadata when altering schema in Hive metastore. #18849

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
aae3abd
[SPARK-21617][SQL] Store correct metadata in Hive for altered DS table.
Aug 2, 2017
2350b10
Fix 2.1 DDL suite to not use SparkSession.
Aug 4, 2017
7ccf474
Proper fix.
Aug 4, 2017
40ebc96
Check for compatibility using serde, not path.
Aug 7, 2017
2f57a3c
equals -> ==
Aug 8, 2017
0b27209
Break out and expand Hive_2_1_DDLSuite.
Aug 8, 2017
6824e35
Feedback.
Aug 8, 2017
7b777ed
Move `isHiveCompatible` to its own method.
Aug 9, 2017
abd6cf1
Move comment.
Aug 9, 2017
4a05b55
Remove special handling of case-sensitive tables.
Aug 15, 2017
3c8a6bb
Merge branch 'master' into SPARK-21617
Aug 15, 2017
b63539d
Merge branch 'master' into SPARK-21617
Aug 21, 2017
cef66ac
Only retry writing to metastore for data source tables.
Aug 21, 2017
c41683a
Redo style change.
Aug 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move comment.
  • Loading branch information
Marcelo Vanzin committed Aug 9, 2017
commit abd6cf119c0b400fda9c96fcb6432b090c2505bf
5 changes: 2 additions & 3 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -907,14 +907,13 @@ private[hive] object HiveClientImpl {
val (partCols, schema) = table.schema.map(toHiveColumn).partition { c =>
table.partitionColumnNames.contains(c.getName)
}
// after SPARK-19279, it is not allowed to create a hive table with an empty schema,
// so here we should not add a default col schema
//
// Because HiveExternalCatalog sometimes writes back "raw" tables that have not been
// completely translated to Spark's view, the provider information needs to be looked
// up in two places.
val provider = table.provider.orElse(
Copy link
Contributor Author

@vanzin vanzin Aug 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change would have fixed the second exception in the bug (about storing an empty schema); but the code was just ending up in that situation because of the other problems this PR is fixing. This change shouldn't be needed for the fix, but I included it for further correctness.

Copy link
Member

@gatorsmile gatorsmile Aug 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the second exception? Could you explain more? If this is fixing a different bug, could you open a new JIRA and put it in the PR title?

Copy link
Contributor Author

@vanzin vanzin Aug 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you look at the bug, there are two exceptions. One gets logged, the second is thrown and caused the test to fail in my 2.1-based branch.

The exception happened because alterTableSchema is writing back the result of getRawTable. That raw table does not have the provider set; instead, it's in the table's properties. This check looks at both places, so that other code that uses getRawTable can properly pass this check.

As I explained in a previous comment, this doesn't happen anymore for alterTableSchema because of the other changes. But there's still code in the catalog class that writes back tables fetched with getRawTable, so this feels safer.

table.properties.get(HiveExternalCatalog.DATASOURCE_PROVIDER))
// after SPARK-19279, it is not allowed to create a hive table with an empty schema,
// so here we should not add a default col schema
if (schema.isEmpty && provider != Some(DDLUtils.HIVE_PROVIDER)) {
// This is a hack to preserve existing behavior. Before Spark 2.0, we do not
// set a default serde here (this was done in Hive), and so if the user provides
Expand Down