[SPARK-13454] [SQL] Allow users to drop a table with a name starting with an underscore. #11349
Conversation
|
This PR is just for branch 1.6. Master does not have this problem. |
|
Test build #51889 has finished for PR 11349 at commit
|
There was a problem hiding this comment.
the comment is incomplete...
|
Test build #51923 has finished for PR 11349 at commit
|
There was a problem hiding this comment.
DropTable also tries to unregister temporary tables of the same name using:
hiveContext.catalog.unregisterTable(TableIdentifier(tableName))Now the tableName argument of DropTable is quoted when it's started with _, but TableIdentifier doesn't parse and unquote the table name. Thus the temporary table can't be dropped in this case.
Another problem of the above line is that, DropTable.tableName can be the form of db.table, but we throw the whole string into TableIdentifier without splitting them. But that's another bug.
|
OK I have addressed the comments. I moved the fix to DropTable. |
|
Test build #52062 has finished for PR 11349 at commit
|
|
Test build #52063 has finished for PR 11349 at commit
|
|
Merging to 1.6 |
…ith an underscore. ## What changes were proposed in this pull request? This change adds a workaround to allow users to drop a table with a name starting with an underscore. Without this patch, we can create such a table, but we cannot drop it. The reason is that Hive's parser unquote an quoted identifier (see https://github.com/apache/hive/blob/release-1.2.1/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g#L453). So, when we issue a drop table command to Hive, a table name starting with an underscore is actually not quoted. Then, Hive will complain about it because it does not support a table name starting with an underscore without using backticks (underscores are allowed as long as it is not the first char though). ## How was this patch tested? Add a test to make sure we can drop a table with a name starting with an underscore. https://issues.apache.org/jira/browse/SPARK-13454 Author: Yin Huai <[email protected]> Closes #11349 from yhuai/fixDropTable.
4 participants
What changes were proposed in this pull request?
This change adds a workaround to allow users to drop a table with a name starting with an underscore. Without this patch, we can create such a table, but we cannot drop it. The reason is that Hive's parser unquote an quoted identifier (see https://github.com/apache/hive/blob/release-1.2.1/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g#L453). So, when we issue a drop table command to Hive, a table name starting with an underscore is actually not quoted. Then, Hive will complain about it because it does not support a table name starting with an underscore without using backticks (underscores are allowed as long as it is not the first char though).
How was this patch tested?
Add a test to make sure we can drop a table with a name starting with an underscore.
https://issues.apache.org/jira/browse/SPARK-13454