Skip to content

[SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec #33424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 3 commits into from
Closed

[SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec #33424

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
afa5539
[SPARK-36213][SQL] Normalize PartitionSpec for DescTable with Partiti…
yaooqinn Jul 19, 2021
92ba197
Merge branch 'master' into SPARK-36213
yaooqinn Jul 21, 2021
ce08ba4
Merge branch 'master' of github.com:apache/spark into SPARK-36213
yaooqinn Jul 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -658,7 +658,12 @@ case class DescribeTableCommand(
throw QueryCompilationErrors.descPartitionNotAllowedOnView(table.identifier)
}
DDLUtils.verifyPartitionProviderIsHive(spark, metadata, "DESC PARTITION")
val partition = catalog.getPartition(table, partitionSpec)
val normalizedPartSpec = PartitioningUtils.normalizePartitionSpec(
partitionSpec,
metadata.partitionSchema,
table.quotedString,
spark.sessionState.conf.resolver)
val partition = catalog.getPartition(table, normalizedPartSpec)
if (isExtended) describeFormattedDetailedPartitionInfo(table, metadata, partition, result)
}

Expand Down
2 changes: 2 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/describe.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ DESC EXTENDED t PARTITION (c='Us', d=1);

DESC FORMATTED t PARTITION (c='Us', d=1);

DESC EXTENDED t PARTITION (C='Us', D=1);

-- NoSuchPartitionException: Partition not found in table
DESC t PARTITION (c='Us', d=2);

Expand Down
33 changes: 32 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/describe.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 41
-- Number of queries: 42


-- !query
Expand Down Expand Up @@ -324,6 +324,37 @@ Location [not included in comparison]/{warehouse_dir}/t
Storage Properties [a=1, b=2]


-- !query
DESC EXTENDED t PARTITION (C='Us', D=1)
Copy link
Contributor

@cloud-fan cloud-fan Jul 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what was the result before this PR?

Copy link
Member Author

@yaooqinn yaooqinn Jul 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

+-- !query
+DESC EXTENDED t PARTITION (C='Us', D=1)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+Partition spec is invalid. The spec (C, D) must match the partition spec (c, d) defined in table '`default`.`t`'
+

-- !query schema
struct<col_name:string,data_type:string,comment:string>
-- !query output
a string
b int
c string
d string
# Partition Information
# col_name data_type comment
c string
d string

# Detailed Partition Information
Database default
Table t
Partition Values [c=Us, d=1]
Location [not included in comparison]/{warehouse_dir}/t/c=Us/d=1
Storage Properties [a=1, b=2]
Created Time [not included in comparison]
Last Access [not included in comparison]

# Storage Information
Num Buckets 2
Bucket Columns [`a`]
Sort Columns [`b`]
Location [not included in comparison]/{warehouse_dir}/t
Storage Properties [a=1, b=2]


-- !query
DESC t PARTITION (c='Us', d=2)
-- !query schema
Expand Down