Skip to content

Commit 609bd48

Browse files
chenhao-dbcloud-fan
authored andcommitted
[SPARK-47572][SQL] Enforce Window partitionSpec is orderable
### What changes were proposed in this pull request? In the `Window` node, both `partitionSpec` and `orderSpec` must be orderable, but the current type check only verifies `orderSpec` is orderable. This can cause an error in later optimizing phases. Given a query: ``` with t as (select id, map(id, id) as m from range(0, 10)) select rank() over (partition by m order by id) from t ``` Before the PR, it fails with an `INTERNAL_ERROR`: ``` org.apache.spark.SparkException: [INTERNAL_ERROR] grouping/join/window partition keys cannot be map type. SQLSTATE: XX000 at org.apache.spark.SparkException$.internalError(SparkException.scala:92) at org.apache.spark.SparkException$.internalError(SparkException.scala:96) at org.apache.spark.sql.catalyst.optimizer.NormalizeFloatingNumbers$.needNormalize(NormalizeFloatingNumbers.scala:103) at org.apache.spark.sql.catalyst.optimizer.NormalizeFloatingNumbers$.org$apache$spark$sql$catalyst$optimizer$NormalizeFloatingNumbers$$needNormalize(NormalizeFloatingNumbers.scala:94) ... ``` After the PR, it fails with a `EXPRESSION_TYPE_IS_NOT_ORDERABLE`, which is expected: ``` org.apache.spark.sql.catalyst.ExtendedAnalysisException: [EXPRESSION_TYPE_IS_NOT_ORDERABLE] Column expression "m" cannot be sorted because its type "MAP<BIGINT, BIGINT>" is not orderable. SQLSTATE: 42822; line 2 pos 53; Project [RANK() OVER (PARTITION BY m ORDER BY id ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#4] +- Project [id#1L, m#0, RANK() OVER (PARTITION BY m ORDER BY id ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#4, RANK() OVER (PARTITION BY m ORDER BY id ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#4] +- Window [rank(id#1L) windowspecdefinition(m#0, id#1L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS RANK() OVER (PARTITION BY m ORDER BY id ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#4], [m#0], [id#1L ASC NULLS FIRST] +- Project [id#1L, m#0] +- SubqueryAlias t +- SubqueryAlias t +- Project [id#1L, map(id#1L, id#1L) AS m#0] +- Range (0, 10, step=1, splits=None) at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:52) ... ``` ### How was this patch tested? Unit test. Closes apache#45730 from chenhao-db/SPARK-47572. Authored-by: Chenhao Li <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent 5259dcc commit 609bd48

File tree

2 files changed

+27
-0
lines changed

2 files changed

+27
-0
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -537,6 +537,19 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog with QueryErrorsB
537537
}
538538
}
539539

540+
case Window(_, partitionSpec, _, _) =>
541+
// Both `partitionSpec` and `orderSpec` must be orderable. We only need an extra check
542+
// for `partitionSpec` here because `orderSpec` has the type check itself.
543+
partitionSpec.foreach { p =>
544+
if (!RowOrdering.isOrderable(p.dataType)) {
545+
p.failAnalysis(
546+
errorClass = "EXPRESSION_TYPE_IS_NOT_ORDERABLE",
547+
messageParameters = Map(
548+
"expr" -> toSQLExpr(p),
549+
"exprType" -> toSQLType(p.dataType)))
550+
}
551+
}
552+
540553
case GlobalLimit(limitExpr, _) => checkLimitLikeClause("limit", limitExpr)
541554

542555
case LocalLimit(limitExpr, child) =>

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1316,4 +1316,18 @@ class AnalysisErrorSuite extends AnalysisTest with DataTypeErrorsBase {
13161316
)
13171317
}
13181318
}
1319+
1320+
errorClassTest(
1321+
"SPARK-47572: Enforce Window partitionSpec is orderable",
1322+
testRelation2.select(
1323+
WindowExpression(
1324+
new Rank(),
1325+
WindowSpecDefinition(
1326+
CreateMap(Literal("key") :: UnresolvedAttribute("a") :: Nil) :: Nil,
1327+
SortOrder(UnresolvedAttribute("b"), Ascending) :: Nil,
1328+
UnspecifiedFrame)).as("window")),
1329+
errorClass = "EXPRESSION_TYPE_IS_NOT_ORDERABLE",
1330+
messageParameters = Map(
1331+
"expr" -> "\"_w0\"",
1332+
"exprType" -> "\"MAP<STRING, STRING>\""))
13191333
}

0 commit comments

Comments
 (0)