Skip to content

Commit 10d4ffd

Browse files
gatorsmileyeshengm
andcommitted
[SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"
## What changes were proposed in this pull request? Explained why "Subquery" and "Join Reorder" optimization batches should be `FixedPoint(1)`, which was introduced in SPARK-28532 and SPARK-28530. ## How was this patch tested? Existing UTs. Closes apache#25320 from yeshengm/SPARK-28530-followup. Lead-authored-by: Xiao Li <[email protected]> Co-authored-by: Yesheng Ma <[email protected]> Signed-off-by: gatorsmile <[email protected]>
1 parent 8ae032d commit 10d4ffd

File tree

1 file changed

+4
-0
lines changed
  • sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer

1 file changed

+4
-0
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,8 @@ abstract class Optimizer(sessionCatalog: SessionCatalog)
152152
PropagateEmptyRelation) ::
153153
Batch("Pullup Correlated Expressions", Once,
154154
PullupCorrelatedPredicates) ::
155+
// Subquery batch applies the optimizer rules recursively. Therefore, it makes no sense
156+
// to enforce idempotence on it and we change this batch from Once to FixedPoint(1).
155157
Batch("Subquery", FixedPoint(1),
156158
OptimizeSubqueries) ::
157159
Batch("Replace Operators", fixedPoint,
@@ -165,6 +167,8 @@ abstract class Optimizer(sessionCatalog: SessionCatalog)
165167
RemoveLiteralFromGroupExpressions,
166168
RemoveRepetitionFromGroupExpressions) :: Nil ++
167169
operatorOptimizationBatch) :+
170+
// Since join costs in AQP can change between multiple runs, there is no reason that we have an
171+
// idempotence enforcement on this batch. We thus make it FixedPoint(1) instead of Once.
168172
Batch("Join Reorder", FixedPoint(1),
169173
CostBasedJoinReorder) :+
170174
Batch("Remove Redundant Sorts", Once,

0 commit comments

Comments
 (0)