[SPARK-53413][SQL] Shuffle cleanup for commands #52157

karuppayya · 2025-08-28T06:07:10Z

What changes were proposed in this pull request?

Changes to cleanup shuffle generated from running commands(eg writes)
This was also brought by @cloud-fan and @ulysses-you here

Why are the changes needed?

To cleanupshuffle generated from commands

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test added

Was this patch authored or co-authored using generative AI tooling?

No

karuppayya · 2025-08-28T06:12:49Z

@cloud-fan @ulysses-you Please help review when you get a chance to.

cloud-fan · 2025-08-28T10:39:39Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala

-                      case exec: ShuffleExchangeLike =>
-                        exec.shuffleId
-                    }
+                  case command: V2CommandExec =>


how about other commands?

I have handled the DataWritingCommandExec,
I think it might be tricky for ExecutedCommandExec/V1Commands, since the spark plan is created internally.
Should we enumerate each SparkPlan type and handle them?

cloud-fan · 2025-08-28T10:40:41Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

      // for eagerly executed commands we mark this place as beginning of execution.
      tracker.setReadyForExecution()
-      val qe = sparkSession.sessionState.executePlan(p, mode)
+      val qe = sparkSession.sessionState.executePlan(p, mode, shuffleCleanupMode)


instead of calling executePlan, can we construct QueryExecution instance directly? I feel this executePlan method is a bit useless.

This seems to have been introduced here. It doesnt mention the rationale behind. But like you mentioned it does nt serve any real purpose, creating new QueryExecution

itskals · 2025-08-28T18:05:45Z

Why do we need this handling in commands? What about other flows? Or how is this different from other flows?

karuppayya · 2025-08-29T06:06:09Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala

-                      case exec: ShuffleExchangeLike =>
-                        exec.shuffleId
-                    }
+                  case command: V2CommandExec =>


I have handled the DataWritingCommandExec,
I think it might be tricky for ExecutedCommandExec/V1Commands, since the spark plan is created internally.
Should we enumerate each SparkPlan type and handle them?

karuppayya · 2025-08-29T06:07:01Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

This is related to this comment

karuppayya · 2025-08-29T06:33:25Z

Why do we need this handling in commands? What about other flows? Or how is this different from other flows?

All other flows were handled here
Commands wrap the actual query execution/Spark paln needed for the command's execution and had not been handled.
@itskals

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala

sql/core/src/test/scala/org/apache/spark/sql/execution/QueryExecutionSuite.scala

Fix

karuppayya · 2025-09-08T22:19:12Z

@cloud-fan Can you help merge this chnage, since there has not been any other comments last couple days?

dongjoon-hyun

+1, LGTM. Thank you, @karuppayya and @cloud-fan .
Merged to master for Apache Spark 4.1.0-preview2.

### What changes were proposed in this pull request? Changes to cleanup shuffle generated from running commands(eg writes) This was also brought by cloud-fan and ulysses-you [here](apache#45930 (comment)) ### Why are the changes needed? To cleanupshuffle generated from commands ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test added ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#52157 from karuppayya/SPARK-53413. Lead-authored-by: Karuppayya Rajendran <[email protected]> Co-authored-by: Karuppayya <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

see apache/spark#52157

…PlanExec` anywhere in plan tree ### What changes were proposed in this pull request? This PR uses `collectFirst` to find the first `AdaptiveSparkPlanExec` node anywhere in the plan tree, instead of assuming the root plan is an `AdaptiveSparkPlanExec`. ### Why are the changes needed? #52157 introduced the `extractShuffleIds` method in `SQLExecution` to find shuffle IDs of `SparkPlan`. Previously, the method implicitly assumed that if AQE is enabled, the `AdaptiveSparkPlanExec` would be at the root of the input. Since Spark only inserts `AdaptiveSparkPlanExec` under Command, this assumption was fine. However, the `AdaptiveSparkPlanExec` may not be the root node in Gluten. Gluten needs to insert a special physical plan to do column to row transition. By using `collectFirst`, we can correctly locate the `AdaptiveSparkPlanExec` regardless of its position in the plan tree, which improves compatibility. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass GHA. ### Was this patch authored or co-authored using generative AI tooling? No Closes #53620 from baibaichen/feature/extractShuffleIds. Authored-by: Chang chen <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit 082de7f) Signed-off-by: Kent Yao <[email protected]>

see apache/spark#52157

## Changes | Cause | Type | Category | Description | Affected Files | |-------|------|----------|-------------|----------------| | N/A | Feat | Build | Update build configuration to support Spark 4.1 UT | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/pom.xml`, `gluten-ut/spark41/pom.xml`, `tools/gluten-it/pom.xml` | | [#52165](apache/spark#52165) | Fix | Dependency | Update Parquet dependency version to 1.16.0 to avoid NoSuchMethodError issue | `gluten-ut/spark41/pom.xml` | | [#51477](apache/spark#51477) | Fix | Compatibility | Update imports to reflect streaming runtime package refactoring in Apache Spark | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala`, `gluten-ut/spark41/.../GlutenStreamingQuerySuite.scala` | | [#50674](apache/spark#50674) | Fix | Compatibility | Fix compatibility issue introduced by `TypedConfigBuilder` | `gluten-substrait/.../ExpressionConverter.scala`, `gluten-ut/spark41/.../GlutenCSVSuite.scala`, `gluten-ut/spark41/.../GlutenJsonSuite.scala` | | [#49766](apache/spark#49766) | Fix | Compatibility | Disable V2 bucketing in GlutenDynamicPartitionPruningSuite since spark.sql.sources.v2.bucketing.enabled is now enabled by default | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala` | | [#42414](apache/spark#42414), [#53038](apache/spark#53038) | Fix | Bug Fix | Resolve an issue introduced by SPARK-42414, as identified in SPARK-53038 | `backends-velox/.../VeloxBloomFilterAggregate.scala` | | N/A | Fix | Bug Fix | Enforce row fallback for unsupported cached batches - keep columnar execution only when schema validation succeeds | `backends-velox/.../ColumnarCachedBatchSerializer.scala` | | [SPARK-53132](apache/spark#53132), [SPARK-53142](apache/spark#53142) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 KeyGroupedPartitioningSuite tests. Excluded tests: `SPARK-53322*`, `SPARK-54439*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [SPARK-53535](https://issues.apache.org/jira/browse/SPARK-53535), [SPARK-54220](https://issues.apache.org/jira/browse/SPARK-54220) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenParquetIOSuite tests. Excluded tests: `SPARK-53535*`, `vectorized reader: missing all struct fields*`, `SPARK-54220*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52645](apache/spark#52645) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenStreamingQuerySuite tests. Excluded tests: `SPARK-53942: changing the number of stateless shuffle partitions via config`, `SPARK-53942: stateful shuffle partitions are retained from old checkpoint` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#47856](apache/spark#47856) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuite and GlutenJoinSuite tests. Excluded tests: `SPARK-49386: Window spill with more than the inMemoryThreshold and spillSizeThreshold`, `SPARK-49386: test SortMergeJoin (with spill by size threshold)` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52157](apache/spark#52157) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests. Excluded test: `#53413: Cleanup shuffle dependencies for commands` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#48470](apache/spark#48470) | 4.1.0 | Test Exclusion | Exclude split test in GlutenRegexpExpressionsSuite. Excluded test: `GlutenRegexpExpressionsSuite.SPLIT` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#51623](apache/spark#51623) | 4.1.0 | Test Exclusion | Add `spark.sql.unionOutputPartitioning=false` to Maven test args. Excluded tests: `GlutenBroadcastExchangeSuite.SPARK-52962`, `GlutenDataFrameSetOperationsSuite.SPARK-52921*` | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/spark41/.../VeloxTestSettings.scala`, `tools/gluten-it/common/.../Suite.scala` | | N/A | 4.1.0 | Test Exclusion | Excludes failed SQL tests that need to be fixed for Spark 4.1 compatibility. Excluded tests: `decimalArithmeticOperations.sql`, `identifier-clause.sql`, `keywords.sql`, `literals.sql`, `operators.sql`, `exists-orderby-limit.sql`, `postgreSQL/date.sql`, `nonansi/keywords.sql`, `nonansi/literals.sql`, `datetime-legacy.sql`, `datetime-parsing-invalid.sql`, `misc-functions.sql` | `gluten-ut/spark41/.../VeloxSQLQueryTestSettings.scala` |

## Changes | Cause | Type | Category | Description | Affected Files | |-------|------|----------|-------------|----------------| | N/A | Feat | Build | Update build configuration to support Spark 4.1 UT | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/pom.xml`, `gluten-ut/spark41/pom.xml`, `tools/gluten-it/pom.xml` | | [#52165](apache/spark#52165) | Fix | Dependency | Update Parquet dependency version to 1.16.0 to avoid NoSuchMethodError issue | `gluten-ut/spark41/pom.xml` | | [#51477](apache/spark#51477) | Fix | Compatibility | Update imports to reflect streaming runtime package refactoring in Apache Spark | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala`, `gluten-ut/spark41/.../GlutenStreamingQuerySuite.scala` | | [#50674](apache/spark#50674) | Fix | Compatibility | Fix compatibility issue introduced by `TypedConfigBuilder` | `gluten-substrait/.../ExpressionConverter.scala`, `gluten-ut/spark41/.../GlutenCSVSuite.scala`, `gluten-ut/spark41/.../GlutenJsonSuite.scala` | | [#49766](apache/spark#49766) | Fix | Compatibility | Disable V2 bucketing in GlutenDynamicPartitionPruningSuite since spark.sql.sources.v2.bucketing.enabled is now enabled by default | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala` | | [#42414](apache/spark#42414), [#53038](apache/spark#53038) | Fix | Bug Fix | Resolve an issue introduced by SPARK-42414, as identified in SPARK-53038 | `backends-velox/.../VeloxBloomFilterAggregate.scala` | | N/A | Fix | Bug Fix | Enforce row fallback for unsupported cached batches - keep columnar execution only when schema validation succeeds | `backends-velox/.../ColumnarCachedBatchSerializer.scala` | | [SPARK-53132](apache/spark#53132), [SPARK-53142](apache/spark#53142) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 KeyGroupedPartitioningSuite tests. Excluded tests: `SPARK-53322*`, `SPARK-54439*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [SPARK-53535](https://issues.apache.org/jira/browse/SPARK-53535), [SPARK-54220](https://issues.apache.org/jira/browse/SPARK-54220) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenParquetIOSuite tests. Excluded tests: `SPARK-53535*`, `vectorized reader: missing all struct fields*`, `SPARK-54220*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52645](apache/spark#52645) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenStreamingQuerySuite tests. Excluded tests: `SPARK-53942: changing the number of stateless shuffle partitions via config`, `SPARK-53942: stateful shuffle partitions are retained from old checkpoint` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#47856](apache/spark#47856) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuite and GlutenJoinSuite tests. Excluded tests: `SPARK-49386: Window spill with more than the inMemoryThreshold and spillSizeThreshold`, `SPARK-49386: test SortMergeJoin (with spill by size threshold)` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52157](apache/spark#52157) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests. Excluded test: `#53413: Cleanup shuffle dependencies for commands` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#48470](apache/spark#48470) | 4.1.0 | Test Exclusion | Exclude split test in GlutenRegexpExpressionsSuite. Excluded test: `GlutenRegexpExpressionsSuite.SPLIT` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#51623](apache/spark#51623) | 4.1.0 | Test Exclusion | Add `spark.sql.unionOutputPartitioning=false` to Maven test args. Excluded tests: `GlutenBroadcastExchangeSuite.SPARK-52962`, `GlutenDataFrameSetOperationsSuite.SPARK-52921*` | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/spark41/.../VeloxTestSettings.scala`, `tools/gluten-it/common/.../Suite.scala` | | N/A | 4.1.0 | Test Exclusion | Excludes failed SQL tests that need to be fixed for Spark 4.1 compatibility. Excluded tests: `decimalArithmeticOperations.sql`, `identifier-clause.sql`, `keywords.sql`, `literals.sql`, `operators.sql`, `exists-orderby-limit.sql`, `postgreSQL/date.sql`, `nonansi/keywords.sql`, `nonansi/literals.sql`, `datetime-legacy.sql`, `datetime-parsing-invalid.sql`, `misc-functions.sql` | `gluten-ut/spark41/.../VeloxSQLQueryTestSettings.scala` | | apache#11252 | 4.1.0 | Test Exclusion | Exclude Gluten test for SPARK-47939: Explain should work with parameterized queries | `gluten-ut/spark41/.../VeloxTestSettings.scala` |

## Changes | Cause | Type | Category | Description | Affected Files | |-------|------|----------|-------------|----------------| | N/A | Feat | Build | Update build configuration to support Spark 4.1 UT | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/pom.xml`, `gluten-ut/spark41/pom.xml`, `tools/gluten-it/pom.xml` | | [#52165](apache/spark#52165) | Fix | Dependency | Update Parquet dependency version to 1.16.0 to avoid NoSuchMethodError issue | `gluten-ut/spark41/pom.xml` | | [#51477](apache/spark#51477) | Fix | Compatibility | Update imports to reflect streaming runtime package refactoring in Apache Spark | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala`, `gluten-ut/spark41/.../GlutenStreamingQuerySuite.scala` | | [#50674](apache/spark#50674) | Fix | Compatibility | Fix compatibility issue introduced by `TypedConfigBuilder` | `gluten-substrait/.../ExpressionConverter.scala`, `gluten-ut/spark41/.../GlutenCSVSuite.scala`, `gluten-ut/spark41/.../GlutenJsonSuite.scala` | | [#49766](apache/spark#49766) | Fix | Compatibility | Disable V2 bucketing in GlutenDynamicPartitionPruningSuite since spark.sql.sources.v2.bucketing.enabled is now enabled by default | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala` | | [#42414](apache/spark#42414), [#53038](apache/spark#53038) | Fix | Bug Fix | Resolve an issue introduced by SPARK-42414, as identified in SPARK-53038 | `backends-velox/.../VeloxBloomFilterAggregate.scala` | | N/A | Fix | Bug Fix | Enforce row fallback for unsupported cached batches - keep columnar execution only when schema validation succeeds | `backends-velox/.../ColumnarCachedBatchSerializer.scala` | | [SPARK-53132](apache/spark#53132), [SPARK-53142](apache/spark#53142) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 KeyGroupedPartitioningSuite tests. Excluded tests: `SPARK-53322*`, `SPARK-54439*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [SPARK-53535](https://issues.apache.org/jira/browse/SPARK-53535), [SPARK-54220](https://issues.apache.org/jira/browse/SPARK-54220) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenParquetIOSuite tests. Excluded tests: `SPARK-53535*`, `vectorized reader: missing all struct fields*`, `SPARK-54220*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52645](apache/spark#52645) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenStreamingQuerySuite tests. Excluded tests: `SPARK-53942: changing the number of stateless shuffle partitions via config`, `SPARK-53942: stateful shuffle partitions are retained from old checkpoint` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#47856](apache/spark#47856) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuite and GlutenJoinSuite tests. Excluded tests: `SPARK-49386: Window spill with more than the inMemoryThreshold and spillSizeThreshold`, `SPARK-49386: test SortMergeJoin (with spill by size threshold)` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52157](apache/spark#52157) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests. Excluded test: `#53413: Cleanup shuffle dependencies for commands` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#48470](apache/spark#48470) | 4.1.0 | Test Exclusion | Exclude split test in GlutenRegexpExpressionsSuite. Excluded test: `GlutenRegexpExpressionsSuite.SPLIT` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#51623](apache/spark#51623) | 4.1.0 | Test Exclusion | Add `spark.sql.unionOutputPartitioning=false` to Maven test args. Excluded tests: `GlutenBroadcastExchangeSuite.SPARK-52962`, `GlutenDataFrameSetOperationsSuite.SPARK-52921*` | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/spark41/.../VeloxTestSettings.scala`, `tools/gluten-it/common/.../Suite.scala` | | N/A | 4.1.0 | Test Exclusion | Excludes failed SQL tests that need to be fixed for Spark 4.1 compatibility. Excluded tests: `decimalArithmeticOperations.sql`, `identifier-clause.sql`, `keywords.sql`, `literals.sql`, `operators.sql`, `exists-orderby-limit.sql`, `postgreSQL/date.sql`, `nonansi/keywords.sql`, `nonansi/literals.sql`, `datetime-legacy.sql`, `datetime-parsing-invalid.sql`, `misc-functions.sql` | `gluten-ut/spark41/.../VeloxSQLQueryTestSettings.scala` | | #11252 | 4.1.0 | Test Exclusion | Exclude Gluten test for SPARK-47939: Explain should work with parameterized queries | `gluten-ut/spark41/.../VeloxTestSettings.scala` |

github-actions bot added the SQL label Aug 28, 2025

karuppayya mentioned this pull request Aug 28, 2025

[SPARK-52777][SQL] Enable shuffle cleanup mode configuration in Spark SQL #51458

Closed

cloud-fan reviewed Aug 28, 2025

View reviewed changes

karuppayya commented Aug 29, 2025

View reviewed changes

karuppayya force-pushed the SPARK-53413 branch from ef482e2 to 25406bc Compare August 29, 2025 06:25

cloud-fan reviewed Sep 1, 2025

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Sep 1, 2025

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/execution/QueryExecutionSuite.scala Outdated Show resolved Hide resolved

cloud-fan approved these changes Sep 1, 2025

View reviewed changes

karuppayya added 4 commits September 1, 2025 09:42

SPARK-53413: Shuffle cleanup for commands

616b5e3

Fix

Add logs, handle DataWritingCommandExec

602dfb1

Write test for DataWritingCommandExec

6cef7bd

Address review comments: Remove unused code, fix test to write csv

6850ebf

karuppayya force-pushed the SPARK-53413 branch 2 times, most recently from 7c12739 to 6850ebf Compare September 3, 2025 04:46

Trigger CI

4fa440c

dongjoon-hyun changed the title ~~[SPARK-53413] Shuffle cleanup for commands~~ [SPARK-53413][SQL] Shuffle cleanup for commands Sep 12, 2025

dongjoon-hyun approved these changes Sep 12, 2025

View reviewed changes

dongjoon-hyun closed this in 1dab449 Sep 12, 2025

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 22, 2025

[Fix] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

93f4366

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 22, 2025

[Fix] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

ce03a81

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 22, 2025

[Fix] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

f693dd0

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[Fix] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

ead47a1

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[Fix] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

176d82b

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 29, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

c88ba42

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 30, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

d1a9803

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

c058467

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

d3edf04

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

3b64339

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

f3ffc6e

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

0a6c385

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

99b8914

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

78652c7

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

161b518

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

2a2bd97

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

262d9a9

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

67ffae4

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

d02504e

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

09b15b3

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

ed3e526

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

fa465fb

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

f5bcdeb

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

c03607b

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

23e6364

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 5, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

398de5d

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 5, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

06595c6

see apache/spark#52157

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 5, 2026

[4.1.0] Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests

be93782

see apache/spark#52157

baibaichen mentioned this pull request Jan 7, 2026

[GLUTEN-11343][CORE][VL] Support Spark 4.1 UT apache/incubator-gluten#11353

Merged

[SPARK-53413][SQL] Shuffle cleanup for commands #52157

[SPARK-53413][SQL] Shuffle cleanup for commands #52157

Uh oh!

Conversation

karuppayya commented Aug 28, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

karuppayya commented Aug 28, 2025

Uh oh!

cloud-fan Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

karuppayya Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

cloud-fan Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

karuppayya Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

itskals commented Aug 28, 2025

Uh oh!

karuppayya Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

karuppayya Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

karuppayya commented Aug 29, 2025

Uh oh!

Uh oh!

Uh oh!

karuppayya commented Sep 8, 2025

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants