Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Aug 1, 2025

What changes were proposed in this pull request?

This PR aims to move TransformWith*Suite tests from sql - other tests pipeline to sql - slow tests pipeline for rebalancing the test workload and avoiding the AS-IS GitHub Action test timeout (120m).

$ find sql/core -name 'TransformWith*Suite.scala'
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateClusterSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateUnsafeRowSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateChainingSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateAvroSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateInitialStateSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithListStateTTLSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithValueStateTTLSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateTTLSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithListStateSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServerSuite.scala

Why are the changes needed?

This PR reduces sql - other tests running time from 101 mins to 84 mins.

BEFORE

Screenshot 2025-08-01 at 12 30 19

AFTER

Does this PR introduce any user-facing change?

No because this is a test change.

How was this patch tested?

Pass the CIs with the rebalanced tests.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun
Copy link
Member Author

Could you review this GitHub Action test rebalancing PR, @huaxingao ?

@dongjoon-hyun
Copy link
Member Author

All tests passed.

Copy link
Contributor

@huaxingao huaxingao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dongjoon-hyun
Copy link
Member Author

Thank you so much, @huaxingao . 😄

@dongjoon-hyun dongjoon-hyun deleted the SPARK-53068 branch August 1, 2025 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants