[pull] master from apache:master #41
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…CY_ERROR_TEMP_2151-2175 ### What changes were proposed in this pull request? This PR proposes to migrate 25 execution errors onto temporary error classes with the prefix `_LEGACY_ERROR_TEMP_2151` to `_LEGACY_ERROR_TEMP_2175`. The error classes are prefixed with `_LEGACY_ERROR_TEMP_` indicates the dev-facing error messages, and won't be exposed to end users. ### Why are the changes needed? To speed-up the error class migration. The migration on temporary error classes allow us to analyze the errors, so we can detect the most popular error classes. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ``` $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite" $ build/sbt "test:testOnly *SQLQuerySuite" $ build/sbt -Phive-thriftserver "hive-thriftserver/testOnly org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite" ``` Closes #38149 from itholic/SPARK-40540-2151-2200. Authored-by: itholic <[email protected]> Signed-off-by: Max Gekk <[email protected]>
Cogrouping two grouped DataFrames in PySpark that have different group key cardinalities raises an error that is not very descriptive:
```python
left.groupby("id", "k")
.cogroup(right.groupby("id"))
```
```
py4j.protocol.Py4JJavaError: An error occurred while calling o726.collectToPython.
: java.lang.IndexOutOfBoundsException: 1
at scala.collection.mutable.ResizableArray.apply(ResizableArray.scala:46)
at scala.collection.mutable.ResizableArray.apply$(ResizableArray.scala:45)
at scala.collection.mutable.ArrayBuffer.apply(ArrayBuffer.scala:49)
at org.apache.spark.sql.catalyst.plans.physical.HashShuffleSpec.$anonfun$createPartitioning$5(partitioning.scala:650)
...
org.apache.spark.sql.execution.exchange.EnsureRequirements.$anonfun$ensureDistributionAndOrdering$14(EnsureRequirements.scala:159)
```
### What changes were proposed in this pull request?
Assert identical size of groupby keys and provide a meaningful error on cogroup.
### Why are the changes needed?
The error does not provide information on how to solve the problem.
### Does this PR introduce _any_ user-facing change?
Yes, raises an `AssertionError: group keys must have same size` instead.
### How was this patch tested?
Adds test `test_different_group_key_cardinality` to `pyspark.sql.tests.test_pandas_cogrouped_map`.
Closes #38036 from EnricoMi/branch-cogroup-key-mismatch.
Authored-by: Enrico Minack <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
### What changes were proposed in this pull request? This PR adds some more command examples to run Spark Connect that you built locally. ### Why are the changes needed? To guide developers to run Spark Connect they built locally. ### Does this PR introduce _any_ user-facing change? No, dev-only and doc-only. ### How was this patch tested? The commands were tested manually in my local. Closes #38236 from HyukjinKwon/minor-docs-spark-cnnect. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
### What changes were proposed in this pull request? Add WHERE to Connect proto and DSL. ### Why are the changes needed? Improve Connect proto testing coverage. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UT Closes #38232 from amaliujia/add_filter_to_dsl. Authored-by: Rui Wang <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
### What changes were proposed in this pull request? This PR refactors `checkCorrelationsInSubquery` in CheckAnalysis to use recursion instead of `foreachUp`. ### Why are the changes needed? Currently, the logic in `checkCorrelationsInSubquery` is inefficient and difficult to understand. It uses `foreachUp` to traverse the subquery plan tree, and traverses down an entire subtree of a plan node to check whether it contains any outer references. We can use recursion instead to traverse the plan tree only once to improve the performance and readability. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing unit tests. Closes #38226 from allisonwang-db/spark-40773-check-subquery. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
…subqueries using reference tracking ### What changes were proposed in this pull request? This PR reverts the previous fix #38052 and adds subquery reference tracking to `MergeScalarSubqueries` to restore previous functionality of merging independent nested subqueries. ### Why are the changes needed? Restore previous functionality but fix the bug discovered in https://issues.apache.org/jira/browse/SPARK-40618. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing and new UTs. Closes #38093 from peter-toth/SPARK-40618-fix-mergescalarsubqueries. Authored-by: Peter Toth <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
…PIVOT ### What changes were proposed in this pull request? Adds more tests for the SQL `UNPIVOT` clause. #37407 (comment) ### Why are the changes needed? Better test coverage. ### Does this PR introduce _any_ user-facing change? No, only more tests and fixing one issue. SQL `UNPIVOT` has not been released yet. ### How was this patch tested? In `UnpivotParserSuite` and `DatasetUnpivotSuite`. Closes #38153 from EnricoMi/branch-sql-unpivot-tests. Authored-by: Enrico Minack <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
…ly equivalent children in `RewriteDistinctAggregates` ### What changes were proposed in this pull request? In `RewriteDistinctAggregates`, when grouping aggregate expressions by function children, treat children that are semantically equivalent as the same. ### Why are the changes needed? This PR will reduce the number of projections in the Expand operator when there are multiple distinct aggregations with superficially different children. In some cases, it will eliminate the need for an Expand operator. Example: In the following query, the Expand operator creates 3\*n rows (where n is the number of incoming rows) because it has a projection for each of function children `b + 1`, `1 + b` and `c`. ``` create or replace temp view v1 as select * from values (1, 2, 3.0), (1, 3, 4.0), (2, 4, 2.5), (2, 3, 1.0) v1(a, b, c); select a, count(distinct b + 1), avg(distinct 1 + b) filter (where c > 0), sum(c) from v1 group by a; ``` The Expand operator has three projections (each producing a row for each incoming row): ``` [a#87, null, null, 0, null, UnscaledValue(c#89)], <== projection #1 (for regular aggregation) [a#87, (b#88 + 1), null, 1, null, null], <== projection #2 (for distinct aggregation of b + 1) [a#87, null, (1 + b#88), 2, (c#89 > 0.0), null]], <== projection #3 (for distinct aggregation of 1 + b) ``` In reality, the Expand only needs one projection for `1 + b` and `b + 1`, because they are semantically equivalent. With the proposed change, the Expand operator's projections look like this: ``` [a#67, null, 0, null, UnscaledValue(c#69)], <== projection #1 (for regular aggregations) [a#67, (b#68 + 1), 1, (c#69 > 0.0), null]], <== projection #2 (for distinct aggregation on b + 1 and 1 + b) ``` With one less projection, Expand produces 2\*n rows instead of 3\*n rows, but still produces the correct result. In the case where all distinct aggregates have semantically equivalent children, the Expand operator is not needed at all. Benchmark code in the JIRA (SPARK-40382). Before the PR: ``` distinct aggregates: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ all semantically equivalent 14721 14859 195 5.7 175.5 1.0X some semantically equivalent 14569 14572 5 5.8 173.7 1.0X none semantically equivalent 14408 14488 113 5.8 171.8 1.0X ``` After the PR: ``` distinct aggregates: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ all semantically equivalent 3658 3692 49 22.9 43.6 1.0X some semantically equivalent 9124 9214 127 9.2 108.8 0.4X none semantically equivalent 14601 14777 250 5.7 174.1 0.3X ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New unit tests. Closes #37825 from bersprockets/rewritedistinct_issue. Authored-by: Bruce Robbins <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
…erval` for `UnsafeRow` ### What changes were proposed in this pull request? As we know, `UnsafeRow` store one 8-byte word per field. Currently, `UnsafeRow` supports read and store `CalendarInterval` with two int field and one long field. The two int field are `months` and `days` of `CalendarInterval`. So `UnsafeRow` calls `putInt` of `Platform` twice and use two 8-byte word to store `months` and `days`. After my investigation, `getLong`/`putLong` of `Platform` have better performance than call `getInt`/`putInt` of `Platform` twice. Another side, if we can store `months` and `days` with one 8-byte word, we could save the memory too. This PR combines two int field `months` and `days` of `CalendarInterval` to a long value. So we could call `getLong` or `putLong` once. The micro benchmark before this PR show below. **Java 8** ``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_311-b11 on Mac OS X 10.16 Intel(R) Core(TM) i7-9750H CPU 2.60GHz CalendarInterval: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Call setInterval & getInterval 1336 1366 43 100.5 10.0 1.0X ``` **Java 11** ``` Java HotSpot(TM) 64-Bit Server VM 11.0.16+11-LTS-199 on Mac OS X 11.5.1 Intel(R) Core(TM) i7-9750H CPU 2.60GHz CalendarInterval: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Call setInterval & getInterval 1548 1560 18 86.7 11.5 1.0X ``` **Java 17** ``` Java HotSpot(TM) 64-Bit Server VM 17.0.4+11-LTS-179 on Mac OS X 11.5.1 Intel(R) Core(TM) i7-9750H CPU 2.60GHz CalendarInterval: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Call setInterval & getInterval 1266 1267 1 106.0 9.4 1.0X ``` After this PR, the micro benchmark outputs show below. **Java 8** ``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_311-b11 on Mac OS X 10.16 Intel(R) Core(TM) i7-9750H CPU 2.60GHz CalendarInterval: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Call setInterval & getInterval 1200 1201 2 111.9 8.9 1.0X ``` We can see about 13% performance improvement. **Java 11** ``` Java HotSpot(TM) 64-Bit Server VM 11.0.16+11-LTS-199 on Mac OS X 11.5.1 Intel(R) Core(TM) i7-9750H CPU 2.60GHz CalendarInterval: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Call setInterval & getInterval 1390 1391 1 96.5 10.4 1.0X ``` We can see about 10% performance improvement. **Java 17** ``` Java HotSpot(TM) 64-Bit Server VM 17.0.4+11-LTS-179 on Mac OS X 11.5.1 Intel(R) Core(TM) i7-9750H CPU 2.60GHz CalendarInterval: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Call setInterval & getInterval 1224 1233 13 109.7 9.1 1.0X ``` We can see about 3% performance improvement. ### Why are the changes needed? Improve the performance of `setInterval` & `getInterval` for `UnsafeRow`. ### Does this PR introduce _any_ user-facing change? 'No'. Just update the underlying implementation. ### How was this patch tested? New micro benchmark. Closes #38046 from beliefer/SPARK-40611. Authored-by: Jiaan Geng <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
### What changes were proposed in this pull request? This pr aims upgrade `jackson-databind` to 2.13.4.1. ### Why are the changes needed? This is a bug fix version related to [CVE-2022-42003] - FasterXML/jackson-databind#3621 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions Closes #38235 from LuciferYang/SPARK-40782. Authored-by: yangjie01 <[email protected]> Signed-off-by: Sean Owen <[email protected]>
…CY_ERROR_TEMP_2176-2220 ### What changes were proposed in this pull request? This PR proposes to migrate 25 execution errors onto temporary error classes with the prefix `_LEGACY_ERROR_TEMP_2176` to `_LEGACY_ERROR_TEMP_2200`. The error classes are prefixed with `_LEGACY_ERROR_TEMP_` indicates the dev-facing error messages, and won't be exposed to end users. ### Why are the changes needed? To speed-up the error class migration. The migration on temporary error classes allow us to analyze the errors, so we can detect the most popular error classes. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ``` $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite" $ build/sbt "test:testOnly *SQLQuerySuite" $ build/sbt -Phive-thriftserver "hive-thriftserver/testOnly org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite" ``` Closes #38169 from itholic/SPARK-40540-2176-2200. Authored-by: itholic <[email protected]> Signed-off-by: Max Gekk <[email protected]>
…lt of `ShowCreateTableAsSerdeCommand` have a fixed order ### What changes were proposed in this pull request? This pr add a sort operation to make the contents of `SERDEPROPERTIES` in the result of `ShowCreateTableAsSerdeCommand` have a fixed order. ### Why are the changes needed? Improve Java version compatibility, make the results of `ShowCreateTableAsSerdeCommand` consistent when using Java version. Run the following command with Java 19: ``` mvn clean install -DskipTests -pl sql/hive -am mvn test -pl sql/hive -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite ``` There are 2 failed test: ``` - SHOW CREATE TABLE using Hive V1 catalog V1 command: hive table with serde info *** FAILED *** "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187) Analysis: "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." - SHOW CREATE TABLE using Hive V1 catalog V2 command: hive table with serde info *** FAILED *** "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187) Analysis: "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." ``` The content of `SERDEPROPERTIES` is correct, but the order is different from result running with Java 8/11/17. ### Does this PR introduce _any_ user-facing change? Yes, the `SERDEPROPERTIES` part of `ShowCreateTableAsSerdeCommand` results will be displayed in order by key. ### How was this patch tested? - Pass Git Hub Actions - Manual test: Run the following command with Java 19: ``` mvn clean install -DskipTests -pl sql/hive -am mvn test -pl sql/hive -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite ``` **Before** ``` - SHOW CREATE TABLE using Hive V1 catalog V1 command: hive table with serde info *** FAILED *** "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187) Analysis: "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." - SHOW CREATE TABLE using Hive V1 catalog V2 command: hive table with serde info *** FAILED *** "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187) Analysis: "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." ``` **After** ``` Run completed in 23 seconds, 930 milliseconds. Total number of tests run: 58 Suites: completed 2, aborted 0 Tests: succeeded 58, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #38238 from LuciferYang/SPARK-40733. Authored-by: yangjie01 <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
…operations ### What changes were proposed in this pull request? This PR adds DS v2 APIs for handling row-level operations for data sources that support deltas of rows. ### Why are the changes needed? These changes are part of the approved SPIP in SPARK-35801. ### Does this PR introduce _any_ user-facing change? Yes, this PR adds new DS v2 APIs per [design doc](https://docs.google.com/document/d/12Ywmc47j3l2WF4anG5vL4qlrhT2OKigb7_EbIKhxg60). ### How was this patch tested? Tests will be part of the implementation PR. Closes #38004 from aokolnychyi/spark-40551. Lead-authored-by: Anton Okolnychyi <[email protected]> Co-authored-by: aokolnychyi <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
…ig options ### What changes were proposed in this pull request? The options passed from spark conf, hive-site.xml, AWS env vars now all record this in their source attribute of the entries. The Configuration Writable methods do not propagate this, so it is not as useful cluster-wide than it could be. It does help with some of the basic troubleshooting. ### Why are the changes needed? Helps when troubleshooting where options make their way down. These can be examined and logged later. For example, my cloudstore diagnosticss JAR can do this in its storediag command and in an s3a AWS credential provider. I may add some of that logging at debug to the ASF hadoop implementations. https://github.com/steveloughran/cloudstore ### Does this PR introduce _any_ user-facing change? Not *really*. It's a very low level diagnostics feature in the Hadoop configuration classes. ### How was this patch tested? New tests added; existing tests enhanced. Closes #38084 from steveloughran/SPARK-40640-spark-conf-propagation. Lead-authored-by: Steve Loughran <[email protected]> Co-authored-by: Steve Loughran <[email protected]> Co-authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
### What changes were proposed in this pull request? This PR cleans up the syntax, and properly copy protobuf assembly jar (currently it copies connect assembly jar by mistake). ### Why are the changes needed? For consistent code style, and correct SBT build. ### Does this PR introduce _any_ user-facing change? No, this isn't released yet. ### How was this patch tested? CI in this PR should test it out. Closes #38240 from HyukjinKwon/SPARK-40654. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
### What changes were proposed in this pull request?
This PR proposes to split the tests into the sub-packages:
**Before**
```
tests
├── __init__.py
├── test_arrow.py
├── test_arrow_map.py
├── test_catalog.py
├── test_column.py
├── test_conf.py
├── test_connect_basic.py
├── test_connect_column_expressions.py
├── test_connect_plan_only.py
├── test_connect_select_ops.py
├── test_context.py
├── test_dataframe.py
├── test_datasources.py
├── test_functions.py
├── test_group.py
├── test_pandas_cogrouped_map.py
├── test_pandas_grouped_map.py
├── test_pandas_grouped_map_with_state.py
├── test_pandas_map.py
├── test_pandas_udf.py
├── test_pandas_udf_grouped_agg.py
├── test_pandas_udf_scalar.py
├── test_pandas_udf_typehints.py
├── test_pandas_udf_typehints_with_future_annotations.py
├── test_pandas_udf_window.py
├── test_readwriter.py
├── test_serde.py
├── test_session.py
├── test_streaming.py
├── test_streaming_listener.py
├── test_types.py
├── test_udf.py
├── test_udf_profiler.py
├── test_utils.py
└── typing
├── ...
```
**After**
```
tests
├── __init__.py
├── connect
│ ├── __init__.py
│ ├── test_connect_basic.py
│ ├── test_connect_column_expressions.py
│ ├── test_connect_plan_only.py
│ └── test_connect_select_ops.py
├── pandas
│ ├── __init__.py
│ ├── test_pandas_cogrouped_map.py
│ ├── test_pandas_grouped_map.py
│ ├── test_pandas_grouped_map_with_state.py
│ ├── test_pandas_map.py
│ ├── test_pandas_udf.py
│ ├── test_pandas_udf_grouped_agg.py
│ ├── test_pandas_udf_scalar.py
│ ├── test_pandas_udf_typehints.py
│ ├── test_pandas_udf_typehints_with_future_annotations.py
│ └── test_pandas_udf_window.py
├── streaming
│ ├── __init__.py
│ ├── test_streaming.py
│ └── test_streaming_listener.py
├── test_arrow.py
├── test_arrow_map.py
├── test_catalog.py
├── test_column.py
├── test_conf.py
├── test_context.py
├── test_dataframe.py
├── test_datasources.py
├── test_functions.py
├── test_group.py
├── test_readwriter.py
├── test_serde.py
├── test_session.py
├── test_types.py
├── test_udf.py
├── test_udf_profiler.py
├── test_utils.py
└── typing
├── ...
```
This way is consistent with `pyspark.pandas.tests`.
### Why are the changes needed?
To make it easier to maintain, track and add the tests.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
CI in this PR should test it out.
Closes #38239 from HyukjinKwon/SPARK-40789.
Lead-authored-by: Hyukjin Kwon <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
### What changes were proposed in this pull request? This PR syncs python generated proto files. The proto changes in this file is generated by https://github.com/apache/spark/blob/master/connector/connect/dev/generate_protos.sh. ### Why are the changes needed? Python client side proto files are out of sync. Other python related PRs needs to re-generate proto files which has caused troubles on code review. We are looking for ways to automatically keep the python proto files in sync. Before that is done, we need to manually update the proto files. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UT Closes #38244 from amaliujia/sync_python_proto. Authored-by: Rui Wang <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>
…leAnalyzer ### What changes were proposed in this pull request? This PR is a followup of #38232 that fixes the corresponding Scaladoc at `SimpleAnalyzer`. ### Why are the changes needed? To document what it does correctly. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Ci should verify it. Closes #38248 from HyukjinKwon/SPARK-40780-followup. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
…rSuite ### What changes were proposed in this pull request? This PR aims to replace 'intercept' with 'Check error classes' in CreateNamespaceParserSuite. ### Why are the changes needed? The changes improve the error framework. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By running the modified test suite: ``` $ build/sbt "test:testOnly *CreateNamespaceParserSuite" ``` Closes #38246 from panbingkun/SPARK-40788. Authored-by: panbingkun <[email protected]> Signed-off-by: Max Gekk <[email protected]>
…abled ### What changes were proposed in this pull request? This PR is a followup of #37825, that change the types in the test relation to make the tests pass with ANSI enalbed. ### Why are the changes needed? To recover the test coverage. Currently it fails with ANSI mode on: https://github.com/apache/spark/actions/runs/3246829492/jobs/5326051798#step:9:20487 ``` [info] - SPARK-40382: eliminate multiple distinct groups due to superficial differences *** FAILED *** (5 milliseconds) [info] org.apache.spark.sql.AnalysisException: [DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE] Cannot resolve "(b + c)" due to data type mismatch: the binary operator requires the input type ("NUMERIC" or "INTERVAL DAY TO SECOND" or "INTERVAL YEAR TO MONTH" or "INTERVAL"), not "STRING". [info] at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.dataTypeMismatch(package.scala:68) [info] at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$5(CheckAnalysis.scala:223) [info] at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$5$adapted(CheckAnalysis.scala:210) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:295) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:294) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:294) [info] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) [info] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) [info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:294) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:2 ``` ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? Manually ran the tests locally. Closes #38250 from HyukjinKwon/SPARK-40382-followup. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
### What changes were proposed in this pull request?
When using the scalafmt.conf file as provided by the project repository to organize and optimize imports there is a case in which un-desired behavior appears.
If the import group does not fit on a single line, Scalafmt will by default try to bin-pack and then break out over mulitple lines.
For example:
import org.apache.spark.sql.catalyst.analysis.{UnresolvedAlias, UnresolvedAttribute, UnresolvedFunction, UnresolvedRelation, UnresolvedStar}
will become
import org.apache.spark.sql.catalyst.analysis.{
UnresolvedAlias,
UnresolvedAttributed,
...
}
In previous code reviews this has been marked as departing from the consistency of the Spark code base. This patch enables an option in Scalafmt to force the import group on a single line even if it exceeds the max line length.
### Why are the changes needed?
Code consistency.
### Does this PR introduce _any_ user-facing change?
No
Closes #38252 from grundprinzip/scalafmt.
Authored-by: Martin Grund <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
…bly jar ### What changes were proposed in this pull request? This pr add `assemblyPackageScala` and `assemblyExcludedJars` to `SparkProtobuf` module sbt build options to exclude redundant jars from spark-protobuf-assembly jar. ### Why are the changes needed? Exclude redundant jars from `spark-protobuf-assembly` jar ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - Manual test run `build/sbt "protobuf/assembly"` **Before** ``` [debug] Including from cache: spark-tags_2.12-3.4.0-SNAPSHOT.jar [debug] Including from cache: unused-1.0.0.jar [debug] Including from cache: spark-protobuf_2.12-3.4.0-SNAPSHOT.jar [debug] Including from cache: scala-collection-compat_2.12-2.2.0.jar [debug] Including from cache: pmml-model-1.4.8.jar [debug] Including from cache: protobuf-java-3.21.1.jar [debug] Including from cache: guava-14.0.1.jar [debug] Including from cache: scala-library-2.12.17.jar ``` **After** ``` [debug] Including: spark-protobuf_2.12-3.4.0-SNAPSHOT.jar [debug] Including from cache: protobuf-java-3.21.1.jar ``` Closes #38247 from LuciferYang/SPARK-40795. Authored-by: yangjie01 <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
### What changes were proposed in this pull request? This pr aims to upgrade netty from 4.1.80 to 4.1.84 , the `netty-tcnative-classes` version has not changed. ### Why are the changes needed? The release notes as follows: - https://netty.io/news/2022/09/08/4-1-81-Final.html - https://netty.io/news/2022/09/13/4-1-82-Final.html - https://netty.io/news/2022/10/11/4-1-84-Final.html There is no official 4.1.83 release notes. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions Closes #38245 from LuciferYang/netty-4184. Authored-by: yangjie01 <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
### What changes were proposed in this pull request? This PR aims to replace 'intercept' with 'Check error classes' in SparkSqlParserSuite. ### Why are the changes needed? The changes improve the error framework. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By running the modified test suite: ``` $ build/sbt "test:testOnly *SparkSqlParserSuite" ``` Closes #38255 from panbingkun/SPARK-40787. Authored-by: panbingkun <[email protected]> Signed-off-by: Max Gekk <[email protected]>
wangyum
pushed a commit
that referenced
this pull request
Jan 9, 2023
### What changes were proposed in this pull request?
Optimize the TransposeWindow rule to extend applicable cases and optimize time complexity.
TransposeWindow rule will try to eliminate unnecessary shuffle:
but the function compatiblePartitions will only take the first n elements of the window2 partition sequence, for some cases, this will not take effect, like the case below:
val df = spark.range(10).selectExpr("id AS a", "id AS b", "id AS c", "id AS d")
df.selectExpr(
"sum(`d`) OVER(PARTITION BY `b`,`a`) as e",
"sum(`c`) OVER(PARTITION BY `a`) as f"
).explain
Current plan
== Physical Plan ==
*(5) Project [e#10L, f#11L]
+- Window [sum(c#4L) windowspecdefinition(a#2L, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS f#11L], [a#2L]
+- *(4) Sort [a#2L ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(a#2L, 200), true, [id=#41]
+- *(3) Project [a#2L, c#4L, e#10L]
+- Window [sum(d#5L) windowspecdefinition(b#3L, a#2L, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS e#10L], [b#3L, a#2L]
+- *(2) Sort [b#3L ASC NULLS FIRST, a#2L ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(b#3L, a#2L, 200), true, [id=#33]
+- *(1) Project [id#0L AS d#5L, id#0L AS b#3L, id#0L AS a#2L, id#0L AS c#4L]
+- *(1) Range (0, 10, step=1, splits=10)
Expected plan:
== Physical Plan ==
*(4) Project [e#924L, f#925L]
+- Window [sum(d#43L) windowspecdefinition(b#41L, a#40L, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS e#924L], [b#41L, a#40L]
+- *(3) Sort [b#41L ASC NULLS FIRST, a#40L ASC NULLS FIRST], false, 0
+- *(3) Project [d#43L, b#41L, a#40L, f#925L]
+- Window [sum(c#42L) windowspecdefinition(a#40L, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS f#925L], [a#40L]
+- *(2) Sort [a#40L ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(a#40L, 200), true, [id=#282]
+- *(1) Project [id#38L AS d#43L, id#38L AS b#41L, id#38L AS a#40L, id#38L AS c#42L]
+- *(1) Range (0, 10, step=1, splits=10)
Also the permutations method has a O(n!) time complexity, which is very expensive when there are many partition columns, we could try to optimize it.
### Why are the changes needed?
We could apply the rule for more cases, which could improve the execution performance by eliminate unnecessary shuffle, and by reducing the time complexity from O(n!) to O(n2), the performance for the rule itself could improve
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
UT
Closes apache#35334 from constzhou/SPARK-38034_optimize_transpose_window_rule.
Authored-by: xzhou <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 0cc331d)
Signed-off-by: Wenchen Fan <[email protected]>
pull bot
pushed a commit
that referenced
this pull request
Jul 21, 2025
…ingBuilder` ### What changes were proposed in this pull request? This PR aims to improve `toString` by `JEP-280` instead of `ToStringBuilder`. In addition, `Scalastyle` and `Checkstyle` rules are added to prevent a future regression. ### Why are the changes needed? Since Java 9, `String Concatenation` has been handled better by default. | ID | DESCRIPTION | | - | - | | JEP-280 | [Indify String Concatenation](https://openjdk.org/jeps/280) | For example, this PR improves `OpenBlocks` like the following. Both Java source code and byte code are simplified a lot by utilizing JEP-280 properly. **CODE CHANGE** ```java - return new ToStringBuilder(this, ToStringStyle.SHORT_PREFIX_STYLE) - .append("appId", appId) - .append("execId", execId) - .append("blockIds", Arrays.toString(blockIds)) - .toString(); + return "OpenBlocks[appId=" + appId + ",execId=" + execId + ",blockIds=" + + Arrays.toString(blockIds) + "]"; ``` **BEFORE** ``` public java.lang.String toString(); Code: 0: new #39 // class org/apache/commons/lang3/builder/ToStringBuilder 3: dup 4: aload_0 5: getstatic #41 // Field org/apache/commons/lang3/builder/ToStringStyle.SHORT_PREFIX_STYLE:Lorg/apache/commons/lang3/builder/ToStringStyle; 8: invokespecial #47 // Method org/apache/commons/lang3/builder/ToStringBuilder."<init>":(Ljava/lang/Object;Lorg/apache/commons/lang3/builder/ToStringStyle;)V 11: ldc #50 // String appId 13: aload_0 14: getfield #7 // Field appId:Ljava/lang/String; 17: invokevirtual #51 // Method org/apache/commons/lang3/builder/ToStringBuilder.append:(Ljava/lang/String;Ljava/lang/Object;)Lorg/apache/commons/lang3/builder/ToStringBuilder; 20: ldc #55 // String execId 22: aload_0 23: getfield #13 // Field execId:Ljava/lang/String; 26: invokevirtual #51 // Method org/apache/commons/lang3/builder/ToStringBuilder.append:(Ljava/lang/String;Ljava/lang/Object;)Lorg/apache/commons/lang3/builder/ToStringBuilder; 29: ldc #56 // String blockIds 31: aload_0 32: getfield #16 // Field blockIds:[Ljava/lang/String; 35: invokestatic #57 // Method java/util/Arrays.toString:([Ljava/lang/Object;)Ljava/lang/String; 38: invokevirtual #51 // Method org/apache/commons/lang3/builder/ToStringBuilder.append:(Ljava/lang/String;Ljava/lang/Object;)Lorg/apache/commons/lang3/builder/ToStringBuilder; 41: invokevirtual #61 // Method org/apache/commons/lang3/builder/ToStringBuilder.toString:()Ljava/lang/String; 44: areturn ``` **AFTER** ``` public java.lang.String toString(); Code: 0: aload_0 1: getfield #7 // Field appId:Ljava/lang/String; 4: aload_0 5: getfield #13 // Field execId:Ljava/lang/String; 8: aload_0 9: getfield #16 // Field blockIds:[Ljava/lang/String; 12: invokestatic #39 // Method java/util/Arrays.toString:([Ljava/lang/Object;)Ljava/lang/String; 15: invokedynamic #43, 0 // InvokeDynamic #0:makeConcatWithConstants:(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String; 20: areturn ``` ### Does this PR introduce _any_ user-facing change? No. This is an `toString` implementation improvement. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#51572 from dongjoon-hyun/SPARK-52880. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
pull bot
pushed a commit
that referenced
this pull request
Nov 1, 2025
### What changes were proposed in this pull request? This PR proposes to add `doCanonicalize` function for DataSourceV2ScanRelation. The implementation is similar to [the one in BatchScanExec](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala#L150), as well as the [the one in LogicalRelation](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala#L52). ### Why are the changes needed? Query optimization rules such as MergeScalarSubqueries check if two plans are identical by [comparing their canonicalized form](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala#L219). For DSv2, for physical plan, the canonicalization goes down in the child hierarchy to the BatchScanExec, which [has a doCanonicalize function](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala#L150); for logical plan, the canonicalization goes down to the DataSourceV2ScanRelation, which, however, does not have a doCanonicalize function. As a result, two logical plans who are semantically identical are not identified. Moreover, for reference, [DSv1 LogicalRelation](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala#L52) also has `doCanonicalize()`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? A new unit test is added to show that `MergeScalarSubqueries` is working for DataSourceV2ScanRelation. For a query ```sql select (select max(i) from df) as max_i, (select min(i) from df) as min_i ``` Before introducing the canonicalization, the plan is ``` == Parsed Logical Plan == 'Project [scalar-subquery#2 [] AS max_i#3, scalar-subquery#4 [] AS min_i#5] : :- 'Project [unresolvedalias('max('i))] : : +- 'UnresolvedRelation [df], [], false : +- 'Project [unresolvedalias('min('i))] : +- 'UnresolvedRelation [df], [], false +- OneRowRelation == Analyzed Logical Plan == max_i: int, min_i: int Project [scalar-subquery#2 [] AS max_i#3, scalar-subquery#4 [] AS min_i#5] : :- Aggregate [max(i#0) AS max(i)#7] : : +- SubqueryAlias df : : +- View (`df`, [i#0, j#1]) : : +- RelationV2[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5 : +- Aggregate [min(i#10) AS min(i)#9] : +- SubqueryAlias df : +- View (`df`, [i#10, j#11]) : +- RelationV2[i#10, j#11] class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5 +- OneRowRelation == Optimized Logical Plan == Project [scalar-subquery#2 [] AS max_i#3, scalar-subquery#4 [] AS min_i#5] : :- Aggregate [max(i#0) AS max(i)#7] : : +- Project [i#0] : : +- RelationV2[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5 : +- Aggregate [min(i#10) AS min(i)#9] : +- Project [i#10] : +- RelationV2[i#10, j#11] class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5 +- OneRowRelation == Physical Plan == AdaptiveSparkPlan isFinalPlan=true +- == Final Plan == ResultQueryStage 0 +- *(1) Project [Subquery subquery#2, [id=#32] AS max_i#3, Subquery subquery#4, [id=#33] AS min_i#5] : :- Subquery subquery#2, [id=#32] : : +- AdaptiveSparkPlan isFinalPlan=true +- == Final Plan == ResultQueryStage 1 +- *(2) HashAggregate(keys=[], functions=[max(i#0)], output=[max(i)#7]) +- ShuffleQueryStage 0 +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=58] +- *(1) HashAggregate(keys=[], functions=[partial_max(i#0)], output=[max#14]) +- *(1) Project [i#0] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] +- == Initial Plan == HashAggregate(keys=[], functions=[max(i#0)], output=[max(i)#7]) +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=19] +- HashAggregate(keys=[], functions=[partial_max(i#0)], output=[max#14]) +- Project [i#0] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] : +- Subquery subquery#4, [id=#33] : +- AdaptiveSparkPlan isFinalPlan=true +- == Final Plan == ResultQueryStage 1 +- *(2) HashAggregate(keys=[], functions=[min(i#10)], output=[min(i)#9]) +- ShuffleQueryStage 0 +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=63] +- *(1) HashAggregate(keys=[], functions=[partial_min(i#10)], output=[min#15]) +- *(1) Project [i#10] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#10, j#11] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] +- == Initial Plan == HashAggregate(keys=[], functions=[min(i#10)], output=[min(i)#9]) +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=30] +- HashAggregate(keys=[], functions=[partial_min(i#10)], output=[min#15]) +- Project [i#10] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#10, j#11] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] +- *(1) Scan OneRowRelation[] +- == Initial Plan == Project [Subquery subquery#2, [id=#32] AS max_i#3, Subquery subquery#4, [id=#33] AS min_i#5] : :- Subquery subquery#2, [id=#32] : : +- AdaptiveSparkPlan isFinalPlan=true +- == Final Plan == ResultQueryStage 1 +- *(2) HashAggregate(keys=[], functions=[max(i#0)], output=[max(i)#7]) +- ShuffleQueryStage 0 +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=58] +- *(1) HashAggregate(keys=[], functions=[partial_max(i#0)], output=[max#14]) +- *(1) Project [i#0] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] +- == Initial Plan == HashAggregate(keys=[], functions=[max(i#0)], output=[max(i)#7]) +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=19] +- HashAggregate(keys=[], functions=[partial_max(i#0)], output=[max#14]) +- Project [i#0] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] : +- Subquery subquery#4, [id=#33] : +- AdaptiveSparkPlan isFinalPlan=true +- == Final Plan == ResultQueryStage 1 +- *(2) HashAggregate(keys=[], functions=[min(i#10)], output=[min(i)#9]) +- ShuffleQueryStage 0 +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=63] +- *(1) HashAggregate(keys=[], functions=[partial_min(i#10)], output=[min#15]) +- *(1) Project [i#10] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#10, j#11] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] +- == Initial Plan == HashAggregate(keys=[], functions=[min(i#10)], output=[min(i)#9]) +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=30] +- HashAggregate(keys=[], functions=[partial_min(i#10)], output=[min#15]) +- Project [i#10] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#10, j#11] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] +- Scan OneRowRelation[] ``` After introducing the canonicalization, the plan is as following, where you can see **ReusedSubquery** ``` == Parsed Logical Plan == 'Project [scalar-subquery#2 [] AS max_i#3, scalar-subquery#4 [] AS min_i#5] : :- 'Project [unresolvedalias('max('i))] : : +- 'UnresolvedRelation [df], [], false : +- 'Project [unresolvedalias('min('i))] : +- 'UnresolvedRelation [df], [], false +- OneRowRelation == Analyzed Logical Plan == max_i: int, min_i: int Project [scalar-subquery#2 [] AS max_i#3, scalar-subquery#4 [] AS min_i#5] : :- Aggregate [max(i#0) AS max(i)#7] : : +- SubqueryAlias df : : +- View (`df`, [i#0, j#1]) : : +- RelationV2[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5 : +- Aggregate [min(i#10) AS min(i)#9] : +- SubqueryAlias df : +- View (`df`, [i#10, j#11]) : +- RelationV2[i#10, j#11] class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5 +- OneRowRelation == Optimized Logical Plan == Project [scalar-subquery#2 [].max(i) AS max_i#3, scalar-subquery#4 [].min(i) AS min_i#5] : :- Project [named_struct(max(i), max(i)#7, min(i), min(i)#9) AS mergedValue#14] : : +- Aggregate [max(i#0) AS max(i)#7, min(i#0) AS min(i)#9] : : +- Project [i#0] : : +- RelationV2[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5 : +- Project [named_struct(max(i), max(i)#7, min(i), min(i)#9) AS mergedValue#14] : +- Aggregate [max(i#0) AS max(i)#7, min(i#0) AS min(i)#9] : +- Project [i#0] : +- RelationV2[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5 +- OneRowRelation == Physical Plan == AdaptiveSparkPlan isFinalPlan=true +- == Final Plan == ResultQueryStage 0 +- *(1) Project [Subquery subquery#2, [id=#40].max(i) AS max_i#3, ReusedSubquery Subquery subquery#2, [id=#40].min(i) AS min_i#5] : :- Subquery subquery#2, [id=#40] : : +- AdaptiveSparkPlan isFinalPlan=true +- == Final Plan == ResultQueryStage 1 +- *(2) Project [named_struct(max(i), max(i)#7, min(i), min(i)#9) AS mergedValue#14] +- *(2) HashAggregate(keys=[], functions=[max(i#0), min(i#0)], output=[max(i)#7, min(i)#9]) +- ShuffleQueryStage 0 +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=71] +- *(1) HashAggregate(keys=[], functions=[partial_max(i#0), partial_min(i#0)], output=[max#16, min#17]) +- *(1) Project [i#0] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] +- == Initial Plan == Project [named_struct(max(i), max(i)#7, min(i), min(i)#9) AS mergedValue#14] +- HashAggregate(keys=[], functions=[max(i#0), min(i#0)], output=[max(i)#7, min(i)#9]) +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=22] +- HashAggregate(keys=[], functions=[partial_max(i#0), partial_min(i#0)], output=[max#16, min#17]) +- Project [i#0] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] : +- ReusedSubquery Subquery subquery#2, [id=#40] +- *(1) Scan OneRowRelation[] +- == Initial Plan == Project [Subquery subquery#2, [id=#40].max(i) AS max_i#3, Subquery subquery#4, [id=#41].min(i) AS min_i#5] : :- Subquery subquery#2, [id=#40] : : +- AdaptiveSparkPlan isFinalPlan=true +- == Final Plan == ResultQueryStage 1 +- *(2) Project [named_struct(max(i), max(i)#7, min(i), min(i)#9) AS mergedValue#14] +- *(2) HashAggregate(keys=[], functions=[max(i#0), min(i#0)], output=[max(i)#7, min(i)#9]) +- ShuffleQueryStage 0 +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=71] +- *(1) HashAggregate(keys=[], functions=[partial_max(i#0), partial_min(i#0)], output=[max#16, min#17]) +- *(1) Project [i#0] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] +- == Initial Plan == Project [named_struct(max(i), max(i)#7, min(i), min(i)#9) AS mergedValue#14] +- HashAggregate(keys=[], functions=[max(i#0), min(i#0)], output=[max(i)#7, min(i)#9]) +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=22] +- HashAggregate(keys=[], functions=[partial_max(i#0), partial_min(i#0)], output=[max#16, min#17]) +- Project [i#0] +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] : +- Subquery subquery#4, [id=#41] : +- AdaptiveSparkPlan isFinalPlan=false : +- Project [named_struct(max(i), max(i)#7, min(i), min(i)#9) AS mergedValue#14] : +- HashAggregate(keys=[], functions=[max(i#0), min(i#0)], output=[max(i)#7, min(i)#9]) : +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=37] : +- HashAggregate(keys=[], functions=[partial_max(i#0), partial_min(i#0)], output=[max#16, min#17]) : +- Project [i#0] : +- BatchScan class org.apache.spark.sql.connector.SimpleDataSourceV2$$anon$5[i#0, j#1] class org.apache.spark.sql.connector.SimpleDataSourceV2$MyScanBuilder RuntimeFilters: [] +- Scan OneRowRelation[] ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#52529 from yhuang-db/scan-canonicalization. Authored-by: yhuang-db <[email protected]> Signed-off-by: Peter Toth <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )