Skip to content

[SPARK-47217][SQL] Fix ambiguity check in self joins #45343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ahshahid wants to merge 12 commits into from
Closed

[SPARK-47217][SQL] Fix ambiguity check in self joins #45343

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f3dd17a
SPARK-47217. bug fix for exception thrown in reused dataframes involv…
Feb 29, 2024
c29366f
SPARK-47217. fix test failures
Mar 1, 2024
31d66c2
SPARK-47217. fix style format issue
Mar 1, 2024
127016c
SPARK-47217 : Fixing tests and code to try and resolve ambiguity in s…
Mar 6, 2024
b8e369c
SPARK-47217 : Fix unused import issue
Mar 6, 2024
872fece
SPARK-47217 : fixed bug and made assertions in existing tests for cor…
Mar 6, 2024
4b13514
Merge branch 'master' into SPARK-47217
Mar 6, 2024
8ed6aa4
SPARK-47217 : added more assetions
Mar 7, 2024
6b3b1d4
SPARK-47217 : fixed a bug and uncommented tests which were inadverten…
Mar 7, 2024
f9653ec
SPARK-47217 : added more tests and fixed inconsistency
Mar 7, 2024
68c2ad3
Merge branch 'master' into SPARK-47217
Mar 8, 2024
7150c98
SPARK-47217 : fixed test failure
Mar 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SPARK-47217. fix style format issue
  • Loading branch information
ashahid committed Mar 1, 2024
commit 31d66c2540423aa852d444c14e88cbfcfaaedc8c
17 changes: 8 additions & 9 deletions sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -571,10 +571,10 @@ public void testDedupBehaviourOnProjection_SPARK_47217() {

//Create table 3 DataFrame
List<Row> table3Data = Arrays.asList(RowFactory.create(1, 2, 3), RowFactory.create(1, 2, 3));
StructType table3Schema = new StructType().
add("col31", DataTypes.IntegerType).
add("col32", DataTypes.IntegerType).
add("col33", DataTypes.IntegerType);
StructType table3Schema = new StructType()
.add("col31", DataTypes.IntegerType)
.add("col32", DataTypes.IntegerType)
.add("col33", DataTypes.IntegerType);

Dataset<Row> table3 = spark.createDataFrame(table3Data, table3Schema);

Expand All @@ -591,16 +591,15 @@ public void testDedupBehaviourOnProjection_SPARK_47217() {
//Perform leftouter join for exchange table2(firstjoin)
srcDf = srcDf.join(
broadcast(table3),
srcDf.col("col12").equalTo(
table3.col("col32")), "left_outer").
select(srcDf.col("col11"), srcDf.col("col12"),
srcDf.col("col12").equalTo(table3.col("col32")),
"left_outer").select(srcDf.col("col11"),
srcDf.col("col12"),
srcDf.col("col13"),
table3.col("col33").as("col33_1"));

//Perform left outer joinfor exchangeRateTable1 again(secondjoin)
Dataset<Row> temp = srcDf.join(broadcast(table3),
srcDf.col("col11").equalTo(
table3.col("col31")), "left_outer");
srcDf.col("col11").equalTo(table3.col("col31")), "left_outer");

srcDf = temp.select(srcDf.col("col11"), srcDf.col("col12"),
srcDf.col("col13"), srcDf.col("col33_1"),
Expand Down