Skip to content

[SPARK-34527][SQL] Resolve duplicated common columns from USING/NATURAL JOIN #31666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
karenfeng wants to merge 44 commits into from
Closed

[SPARK-34527][SQL] Resolve duplicated common columns from USING/NATURAL JOIN #31666

Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
1c5ab03
Only use metadata columns for resolution as last resort
karenfeng Feb 26, 2021
2fe733f
Resolve deduplicated common columns in NATURAL/USING JOIN with hidden…
karenfeng Feb 26, 2021
2c261bb
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Feb 26, 2021
80beda8
Fix behavior in Scala
karenfeng Feb 27, 2021
e1719d3
Fix nested expression
karenfeng Feb 27, 2021
6fa70ba
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 2, 2021
0ba1916
Address comments
karenfeng Mar 4, 2021
2b7e730
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 4, 2021
0c116a5
Push SQL output
karenfeng Mar 4, 2021
bf87f55
Re-do changes
karenfeng Mar 7, 2021
b5dc44f
Fixup tag
karenfeng Mar 8, 2021
6e32b3d
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 8, 2021
7c3f5df
Add another failing test
karenfeng Mar 8, 2021
181751a
Merge add metadata and resolve missing references
karenfeng Mar 29, 2021
ad5e824
Resolve down to avoid prematurely projecting out
karenfeng Mar 29, 2021
e36e853
Remove printlns
karenfeng Mar 29, 2021
db44c53
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 29, 2021
73b7c8a
Clean up
karenfeng Mar 29, 2021
1eb01e2
Style fixup
karenfeng Mar 29, 2021
9fd2490
Formatting fix
karenfeng Mar 29, 2021
f5cc3ae
Address comments
karenfeng Mar 30, 2021
fa7207e
Metadata output should be empty by default
karenfeng Mar 31, 2021
7af12ae
Clean up
karenfeng Mar 31, 2021
07f9ad5
Always resolve with metadata
karenfeng Mar 31, 2021
c474745
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 6, 2021
0f267e7
Revert accidental metadata changes
karenfeng Apr 6, 2021
ed0270c
Revert DatasourceV2SQLSuite
karenfeng Apr 6, 2021
66ad572
Revert unneeded style changes
karenfeng Apr 6, 2021
fc3b16d
Revert accidental metadata output change
karenfeng Apr 6, 2021
f665030
Add childrens' hidden output to Project metadataOutput
karenfeng Apr 7, 2021
85b81b1
Retrigger tests
karenfeng Apr 7, 2021
eab7964
Add comments
karenfeng Apr 7, 2021
44ee9f8
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 7, 2021
c84f396
Address comments and fix propagation through Projects
karenfeng Apr 7, 2021
0fe04a2
Retrigger tests
karenfeng Apr 7, 2021
c7c3df6
Retrigger tests
karenfeng Apr 7, 2021
b1bf28d
Retrigger tests
karenfeng Apr 8, 2021
47be66d
Wording
karenfeng Apr 9, 2021
8c5144e
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 9, 2021
333a815
propagate hidden columns from nested NATURAL/USING JOINs
karenfeng Apr 12, 2021
49de5c5
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 12, 2021
9e62d7d
Retrigger tests
karenfeng Apr 12, 2021
446d4bc
address comments
karenfeng Apr 13, 2021
8f70c2d
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 13, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Push SQL output 
Signed-off-by: Karen Feng <[email protected]>
  • Loading branch information
@karenfeng
karenfeng committed Mar 4, 2021
commit 0c116a59aff69ce4aca4c504099abeef600f1e55
16 changes: 8 additions & 8 deletions sql/core/src/test/resources/sql-tests/results/udf/postgreSQL/udf-join.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1790,19 +1790,19 @@ SELECT udf(udf('')) AS `xxx`, udf(i), udf(j), udf(t), udf(k)
-- !query schema
struct<xxx:string,udf(i):int,udf(j):int,udf(t):string,udf(k):int>
-- !query output
8 8 eight NULL
Copy link
Contributor Author

@karenfeng karenfeng Mar 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why these changed...

Copy link
Contributor

@cloud-fan cloud-fan Mar 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we output udf(udf(i)), udf(k), udf(t) in the SELECT list, and check if their values are changed or not?

Copy link
Contributor Author

@karenfeng karenfeng Mar 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the values didn't change, but the order did.

Copy link
Contributor Author

@karenfeng karenfeng Mar 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems wrong given that 8 should definitely not come before 4, and there is an ORDER BY term. I can't quite see how my changes triggered this though.

4 1 four NULL
NULL NULL null NULL
7 7 seven NULL
6 6 six NULL
NULL 0 zero NULL
0 NULL zero NULL
1 4 one -1
2 3 two 2
2 3 two 4
3 2 three -3
4 1 four NULL
5 0 five -5
5 0 five -5
6 6 six NULL
7 7 seven NULL
8 8 eight NULL


-- !query
Expand All @@ -1812,19 +1812,19 @@ SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t), udf(k)
-- !query schema
struct<xxx:string,udf(i):int,udf(j):int,udf(t):string,udf(k):int>
-- !query output
8 8 eight NULL
4 1 four NULL
NULL NULL null NULL
7 7 seven NULL
6 6 six NULL
NULL 0 zero NULL
0 NULL zero NULL
1 4 one -1
2 3 two 2
2 3 two 4
3 2 three -3
4 1 four NULL
5 0 five -5
5 0 five -5
6 6 six NULL
7 7 seven NULL
8 8 eight NULL


-- !query
Expand Down