Skip to content

[SPARK-34527][SQL] Resolve duplicated common columns from USING/NATURAL JOIN #31666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
karenfeng wants to merge 44 commits into from
Closed

[SPARK-34527][SQL] Resolve duplicated common columns from USING/NATURAL JOIN #31666

Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
1c5ab03
Only use metadata columns for resolution as last resort
karenfeng Feb 26, 2021
2fe733f
Resolve deduplicated common columns in NATURAL/USING JOIN with hidden…
karenfeng Feb 26, 2021
2c261bb
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Feb 26, 2021
80beda8
Fix behavior in Scala
karenfeng Feb 27, 2021
e1719d3
Fix nested expression
karenfeng Feb 27, 2021
6fa70ba
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 2, 2021
0ba1916
Address comments
karenfeng Mar 4, 2021
2b7e730
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 4, 2021
0c116a5
Push SQL output
karenfeng Mar 4, 2021
bf87f55
Re-do changes
karenfeng Mar 7, 2021
b5dc44f
Fixup tag
karenfeng Mar 8, 2021
6e32b3d
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 8, 2021
7c3f5df
Add another failing test
karenfeng Mar 8, 2021
181751a
Merge add metadata and resolve missing references
karenfeng Mar 29, 2021
ad5e824
Resolve down to avoid prematurely projecting out
karenfeng Mar 29, 2021
e36e853
Remove printlns
karenfeng Mar 29, 2021
db44c53
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 29, 2021
73b7c8a
Clean up
karenfeng Mar 29, 2021
1eb01e2
Style fixup
karenfeng Mar 29, 2021
9fd2490
Formatting fix
karenfeng Mar 29, 2021
f5cc3ae
Address comments
karenfeng Mar 30, 2021
fa7207e
Metadata output should be empty by default
karenfeng Mar 31, 2021
7af12ae
Clean up
karenfeng Mar 31, 2021
07f9ad5
Always resolve with metadata
karenfeng Mar 31, 2021
c474745
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 6, 2021
0f267e7
Revert accidental metadata changes
karenfeng Apr 6, 2021
ed0270c
Revert DatasourceV2SQLSuite
karenfeng Apr 6, 2021
66ad572
Revert unneeded style changes
karenfeng Apr 6, 2021
fc3b16d
Revert accidental metadata output change
karenfeng Apr 6, 2021
f665030
Add childrens' hidden output to Project metadataOutput
karenfeng Apr 7, 2021
85b81b1
Retrigger tests
karenfeng Apr 7, 2021
eab7964
Add comments
karenfeng Apr 7, 2021
44ee9f8
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 7, 2021
c84f396
Address comments and fix propagation through Projects
karenfeng Apr 7, 2021
0fe04a2
Retrigger tests
karenfeng Apr 7, 2021
c7c3df6
Retrigger tests
karenfeng Apr 7, 2021
b1bf28d
Retrigger tests
karenfeng Apr 8, 2021
47be66d
Wording
karenfeng Apr 9, 2021
8c5144e
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 9, 2021
333a815
propagate hidden columns from nested NATURAL/USING JOINs
karenfeng Apr 12, 2021
49de5c5
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 12, 2021
9e62d7d
Retrigger tests
karenfeng Apr 12, 2021
446d4bc
address comments
karenfeng Apr 13, 2021
8f70c2d
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 13, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add childrens' hidden output to Project metadataOutput 
Signed-off-by: Karen Feng <[email protected]>
  • Loading branch information
@karenfeng
karenfeng committed Apr 7, 2021
commit f665030c32af34b2d865d1c40e0da1bdc302741c
3 changes: 2 additions & 1 deletion ...st/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,8 @@ case class Project(projectList: Seq[NamedExpression], child: LogicalPlan)
getAllValidConstraints(projectList)

override def metadataOutput: Seq[Attribute] =
getTagValue(Project.hiddenOutputTag).getOrElse(Seq.empty[Attribute])
child.metadataOutput.filter(_.isHiddenCol) ++
Copy link
Contributor Author

@karenfeng karenfeng Apr 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed to propagate hidden columns through nested Projections (eg. multiple natural joins).

Copy link
Contributor

@cloud-fan cloud-fan Apr 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have an example query? It's a bit weird that metadata columns can be propagated through project.

Copy link
Contributor Author

@karenfeng karenfeng Apr 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was used in natural-join.sql:

SELECT nt1.*, nt2.*, nt3.* FROM nt1 natural join nt2 natural join nt3;

Which would otherwise fail to expand nt2.* to include the key column.

Copy link
Contributor Author

@karenfeng karenfeng Apr 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this also breaks the ordering in udf/postgreSQL/udf-join.sql.

Copy link
Contributor Author

@karenfeng karenfeng Apr 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this so hidden output are only propagated if a Project already has hidden output. This allows us to propagate hidden output through nested NATURAL/USING JOINs.

getTagValue(Project.hiddenOutputTag).getOrElse(Seq.empty[Attribute])
}

object Project {
Expand Down