Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
1c5ab03
Only use metadata columns for resolution as last resort
karenfeng Feb 26, 2021
2fe733f
Resolve deduplicated common columns in NATURAL/USING JOIN with hidden…
karenfeng Feb 26, 2021
2c261bb
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Feb 26, 2021
80beda8
Fix behavior in Scala
karenfeng Feb 27, 2021
e1719d3
Fix nested expression
karenfeng Feb 27, 2021
6fa70ba
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 2, 2021
0ba1916
Address comments
karenfeng Mar 4, 2021
2b7e730
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 4, 2021
0c116a5
Push SQL output
karenfeng Mar 4, 2021
bf87f55
Re-do changes
karenfeng Mar 7, 2021
b5dc44f
Fixup tag
karenfeng Mar 8, 2021
6e32b3d
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 8, 2021
7c3f5df
Add another failing test
karenfeng Mar 8, 2021
181751a
Merge add metadata and resolve missing references
karenfeng Mar 29, 2021
ad5e824
Resolve down to avoid prematurely projecting out
karenfeng Mar 29, 2021
e36e853
Remove printlns
karenfeng Mar 29, 2021
db44c53
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Mar 29, 2021
73b7c8a
Clean up
karenfeng Mar 29, 2021
1eb01e2
Style fixup
karenfeng Mar 29, 2021
9fd2490
Formatting fix
karenfeng Mar 29, 2021
f5cc3ae
Address comments
karenfeng Mar 30, 2021
fa7207e
Metadata output should be empty by default
karenfeng Mar 31, 2021
7af12ae
Clean up
karenfeng Mar 31, 2021
07f9ad5
Always resolve with metadata
karenfeng Mar 31, 2021
c474745
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 6, 2021
0f267e7
Revert accidental metadata changes
karenfeng Apr 6, 2021
ed0270c
Revert DatasourceV2SQLSuite
karenfeng Apr 6, 2021
66ad572
Revert unneeded style changes
karenfeng Apr 6, 2021
fc3b16d
Revert accidental metadata output change
karenfeng Apr 6, 2021
f665030
Add childrens' hidden output to Project metadataOutput
karenfeng Apr 7, 2021
85b81b1
Retrigger tests
karenfeng Apr 7, 2021
eab7964
Add comments
karenfeng Apr 7, 2021
44ee9f8
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 7, 2021
c84f396
Address comments and fix propagation through Projects
karenfeng Apr 7, 2021
0fe04a2
Retrigger tests
karenfeng Apr 7, 2021
c7c3df6
Retrigger tests
karenfeng Apr 7, 2021
b1bf28d
Retrigger tests
karenfeng Apr 8, 2021
47be66d
Wording
karenfeng Apr 9, 2021
8c5144e
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 9, 2021
333a815
propagate hidden columns from nested NATURAL/USING JOINs
karenfeng Apr 12, 2021
49de5c5
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 12, 2021
9e62d7d
Retrigger tests
karenfeng Apr 12, 2021
446d4bc
address comments
karenfeng Apr 13, 2021
8f70c2d
Merge branch 'master' of https://github.com/apache/spark into spark-3…
karenfeng Apr 13, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add comments
Signed-off-by: Karen Feng <[email protected]>
  • Loading branch information
karenfeng committed Apr 7, 2021
commit eab7964456ad3d3178a550a072268bf98e08c5a3
Original file line number Diff line number Diff line change
Expand Up @@ -961,6 +961,8 @@ class Analyzer(override val catalogManager: CatalogManager)
lazy val childMetadataOutput = plan.children.flatMap(_.metadataOutput)
val hasMetaCol = plan.expressions.exists(_.find {
case a: Attribute =>
// If an attribute is resolved before being labeled as metadata
// (i.e. from the originating Dataset), we check with expression ID
a.isMetadataCol || childMetadataOutput.exists(_.exprId == a.exprId)
case _ => false
}.isDefined)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ package object util extends Logging {
val METADATA_COL_ATTR_KEY = "__metadata_col"

/**
* Hidden columns are a type of metadata column that are not propagated through subquery aliases,
* Hidden columns are a type of metadata column that are propagated through Project
* and are candidates during qualified star expansions.
*/
val HIDDEN_COL_ATTR_KEY = "__hidden_col"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The semantic is clear now, let's refine the naming.

We only have metadata column, and metadata column can be included in qualified star if required. We can just add a new property to metadata columns to indicate it.

The property name can be __support_qualified_star, and the helper class can be

implicit class MetadataColumnHelper(attr: Attribute) {
  def isMetadataCol: Boolean ...
  def supportQualifiedStar: Boolean ...
  def markAsSupportQualifiedStar: Attribute ...
}

Expand Down