Skip to content

[SPARK-41630][SQL] Support implicit lateral column alias resolution on Project #38776

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
anchovYu wants to merge 18 commits into from
Closed

[SPARK-41630][SQL] Support implicit lateral column alias resolution on Project #38776

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
04959c2
refactor analyzer adding a new object
anchovYu Nov 23, 2022
6f44c85
lca code
anchovYu Nov 23, 2022
725e5ac
add tests, refine logic
anchovYu Nov 28, 2022
660e1d2
move lca rule to a new file
anchovYu Nov 28, 2022
fd06094
rename conf
anchovYu Nov 28, 2022
7d4f80f
test failure
anchovYu Nov 29, 2022
b9704d5
small fix
anchovYu Nov 29, 2022
97ee293
Merge remote-tracking branch 'apache/master' into SPARK-27561-refactor
anchovYu Dec 1, 2022
5785943
make changes to accomodate the recent refactor
anchovYu Dec 2, 2022
757cffb
introduce leaf exp in Project as well
anchovYu Dec 5, 2022
29de892
handle a corner case
anchovYu Dec 5, 2022
72991c6
add more tests; add check rule
anchovYu Dec 6, 2022
d45fe31
uplift the necessity to resolve expression in second phase; add more …
anchovYu Dec 8, 2022
1f55f73
address comments to add tests for LCA off
anchovYu Dec 8, 2022
f753529
revert the refactor, split LCA into two rules
anchovYu Dec 9, 2022
b9f706f
better refactor
anchovYu Dec 9, 2022
94d5c9e
address comments
anchovYu Dec 9, 2022
8d20986
address comments
anchovYu Dec 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
small fix
  • Loading branch information
@anchovYu
anchovYu committed Nov 29, 2022
commit b9704d5428fa2f25de9b6da076c972168ee0477d
6 changes: 3 additions & 3 deletions ...yst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveLateralColumnAlias.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ object ResolveLateralColumnAlias extends Rule[LogicalPlan] {
val prevAliases = aliasMap.getOrElse(a.name, Seq.empty[AliasEntry])
aliasMap += (a.name -> (prevAliases :+ AliasEntry(a, idx)))
}
def lookUpLCA(e: Expression): Option[AliasEntry] = {
var matchedLCA: Option[AliasEntry] = None
def lookUpLCA(e: Expression): Seq[AliasEntry] = {
var matchedLCA: Seq[AliasEntry] = Seq.empty[AliasEntry]
e.transformWithPruning(_.containsPattern(UNRESOLVED_ATTRIBUTE)) {
case u: UnresolvedAttribute if aliasMap.contains(u.nameParts.head) &&
Analyzer.resolveExpressionByPlanChildren(u, p, resolver)
Expand All @@ -75,7 +75,7 @@ object ResolveLateralColumnAlias extends Rule[LogicalPlan] {
val referencedAlias = aliases.head
// Only resolved alias can be the lateral column alias
if (referencedAlias.alias.resolved) {
matchedLCA = Some(referencedAlias)
matchedLCA :+= referencedAlias
}
}
u
Expand Down