Skip to content

[SPARK-22771][SQL] Concatenate binary inputs into a binary output #19977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
maropu wants to merge 16 commits into from
Closed

[SPARK-22771][SQL] Concatenate binary inputs into a binary output #19977

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
628165a
Concat binary as binary
maropu Dec 14, 2017
b87e61e
Fix
maropu Dec 14, 2017
4f8a762
Add entry in Migration Guide
maropu Dec 15, 2017
583bc5d
Add option to disable binary mode
maropu Dec 15, 2017
1b57428
Address reviews
maropu Dec 20, 2017
5a771f0
Fix
maropu Dec 20, 2017
8faebdb
Apply more comments
maropu Dec 21, 2017
de2f808
Fix
maropu Dec 22, 2017
9ddb231
Fix
maropu Dec 22, 2017
766e0e6
Update docs
maropu Dec 23, 2017
fbe266c
Fix optimizer issues
maropu Dec 23, 2017
179c6fd
Fix
maropu Dec 28, 2017
1c94418
Fix
maropu Dec 28, 2017
1e13b70
Fix
maropu Dec 28, 2017
57a9d1e
Fix
maropu Dec 29, 2017
b9febbd
Fix
maropu Dec 29, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Fix
  • Loading branch information
@maropu
maropu committed Dec 29, 2017
commit b9febbdd928c4fa2cba29eaaef85ffcc173c1b44
12 changes: 7 additions & 5 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -659,12 +659,14 @@ object CombineConcats extends Rule[LogicalPlan] {
Concat(flattened)
}

private def hasNestedConcats(concat: Concat): Boolean = concat.children.exists {
case c: Concat => true
case c @ Cast(Concat(children), StringType, _) => true
case _ => false
}

def apply(plan: LogicalPlan): LogicalPlan = plan.transformExpressionsDown {
case concat: Concat if concat.children.exists {
case c: Concat => true
case c @ Cast(Concat(children), StringType, _) => true
case _ => false
} =>
case concat: Concat if hasNestedConcats(concat) =>
flattenConcats(concat)
}
}