Skip to content

[SPARK-22771][SQL] Concatenate binary inputs into a binary output #19977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
maropu wants to merge 16 commits into from
Closed

[SPARK-22771][SQL] Concatenate binary inputs into a binary output #19977

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
628165a
Concat binary as binary
maropu Dec 14, 2017
b87e61e
Fix
maropu Dec 14, 2017
4f8a762
Add entry in Migration Guide
maropu Dec 15, 2017
583bc5d
Add option to disable binary mode
maropu Dec 15, 2017
1b57428
Address reviews
maropu Dec 20, 2017
5a771f0
Fix
maropu Dec 20, 2017
8faebdb
Apply more comments
maropu Dec 21, 2017
de2f808
Fix
maropu Dec 22, 2017
9ddb231
Fix
maropu Dec 22, 2017
766e0e6
Update docs
maropu Dec 23, 2017
fbe266c
Fix optimizer issues
maropu Dec 23, 2017
179c6fd
Fix
maropu Dec 28, 2017
1c94418
Fix
maropu Dec 28, 2017
1e13b70
Fix
maropu Dec 28, 2017
57a9d1e
Fix
maropu Dec 29, 2017
b9febbd
Fix
maropu Dec 29, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix
  • Loading branch information
@maropu
maropu committed Dec 28, 2017
commit 1e13b702185246dc509847fc152a68cfe8f2c954
1 change: 0 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -672,7 +672,6 @@ object TypeCoercion {
p transformExpressionsUp {
// Skip nodes if unresolved or empty children
case c @ Concat(children) if !c.childrenResolved || children.isEmpty => c

case c @ Concat(children) if conf.concatBinaryAsString ||
!children.map(_.dataType).forall(_ == BinaryType) =>
val newChildren = c.children.map { e =>
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ select left(null, -2), left("abcd", -2), left("abcd", 0), left("abcd", 'a');
select right("abcd", 2), right("abcd", 5), right("abcd", '2'), right("abcd", null);
select right(null, -2), right("abcd", -2), right("abcd", 0), right("abcd", 'a');

-- turn on concatBinaryAsString
-- turn off concatBinaryAsString
set spark.sql.function.concatBinaryAsString=false;
Copy link
Member

@viirya viirya Dec 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

turn on?

Copy link
Member Author

@maropu maropu Dec 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since most of other dbms-like systems concat binary inputs as binary, IMO turning off by default is okay to me.

Copy link
Member

@viirya viirya Dec 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant you said turn on in the comment (L28).

Copy link
Member Author

@maropu maropu Dec 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh....


-- Check if catalyst combine nested `Concat`s if concatBinaryAsString=false
Expand Down