Skip to content

[SPARK-23914][SQL] Add array_union function #21061

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kiszk wants to merge 34 commits into from
Closed

[SPARK-23914][SQL] Add array_union function #21061

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
dc9d6f0
initial commit
kiszk Apr 13, 2018
3019840
update description
kiszk Apr 13, 2018
8cee6cf
fix test failure
kiszk Apr 13, 2018
2041ec4
address review comments
kiszk Apr 17, 2018
8c2280b
introduce ArraySetUtils to reuse code among array_union/array_interse…
kiszk Apr 17, 2018
b3a3132
fix python test failure
kiszk Apr 18, 2018
a2c7dd1
fix python test failure
kiszk Apr 18, 2018
5313680
simplification
kiszk Apr 18, 2018
98f8d1f
fix pyspark test failure
kiszk Apr 19, 2018
30ee7fc
address review comments
kiszk Apr 20, 2018
cd347e9
add new tests based on review comment
kiszk Apr 20, 2018
d2eaee3
fix mistakes in rebase
kiszk Apr 20, 2018
2ddeb06
fix unexpected changes
kiszk Apr 20, 2018
71b31f0
merge changes in #21103
kiszk Apr 20, 2018
7e71340
use GenericArrayData if UnsafeArrayData cannot be used
kiszk May 4, 2018
04c97c3
use BinaryArrayExpressionWithImplicitCast
kiszk May 4, 2018
401ca7a
update test cases
kiszk May 4, 2018
15b953b
rebase with master
kiszk May 17, 2018
f050922
support complex types
kiszk May 18, 2018
8a27667
add test cases with duplication in an array
kiszk May 19, 2018
e50bc55
rebase with master
kiszk Jun 1, 2018
7e3f2ef
address review comments
kiszk Jun 1, 2018
e5401e7
address review comment
kiszk Jun 1, 2018
3e21e48
keep the order of input array elements
kiszk Jun 10, 2018
3c39506
address review comments
kiszk Jun 20, 2018
6654742
fix scala style error
kiszk Jun 20, 2018
be9f331
address review comment
kiszk Jun 20, 2018
90e84b3
address review comments
kiszk Jun 22, 2018
6f721f0
address review comments
kiszk Jun 22, 2018
0c0d3ba
address review comments
kiszk Jul 8, 2018
4a217bc
cleanup
kiszk Jul 8, 2018
f5ebbe8
eliminate duplicated code
kiszk Jul 8, 2018
763a1f8
address review comments
kiszk Jul 9, 2018
7b51564
address review comment
kiszk Jul 11, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
address review comment
  • Loading branch information
@kiszk
kiszk committed Jul 11, 2018
commit 7b515649c75dc68d5d74ebca15628b6377a05344
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -477,7 +477,7 @@ public static UnsafeArrayData forPrimitiveArray(int offset, int length, int elem
return fromPrimitiveArray(null, offset, length, elementSize);
}

public static boolean canUseGenericArrayData(int elementSize, int length) {
public static boolean shouldUseGenericArrayData(int elementSize, int length) {
final long headerInBytes = calculateHeaderPortionInBytes(length);
final long valueRegionInBytes = (long)elementSize * length;
final long totalSizeInLongs = (headerInBytes + valueRegionInBytes + 7) / 8;
Expand Down
4 changes: 2 additions & 2 deletions ...alyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3395,7 +3395,7 @@ case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike
hsInt = new OpenHashSet[Int]
val elements = evalIntLongPrimitiveType(array1, array2, null, false)
hsInt = new OpenHashSet[Int]
val resultArray = if (UnsafeArrayData.canUseGenericArrayData(
val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
IntegerType.defaultSize, elements)) {
new GenericArrayData(new Array[Any](elements))
} else {
Expand All @@ -3410,7 +3410,7 @@ case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike
hsLong = new OpenHashSet[Long]
val elements = evalIntLongPrimitiveType(array1, array2, null, true)
hsLong = new OpenHashSet[Long]
Copy link
Member

@viirya viirya Jul 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we obtain unique elements of two arrays in the hash set, can't we get final array elements from it directly instead of scanning two arrays again?

Copy link
Member Author

@kiszk kiszk Jul 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be. Originally, I took that approach.
After discussed with @ueshin, I decided to generate a result array from the original arrays instead of the hash. This is because we generate a result array in a unique deterministic order among the different paths in array_union.

Copy link
Member

@viirya viirya Jul 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, though I think there will be some performance issue.

val resultArray = if (UnsafeArrayData.canUseGenericArrayData(
val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
LongType.defaultSize, elements)) {
new GenericArrayData(new Array[Any](elements))
} else {
Expand Down