Skip to content

[SPARK-23915][SQL] Add array_except function #21103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kiszk wants to merge 29 commits into from
Closed

[SPARK-23915][SQL] Add array_except function #21103

Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
491631d
initial commit
kiszk Apr 18, 2018
e746948
fix pyspark test failure
kiszk Apr 19, 2018
a0bc356
address review comments in #21061
kiszk Apr 19, 2018
0334407
fix style error
kiszk Apr 19, 2018
08c3670
fix pyspark test failure
kiszk Apr 20, 2018
d989fd7
fix failures
kiszk Apr 20, 2018
71374a8
update
kiszk Jul 14, 2018
bfd0509
update
kiszk Jul 19, 2018
7966cc3
address review comment
kiszk Jul 21, 2018
c68a75f
fix scala style
kiszk Jul 21, 2018
325e220
address review comments
kiszk Jul 24, 2018
27d0f7a
address review comment
kiszk Jul 26, 2018
02b809b
address review comments
kiszk Jul 26, 2018
0d24fa9
use better method name
kiszk Jul 27, 2018
5115a49
address review comment
kiszk Jul 27, 2018
e038766
address review comments
kiszk Jul 27, 2018
447d274
add a missing file
kiszk Jul 27, 2018
0b0a2ae
rebase with master
kiszk Jul 27, 2018
d8e5d79
fix scala style error
kiszk Jul 27, 2018
4a70e0e
fix compilation error
kiszk Jul 27, 2018
548f336
drop unused code
kiszk Jul 27, 2018
f4d4f8c
update comment
kiszk Jul 27, 2018
6ef1f22
update hasher
kiszk Jul 27, 2018
3639b5b
optimize nullchecks in generated code
kiszk Jul 29, 2018
4d943c8
address review comment
kiszk Jul 31, 2018
49b5ab3
address review comment
kiszk Jul 31, 2018
282445c
review address comment
kiszk Jul 31, 2018
72ef664
fix pyspark test failure
kiszk Jul 31, 2018
93e7979
address array overflow
kiszk Jul 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use better method name
  • Loading branch information
@kiszk
kiszk committed Jul 27, 2018
commit 0d24fa95edf3cf8ec18a7bca34382e8aff24e505
4 changes: 2 additions & 2 deletions ...alyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4055,7 +4055,7 @@ case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
pos
}

val exceptEquals: (ArrayData, ArrayData) => ArrayData = {
val evalExcept: (ArrayData, ArrayData) => ArrayData = {
(array1: ArrayData, array2: ArrayData) =>
if (elementTypeSupportEquals) {
elementType match {
Expand Down Expand Up @@ -4176,7 +4176,7 @@ case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
val array1 = input1.asInstanceOf[ArrayData]
val array2 = input2.asInstanceOf[ArrayData]

exceptEquals(array1, array2)
evalExcept(array1, array2)
}

override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
Copy link
Contributor

@cloud-fan cloud-fan Jul 26, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think codegen can provide much value, given the complexity of this function. We do codegen mostly for supporting whole-stage-codegen. How about

val expr = ctx.addReference("arrayExceptExpr", this)
nullSafeCodeGen(ctx, ev, (array1, array2) => {
  s"$expr.nullSafeEval($array1, $array2)"
} 
"""

cc @rednaxelafx @rxin

Copy link
Member Author

@kiszk kiszk Jul 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seem to make sense. Codegen can only increase the number of supporting types (i.e. byte/short).

Expand Down