Skip to content

[SPARK-23915][SQL] Add array_except function #21103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kiszk wants to merge 29 commits into from
Closed

[SPARK-23915][SQL] Add array_except function #21103

Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
491631d
initial commit
kiszk Apr 18, 2018
e746948
fix pyspark test failure
kiszk Apr 19, 2018
a0bc356
address review comments in #21061
kiszk Apr 19, 2018
0334407
fix style error
kiszk Apr 19, 2018
08c3670
fix pyspark test failure
kiszk Apr 20, 2018
d989fd7
fix failures
kiszk Apr 20, 2018
71374a8
update
kiszk Jul 14, 2018
bfd0509
update
kiszk Jul 19, 2018
7966cc3
address review comment
kiszk Jul 21, 2018
c68a75f
fix scala style
kiszk Jul 21, 2018
325e220
address review comments
kiszk Jul 24, 2018
27d0f7a
address review comment
kiszk Jul 26, 2018
02b809b
address review comments
kiszk Jul 26, 2018
0d24fa9
use better method name
kiszk Jul 27, 2018
5115a49
address review comment
kiszk Jul 27, 2018
e038766
address review comments
kiszk Jul 27, 2018
447d274
add a missing file
kiszk Jul 27, 2018
0b0a2ae
rebase with master
kiszk Jul 27, 2018
d8e5d79
fix scala style error
kiszk Jul 27, 2018
4a70e0e
fix compilation error
kiszk Jul 27, 2018
548f336
drop unused code
kiszk Jul 27, 2018
f4d4f8c
update comment
kiszk Jul 27, 2018
6ef1f22
update hasher
kiszk Jul 27, 2018
3639b5b
optimize nullchecks in generated code
kiszk Jul 29, 2018
4d943c8
address review comment
kiszk Jul 31, 2018
49b5ab3
address review comment
kiszk Jul 31, 2018
282445c
review address comment
kiszk Jul 31, 2018
72ef664
fix pyspark test failure
kiszk Jul 31, 2018
93e7979
address array overflow
kiszk Jul 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address review comment
  • Loading branch information
@kiszk
kiszk committed Jul 31, 2018
commit 49b5ab371af9783be8f2d6351cf664a769957a4e
27 changes: 12 additions & 15 deletions ...alyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4081,11 +4081,11 @@ case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
val value = ctx.freshName("value")
val hsValue = ctx.freshName("hsValue")
val size = ctx.freshName("size")
val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
getter, setter, javaTypeName, primitiveTypeName, arrayDataBuilder) =
if (elementTypeSupportEquals) {
val ptName = CodeGenerator.primitiveTypeName(elementType)
val unsafeArray = ctx.freshName("unsafeArray")
if (elementTypeSupportEquals) {
val ptName = CodeGenerator.primitiveTypeName(elementType)
val unsafeArray = ctx.freshName("unsafeArray")
val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
getter, setter, javaTypeName, primitiveTypeName, arrayDataBuilder) =
elementType match {
case BooleanType | ByteType | ShortType | IntegerType =>
Copy link
Contributor

@cloud-fan cloud-fan Jul 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

technically we can have special handling for boolean type(at most 2 outputs), but I think it's very rare to see boolean type in array_except. Let's leave it unless users require it.

Copy link
Member Author

@kiszk kiszk Jul 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agree that it is very rare. Let us remove the special handling code for BooleanType.

("$mcI$sp", "Int", "int",
Expand Down Expand Up @@ -4120,13 +4120,8 @@ case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
s"get($i, null)", s"update($pos, $value)", "Object", "Ref",
s"${ev.value} = new $genericArrayData(new Object[$size]);")
}
} else {
("", "", "", "", "", "", "", "", "")
}

nullSafeCodeGen(ctx, ev, (array1, array2) => {
if (openHashElementType != "") {
// Here, we ensure elementTypeSupportEquals is true
nullSafeCodeGen(ctx, ev, (array1, array2) => {
val notFoundNullElement = ctx.freshName("notFoundNullElement")
val nullElementIndex = ctx.freshName("nullElementIndex")
val builder = ctx.freshName("builder")
Expand Down Expand Up @@ -4184,7 +4179,7 @@ case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
|$hsJavaTypeName $hsValue = $genHsValue;
|if (!$hs.contains($hsValue)) {
| $hs.add$postFix($hsValue);
| $builder.$$plus$$eq($value);
| $builder.$$plus$$eq($value);
| $size++;
|}
""".stripMargin
Expand Down Expand Up @@ -4240,11 +4235,13 @@ case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
|}
|${buildResultArrayData(nonNullArrayDataBuild)}
""".stripMargin
} else {
})
} else {
nullSafeCodeGen(ctx, ev, (array1, array2) => {
val expr = ctx.addReferenceObj("arrayExceptExpr", this)
s"${ev.value} = ($arrayData)$expr.nullSafeEval($array1, $array2);"
}
})
})
}
}

override def prettyName: String = "array_except"
Expand Down