Skip to content

[SPARK-23915][SQL] Add array_except function #21103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kiszk wants to merge 29 commits into from
Closed

[SPARK-23915][SQL] Add array_except function #21103

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
491631d
initial commit
kiszk Apr 18, 2018
e746948
fix pyspark test failure
kiszk Apr 19, 2018
a0bc356
address review comments in #21061
kiszk Apr 19, 2018
0334407
fix style error
kiszk Apr 19, 2018
08c3670
fix pyspark test failure
kiszk Apr 20, 2018
d989fd7
fix failures
kiszk Apr 20, 2018
71374a8
update
kiszk Jul 14, 2018
bfd0509
update
kiszk Jul 19, 2018
7966cc3
address review comment
kiszk Jul 21, 2018
c68a75f
fix scala style
kiszk Jul 21, 2018
325e220
address review comments
kiszk Jul 24, 2018
27d0f7a
address review comment
kiszk Jul 26, 2018
02b809b
address review comments
kiszk Jul 26, 2018
0d24fa9
use better method name
kiszk Jul 27, 2018
5115a49
address review comment
kiszk Jul 27, 2018
e038766
address review comments
kiszk Jul 27, 2018
447d274
add a missing file
kiszk Jul 27, 2018
0b0a2ae
rebase with master
kiszk Jul 27, 2018
d8e5d79
fix scala style error
kiszk Jul 27, 2018
4a70e0e
fix compilation error
kiszk Jul 27, 2018
548f336
drop unused code
kiszk Jul 27, 2018
f4d4f8c
update comment
kiszk Jul 27, 2018
6ef1f22
update hasher
kiszk Jul 27, 2018
3639b5b
optimize nullchecks in generated code
kiszk Jul 29, 2018
4d943c8
address review comment
kiszk Jul 31, 2018
49b5ab3
address review comment
kiszk Jul 31, 2018
282445c
review address comment
kiszk Jul 31, 2018
72ef664
fix pyspark test failure
kiszk Jul 31, 2018
93e7979
address array overflow
kiszk Jul 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update
  • Loading branch information
@kiszk
kiszk committed Jul 27, 2018
commit bfd0509b092f48d075c493b70ba021262edf66af
6 changes: 2 additions & 4 deletions core/src/main/scala/org/apache/spark/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ package object spark {
val resourceStream = Thread.currentThread().getContextClassLoader.
getResourceAsStream("spark-version-info.properties")
if (resourceStream == null) {
// throw new SparkException("Could not find spark-version-info.properties")
throw new SparkException("Could not find spark-version-info.properties")
}

try {
Expand All @@ -74,9 +74,7 @@ package object spark {
)
} catch {
case e: Exception =>
val unknownProp = "<unknown>"
(unknownProp, unknownProp, unknownProp, unknownProp, unknownProp, unknownProp)
// throw new SparkException("Error loading properties from spark-version-info.properties", e)
throw new SparkException("Error loading properties from spark-version-info.properties", e)
} finally {
if (resourceStream != null) {
try {
Expand Down
12 changes: 6 additions & 6 deletions ...alyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3652,11 +3652,6 @@ case class ArrayDistinct(child: Expression)
* Will become common base class for [[ArrayUnion]], ArrayIntersect, and [[ArrayExcept]].
*/
abstract class ArraySetLike extends BinaryArrayExpressionWithImplicitCast {
Copy link
Contributor

@cloud-fan cloud-fan Jul 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we extend ComplexTypeMergingExpression here?

Copy link
Contributor

@cloud-fan cloud-fan Jul 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the question is, shall we apply type coercion rules to these array functions?

Copy link
Contributor

@mn-mikke mn-mikke Jul 30, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure to what level Presto is considered as a reference. Just FYI these operations in Presto can accept arrays of different types.

presto:default> SELECT array_except(ARRAY[5, 1, 7], ARRAY[7.0, 1.0, 3.0]);
 _col0 
-------
 [5.0] 
(1 row)

override def dataType: DataType = {
val dataTypes = children.map(_.dataType.asInstanceOf[ArrayType])
ArrayType(elementType, dataTypes.exists(_.containsNull))
}

override def checkInputDataTypes(): TypeCheckResult = {
val typeCheckResult = super.checkInputDataTypes()
if (typeCheckResult.isSuccess) {
Expand Down Expand Up @@ -3700,7 +3695,8 @@ object ArraySetLike {
array(1, 2, 3, 5)
""",
since = "2.4.0")
case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike {
case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike
with ComplexTypeMergingExpression {
var hsInt: OpenHashSet[Int] = _
var hsLong: OpenHashSet[Long] = _

Expand Down Expand Up @@ -3982,6 +3978,10 @@ object ArrayUnion {
""",
since = "2.4.0")
case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
override def dataType: DataType = ArrayType(elementType,
left.dataType.asInstanceOf[ArrayType].containsNull &&
!right.dataType.asInstanceOf[ArrayType].containsNull)

var hsInt: OpenHashSet[Int] = _
var hsLong: OpenHashSet[Long] = _

Expand Down
5 changes: 3 additions & 2 deletions ...src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1604,8 +1604,9 @@ class CollectionExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(ArrayExcept(aa1, aa0), Seq[Seq[Int]](Seq[Int](2, 1)))

assert(ArrayExcept(a00, a01).dataType.asInstanceOf[ArrayType].containsNull === false)
assert(ArrayExcept(a00, a04).dataType.asInstanceOf[ArrayType].containsNull === true)
assert(ArrayExcept(a04, a02).dataType.asInstanceOf[ArrayType].containsNull === true)
assert(ArrayExcept(a04, a05).dataType.asInstanceOf[ArrayType].containsNull === false)
assert(ArrayExcept(a20, a21).dataType.asInstanceOf[ArrayType].containsNull === false)
assert(ArrayExcept(a20, a24).dataType.asInstanceOf[ArrayType].containsNull === true)
assert(ArrayExcept(a24, a22).dataType.asInstanceOf[ArrayType].containsNull === true)
}
}