Skip to content

[SPARK-24605][SQL] size(null) returns null instead of -1 #21598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 12 commits into from
Closed

[SPARK-24605][SQL] size(null) returns null instead of -1 #21598

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
2338b78
New implementation of size returns null for null input
MaxGekk Jun 20, 2018
e18568f
Test for legacy and new implementation of size()
MaxGekk Jun 20, 2018
64dc3b2
Call single param constructor
MaxGekk Jun 20, 2018
b6539a5
Fix expression tests
MaxGekk Jun 20, 2018
80114b3
isPublic flag is reverted back for the global config
MaxGekk Jun 20, 2018
5d58d82
Replacing config name by var name in SQLConf
MaxGekk Jun 20, 2018
431fca8
Folding Array and Map cases to one, throwing exception for unsupporte…
MaxGekk Jun 20, 2018
64a1b7d
Changing config description
MaxGekk Jun 20, 2018
10e7a2d
Updated description and added a couple examples
MaxGekk Jun 26, 2018
cf9364d
Unifying code for legacy and new path
MaxGekk Jun 26, 2018
bb21e13
Merge remote-tracking branch 'origin/master' into size-of-null2
MaxGekk Jun 26, 2018
ae2c7f1
Fix merge conflicts
MaxGekk Jun 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix expression tests
  • Loading branch information
@MaxGekk
MaxGekk committed Jun 20, 2018
commit b6539a5a8fe45c81cac506caba8c438269dad881
30 changes: 21 additions & 9 deletions ...src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,25 +24,37 @@ import org.apache.spark.sql.types._

class CollectionExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {

test("Array and Map Size") {
def testSize(legacySizeOfNull: Boolean, sizeOfNull: Any): Unit = {
val a0 = Literal.create(Seq(1, 2, 3), ArrayType(IntegerType))
val a1 = Literal.create(Seq[Integer](), ArrayType(IntegerType))
val a2 = Literal.create(Seq(1, 2), ArrayType(IntegerType))

checkEvaluation(Size(a0), 3)
checkEvaluation(Size(a1), 0)
checkEvaluation(Size(a2), 2)
checkEvaluation(Size(a0, legacySizeOfNull), 3)
checkEvaluation(Size(a1, legacySizeOfNull), 0)
checkEvaluation(Size(a2, legacySizeOfNull), 2)

val m0 = Literal.create(Map("a" -> "a", "b" -> "b"), MapType(StringType, StringType))
val m1 = Literal.create(Map[String, String](), MapType(StringType, StringType))
val m2 = Literal.create(Map("a" -> "a"), MapType(StringType, StringType))

checkEvaluation(Size(m0), 2)
checkEvaluation(Size(m1), 0)
checkEvaluation(Size(m2), 1)
checkEvaluation(Size(m0, legacySizeOfNull), 2)
checkEvaluation(Size(m1, legacySizeOfNull), 0)
checkEvaluation(Size(m2, legacySizeOfNull), 1)

checkEvaluation(
Size(Literal.create(null, MapType(StringType, StringType)), legacySizeOfNull),
expected = sizeOfNull)
checkEvaluation(
Size(Literal.create(null, ArrayType(StringType)), legacySizeOfNull),
expected = sizeOfNull)
}

test("Array and Map Size - legacy") {
testSize(legacySizeOfNull = true, sizeOfNull = -1)
}

checkEvaluation(Size(Literal.create(null, MapType(StringType, StringType))), -1)
checkEvaluation(Size(Literal.create(null, ArrayType(StringType))), -1)
test("Array and Map Size") {
testSize(legacySizeOfNull = false, sizeOfNull = null)
}

test("MapKeys/MapValues") {
Expand Down