Skip to content

[SPARK-32585][SQL] Support scala enumeration in ScalaReflection #29403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ulysses-you wants to merge 7 commits into from
Closed

[SPARK-32585][SQL] Support scala enumeration in ScalaReflection #29403

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
556b709
init
ulysses-you Aug 11, 2020
0f98bef
fix ut
ulysses-you Aug 11, 2020
462b0f7
fix
ulysses-you Aug 12, 2020
623a781
null
ulysses-you Sep 27, 2020
68c163a
ExpressionEncoderSuite
ulysses-you Sep 27, 2020
ed81055
typo
ulysses-you Sep 27, 2020
2d675ca
mirror
ulysses-you Sep 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
null
  • Loading branch information
@ulysses-you
ulysses-you committed Sep 27, 2020
commit 623a781df860a16d51b10b858858b6ef03a81af1
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -772,7 +772,7 @@ object ScalaReflection extends ScalaReflection {
StructField(fieldName, dataType, nullable)
}), nullable = true)
case t if isSubtype(t, localTypeOf[Enumeration#Value]) =>
Schema(StringType, nullable = false)
Schema(StringType, nullable = true)
Copy link
Contributor

@tdas tdas Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we are returning the schema generated from enum as nullable, but the serializer expression is not nullable (because it does not produce null). Why is that?

Copy link
Contributor Author

@ulysses-you ulysses-you Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two side of nullable.

  • For schema, nullable means the column can be null value.
  • For returnNullable which used in Invoke and StaticInvoke means the method promise will never produce a null value.

Some code get from Invoke

def invoke(
      obj: Any,
      method: Method,
      arguments: Seq[Expression],
      input: InternalRow,
      dataType: DataType): Any = {
    val args = arguments.map(e => e.eval(input).asInstanceOf[Object])
    if (needNullCheck && args.exists(_ == null)) {
      // return null if one of arguments is null
      null
    } else {
      val ret = method.invoke(obj, args: _*)
      val boxedClass = ScalaReflection.typeBoxedJavaMapping.get(dataType)
      if (boxedClass.isDefined) {
        boxedClass.get.cast(ret)
      } else {
        ret
      }
    }
  }

Copy link
Contributor

@tdas tdas Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So effectively, in the serialized form, we are allowing nulls to be present in the column mapping to the enum field. But if there is indeed a row with a null in that column and we attempt to deserialize that rows, then it will cause a runtime failure...isnt it?

If this understanding is correct, then serializing will never produce null, and even if there is a null, serializing will fail. Then why keep this column nullable = true? I am not necessarily opposed to it, I am just trying to understand the rationale. cc @marmbrus

Copy link
Contributor Author

@ulysses-you ulysses-you Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if there is indeed a row with a null in that column and we attempt to deserialize that rows, then it will cause a runtime failure...isnt it

Actually we will get null back. It's a trick that if input value is null the serializing will return null.

case other =>
throw new UnsupportedOperationException(s"Schema for type $other is not supported")
}
Expand Down
6 changes: 6 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1932,6 +1932,12 @@ class DatasetSuite extends QueryTest
Seq(FooClassWithEnum(1, FooEnum.E1), FooClassWithEnum(2, FooEnum.E2)).toDS(),
Seq(FooClassWithEnum(1, FooEnum.E1), FooClassWithEnum(2, FooEnum.E2)): _*
Copy link
Contributor

@marmbrus marmbrus Sep 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to the above comment, I would add a test case where the enum value is null.

Copy link
Contributor Author

@ulysses-you ulysses-you Sep 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the null test.

)

// test null
checkDataset(
Seq(FooClassWithEnum(1, null), FooClassWithEnum(2, FooEnum.E2)).toDS(),
Seq(FooClassWithEnum(1, null), FooClassWithEnum(2, FooEnum.E2)): _*
)
}
}

Expand Down