Skip to content

Commit 2816017

Browse files
daijycsrowen
authored andcommitted
[SPARK-37556][SQL] Deser void class fail with Java serialization
**What changes were proposed in this pull request?** Change the deserialization mapping for primitive type void. **Why are the changes needed?** The void primitive type in Scala should be classOf[Unit] not classOf[Void]. Spark erroneously [map it](https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala#L80) differently than all other primitive types. Here is the code: ``` private object JavaDeserializationStream { val primitiveMappings = Map[String, Class[_]]( "boolean" -> classOf[Boolean], "byte" -> classOf[Byte], "char" -> classOf[Char], "short" -> classOf[Short], "int" -> classOf[Int], "long" -> classOf[Long], "float" -> classOf[Float], "double" -> classOf[Double], "void" -> classOf[Void] ) } ``` Spark code is Here is the demonstration: ``` scala> classOf[Long] val res0: Class[Long] = long scala> classOf[Double] val res1: Class[Double] = double scala> classOf[Byte] val res2: Class[Byte] = byte scala> classOf[Void] val res3: Class[Void] = class java.lang.Void <--- this is wrong scala> classOf[Unit] val res4: Class[Unit] = void <---- this is right ``` It will result in Spark deserialization error if the Spark code contains void primitive type: `java.io.InvalidClassException: java.lang.Void; local class name incompatible with stream class name "void"` **Does this PR introduce any user-facing change?** no **How was this patch tested?** Changed test, also tested e2e with the code results deserialization error and it pass now. Closes #34816 from daijyc/voidtype. Authored-by: Daniel Dai <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit fb40c0e) Signed-off-by: Sean Owen <[email protected]>
1 parent 3bc3b13 commit 2816017

File tree

2 files changed

+3
-3
lines changed

2 files changed

+3
-3
lines changed

core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,8 +87,8 @@ private object JavaDeserializationStream {
8787
"long" -> classOf[Long],
8888
"float" -> classOf[Float],
8989
"double" -> classOf[Double],
90-
"void" -> classOf[Void]
91-
)
90+
"void" -> classOf[Unit])
91+
9292
}
9393

9494
private[spark] class JavaSerializerInstance(

core/src/test/scala/org/apache/spark/serializer/JavaSerializerSuite.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,5 +47,5 @@ private class ContainsPrimitiveClass extends Serializable {
4747
val floatClass = classOf[Float]
4848
val booleanClass = classOf[Boolean]
4949
val byteClass = classOf[Byte]
50-
val voidClass = classOf[Void]
50+
val voidClass = classOf[Unit]
5151
}

0 commit comments

Comments
 (0)