Commit 2816017
[SPARK-37556][SQL] Deser void class fail with Java serialization
**What changes were proposed in this pull request?**
Change the deserialization mapping for primitive type void.
**Why are the changes needed?**
The void primitive type in Scala should be classOf[Unit] not classOf[Void]. Spark erroneously [map it](https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala#L80) differently than all other primitive types. Here is the code:
```
private object JavaDeserializationStream {
val primitiveMappings = Map[String, Class[_]](
"boolean" -> classOf[Boolean],
"byte" -> classOf[Byte],
"char" -> classOf[Char],
"short" -> classOf[Short],
"int" -> classOf[Int],
"long" -> classOf[Long],
"float" -> classOf[Float],
"double" -> classOf[Double],
"void" -> classOf[Void]
)
}
```
Spark code is Here is the demonstration:
```
scala> classOf[Long]
val res0: Class[Long] = long
scala> classOf[Double]
val res1: Class[Double] = double
scala> classOf[Byte]
val res2: Class[Byte] = byte
scala> classOf[Void]
val res3: Class[Void] = class java.lang.Void <--- this is wrong
scala> classOf[Unit]
val res4: Class[Unit] = void <---- this is right
```
It will result in Spark deserialization error if the Spark code contains void primitive type:
`java.io.InvalidClassException: java.lang.Void; local class name incompatible with stream class name "void"`
**Does this PR introduce any user-facing change?**
no
**How was this patch tested?**
Changed test, also tested e2e with the code results deserialization error and it pass now.
Closes #34816 from daijyc/voidtype.
Authored-by: Daniel Dai <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit fb40c0e)
Signed-off-by: Sean Owen <[email protected]>1 parent 3bc3b13 commit 2816017
File tree
2 files changed
+3
-3
lines changed- core/src
- main/scala/org/apache/spark/serializer
- test/scala/org/apache/spark/serializer
2 files changed
+3
-3
lines changedLines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
90 | | - | |
91 | | - | |
| 90 | + | |
| 91 | + | |
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
| 50 | + | |
51 | 51 | | |
0 commit comments