[SPARK-34596][SQL][2.4] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode

[maropu]

What changes were proposed in this pull request?

Use Utils.getSimpleName to avoid hitting Malformed class name error in NewInstance.doGenCode.

NOTE: branch-2.4 does not have the interpreted implementation of SafeProjection, so it does not fall back into the interpreted mode if the compilation fails. Therefore, the test in this PR just checks that the compilation error happens instead of checking that the interpreted mode works well.

This is the backport PR of #31709 and the credit should be @rednaxelafx .

Why are the changes needed?

On older JDK versions (e.g. JDK8u), nested Scala classes may trigger java.lang.Class.getSimpleName to throw an java.lang.InternalError: Malformed class name error.
In this particular case, creating an ExpressionEncoder on such a nested Scala class would create a NewInstance expression under the hood, which will trigger the problem during codegen.

Similar to #29050, we should use Spark's Utils.getSimpleName utility function in place of Class.getSimpleName to avoid hitting the issue.

There are two other occurrences of java.lang.Class.getSimpleName in the same file, but they're safe because they're only guaranteed to be only used on Java classes, which don't have this problem, e.g.:

    // Make a copy of the data if it's unsafe-backed
    def makeCopyIfInstanceOf(clazz: Class[_ <: Any], value: String) =
      s"$value instanceof ${clazz.getSimpleName}? ${value}.copy() : $value"
    val genFunctionValue: String = lambdaFunction.dataType match {
      case StructType(_) => makeCopyIfInstanceOf(classOf[UnsafeRow], genFunction.value)
      case ArrayType(_, _) => makeCopyIfInstanceOf(classOf[UnsafeArrayData], genFunction.value)
      case MapType(_, _, _) => makeCopyIfInstanceOf(classOf[UnsafeMapData], genFunction.value)
      case _ => genFunction.value
    }

The Unsafe-* family of types are all Java types, so they're okay.

Does this PR introduce any user-facing change?

Fixes a bug that throws an error when using ExpressionEncoder on some nested Scala types, otherwise no changes.

How was this patch tested?

Added a test case to org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite. It'll fail on JDK8u before the fix, and pass after the fix.

[maropu]

cc: @viirya @rednaxelafx

NOTE: This PR comes from the discussion with @viirya #31747 (comment)

[viirya]

@maropu Can you add few words to the PR description about why you change the test case? Thanks.

[viirya]

lgtm if we clearly state why we change the test case.

[maropu]

sure, I'll update it.

[maropu]

NOTE: branch-2.4 does not have the interpreted implementation of SafeProjection, so it does not fall back into 
the interpreted mode if the compilation fails. Therefore, the test in this PR just checks that the compilation 
error happens instead of checking that the interpreted mode works well.

@viirya okay, I updated it and please check it.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40857/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40857/

[SparkQA]

Test build #136275 has finished for PR 31888 at commit 5152fff.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

Is this the compilation error?

[info]   Cause: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 32, Column 124: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 32, Column 124: "org.apache.
spark.sql.catalyst.encoders.ExpressionEncoderSuite$MalformedClassObject$" has no member type "ExpressionEncoderSuite$MalformedClassObject$MalformedNameExample"

[viirya]

But by moving MalformedClassObject out of ExpressionEncoderSuite, the generated code can be compiled.

I'm not sure what this test wants to test actually. Am I confused or do I misunderstand it?

[maropu]

Is this the compilation error?

Yea, that's the expected error. Actually, the master has same error when useFallback = false:

spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala

Line 219 in d32bb4e

useFallback = true)

// master
[info]   Cause: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 32, Column 124: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 32, Column 124: "org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$MalformedClassObject$" has no member type "ExpressionEncoderSuite$MalformedClassObject$MalformedNameExample"
[info]   at org.apache.spark.sql.errors.QueryExecutionErrors$.compilerError(QueryExecutionErrors.scala:328)

But by moving MalformedClassObject out of ExpressionEncoderSuite, the generated code can be compiled.

IIUC the condition where the error happens is that MalformedClassObject is placed in ExpressionEncoderSuite as Kris said in the previous PR: #31709 (comment)

Actually, what's more interesting is the different rules of Java's nested classes and Scala's. Since there is no such
 notion of a "companion object" or built-in "singleton object" notion on the Java language level, at least not in 
Java 8, the Java language syntax for creating a new instance of a inner class object doesn't work well for Scala's
 class Foo { object Bar { class Baz {...} } } kind of nesting, i.e. even when we do use the correct "simple name" 
(e.g. MalformedNameExample), there would still be no valid Java syntax to create it (i.e. outerObj.new 
InnerClass(...) doesn't work when outerObj is a Scala object)

cc: @rednaxelafx

[viirya]

Got it. Thanks for the reference.

[viirya]

cc @cloud-fan @dongjoon-hyun too

[viirya]

I will merge this tomorrow if no more comment.

[viirya]

Thanks all. Merging to 2.4.