[SPARK-9162][SQL] Implement code generation for ScalaUDF

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-9162

Currently ScalaUDF extends CodegenFallback and doesn't provide code generation implementation. This path implements code generation for ScalaUDF.

[rxin]

can we put these branches in a loop?

[viirya]

You meant using a script to generate it like the f?

[rxin]

Maybe I'm missing something, but I think you can just write a loop instead of having branches?

...
val funcClassName = callFunc.getClass.getName
...
val evals = children.map(_.gen(ctx))
...
// generate callFunc

[viirya]

Hmm, part of this can be reduced like you show. But seems we will still have a (smaller) pattern matching, at least for something like this: classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName. I will do it later.

[rxin]

For that one, can you just do callFunc.getClass.getName?

[rxin]

Why wouldn't it work? Isn't that better because we can even specialize it?

[viirya]

Hmm, you are right. I should have it a try.

[viirya]

As I tried, using function.getClass.getName causes java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Incompatible return types in ZeroArgument UDF test. If I use classOf[() => Any].getName for this zero argument function, it is ok then.

Because other tests are passed, I added a condition to take care of zero argument function now.

[viirya]

Looks like function.getClass.getName will cause other problem (java.lang.IncompatibleClassChangeError) as the following failed tests show.

[viirya]

But classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName works. So I will revert to it then.

[SparkQA]

Test build #44315 has finished for PR 9270 at commit 5e8efea.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44337 has finished for PR 9270 at commit b92ca97.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #44343 has finished for PR 9270 at commit a9f31c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

@rxin any other comments?

[viirya]

ping @rxin

[rxin]

I'm traveling right now. Will take a look when I get back.

cc @davies also.

[davies]

Why index here? All freshName will be unique.

[davies]

@viirya Could you add unit tests for every data types? it's easy to be wrong for some types.

[SparkQA]

Test build #44893 has finished for PR 9270 at commit 4a1ec67.

• This patch fails Scala style tests.
• This patch does not merge cleanly.
• This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n * public class JavaIsotonicRegressionExample\n * public class JavaNaiveBayesExample\n * case class UnresolvedStar(target: Option[Seq[String]]) extends Star with Unevaluable\n

[viirya]

@davies May I ask what the data types you meant? scala.Function0 to scala.Function22?

[davies]

It's better to save the index when you push the expression into ctx.references, otherwise it's easy to be wrong.

[viirya]

I've added that. Thanks.

[davies]

@viirya I meant the SQL data types, for example, DateType, ArrayType, StructType and so on.

[SparkQA]

Test build #44897 has finished for PR 9270 at commit abb3750.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class UnresolvedStar(target: Option[Seq[String]]) extends Star with Unevaluable\n

[SparkQA]

Test build #44898 has finished for PR 9270 at commit 6b94a38.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

@viirya Do you have time to add more tests for this?

[viirya]

@davies yes. I will do it soon.

[SparkQA]

Test build #45199 has finished for PR 9270 at commit b355895.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * final class ShuffleSortDataFormat extends SortDataFormat<PackedRecordPointer, LongArray>\n * final class UnsafeSortDataFormat extends SortDataFormat<RecordPointerAndKeyPrefix, LongArray>\n

[davies]

@viirya This patch looks good to me now. Another thing is that we can generate code for catalyst converter now (see RowEncoder), and we have a plan to replace catalyst converter with the generated one. Do you want to try it in this PR or leave that as a follow up one?

cc @marmbrus

[viirya]

@davies I think it would be better to do that in a follow up pr. Thanks.

[davies]

LGTM, merging this into master and 1.6 branch.