[SPARK-9058][SQL] Split projectionCode if it is too large for JVM

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-9058

There is a limit for code size in a function in JVM. If the generated projectionCode is too large, we split it to multiple functions and avoid causing JVM failed.

[SparkQA]

Test build #37349 has finished for PR 7418 at commit d715fd5.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

This patch seems to be duplicated with #7076.
Why you make a new PR?

[maropu]

How did you decide this number '50' for the JVM code size limitation?

[viirya]

This is created for the new JIRA ticket. I didn't notice there is already a related one.

[SparkQA]

Test build #37363 has finished for PR 7418 at commit 8775359.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[chenghao-intel]

Make it private final? And also instead of the make the argument type as Object, can we make it as InternalRow?

[chenghao-intel]

Since the codegen aim to inline the execution, probably we'd better not to increase the overhead for type casting.
Even, we'd better to put every 50(says) expressions into a single codegen function?

[SparkQA]

Test build #37444 has finished for PR 7418 at commit 7435454.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #37461 has finished for PR 7418 at commit 12d3794.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #37464 has finished for PR 7418 at commit b8e274e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[chenghao-intel]

My concern on this fix is, that probably cause performance issue, if we add a function call for each of the projection field, as codegen aim to inline the logic into a single function.
Can you give a benchmark result for this fixing?

[chenghao-intel]

Or at least, we can group several project fields in a codegened function, and give a configuration to enable the codegen grouping.

[viirya]

@chenghao-intel This pr already groups a certain number of projection columns (now it is 50) into a function and call it. It is due to the problem caused by a very long single function inlined with too many projection columns (as reported in the JIRA ticket, it is more than 100).

If the projection columns is less than the number, it doesn't doing column grouping and works as before.

I think this should be a very rare use case. For this problem, we shouldn't add a configuration because it is not a feature or function. It is a bug fixing to make this kind of projection work.

[rxin]

@viirya let's use #7076 since it was first submitted and had a test case. If @saurfang doesn't respond, let's merge this one. Thanks.

[viirya]

@rxin sure, no problem.

[rxin]

Thanks for understanding. Do you mind closing this one first? If we need to merge this one, let's reopen it.

[viirya]

ok. no problem.

[chenghao-intel]

@viirya, I mean you at least have a single group (less than 50 fields), it probably cause performance issues for case like:
select a+b from src, as the overhead of function invoke is heavy compare to the a+b itself.

[viirya]

@chenghao-intel no, if there are less 50 columns, the generated java codes are as the same as the before. No extra function is added.

[chenghao-intel]

Oh, right! Here. Thanks for explanation.