[SPARK-3855][SQL] Preserve the result attribute of python UDFs though transformations

[marmbrus]

In the current implementation it was possible for the reference to change after analysis.

[SparkQA]

QA tests have started for PR 2717 at commit 9533286.

• This patch merges cleanly.

[davies]

It's reproducable by this query:

SELECT strlen(a) FROM test WHERE strlen(a) > 1

[SparkQA]

QA tests have finished for PR 2717 at commit 9533286.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class EvaluatePython(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21485/Test PASSed.

[davies]

@marmbrus Could you add a test in pyhon/pyspark/tests.py (SQLTests) ?

[SparkQA]

QA tests have started for PR 2717 at commit 6343bcb.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2717 at commit 6343bcb.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class EvaluatePython(

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21549/Test FAILed.

[SparkQA]

QA tests have started for PR 2717 at commit 6343bcb.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2717 at commit 6343bcb.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class EvaluatePython(

[SparkQA]

QA tests have started for PR 2717 at commit 6343bcb.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2717 at commit 6343bcb.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class EvaluatePython(

[davies]

data type should be IntegerType

[SparkQA]

QA tests have started for PR 2717 at commit da14879.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2717 at commit da14879.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class EvaluatePython(

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21599/Test FAILed.

[davies]

@marmbrus LGTM, just wonder that why you do not use IntegerType as returnType in the tests? I looks wired that we compare stringType with > 1

[marmbrus]

@davies, I was lazy and didn't want to look up the syntax for that :P

You are right its a little weird, but Hive supports it.

[pwendell]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 2717 at commit da14879.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2717 at commit da14879.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class EvaluatePython(

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21609/Test FAILed.

[pwendell]

Jenkins - retest this please.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21701/
Test FAILed.

[SparkQA]

QA tests have started for PR 2717 at commit da14879.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2717 at commit da14879.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class EvaluatePython(

[pwendell]

I'd like to pull this in - is that alright @marmbrus?

[marmbrus]

Yes, please do.
On Oct 17, 2014 5:10 PM, "Patrick Wendell" [email protected] wrote:

I'd like to pull this in - is that alright @marmbrus
https://github.com/marmbrus?

—
Reply to this email directly or view it on GitHub
#2717 (comment).