[SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils #29217

HyukjinKwon · 2020-07-24T04:09:39Z

What changes were proposed in this pull request?

This PR uses python3 instead of python3.6 executable as a fallback in IntegratedUDFTestUtils.

Why are the changes needed?

Currently, GitHub Actions skips pandas UDFs. Python 3.8 is installed explicitly but somehow python3.6 looks available in GitHub Actions build environment by default.

[info] - udf/postgreSQL/udf-case.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
...
[info] - udf/postgreSQL/udf-select_having.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
...

It was chosen as python3.6 for Jenkins to pick one Python explicitly; however, looks we're already using python3 here and there.

It will also reduce the overhead to fix when we deprecate or drop Python versions.

Does this PR introduce any user-facing change?

No, dev-only.

How was this patch tested?

It should be tested in Jenkins and GitHub Actions environments here.

…thub actions" This reverts commit 026b0b9.

…ts of github actions"" This reverts commit 826689a.

HyukjinKwon · 2020-07-24T04:59:43Z

I reverted 826689a here and get it back again to confirm IntegratedUDFTestUtils and pandas related tests are not being skipped.

HyukjinKwon · 2020-07-24T05:46:55Z

It is now tested properly: https://github.com/apache/spark/runs/905415145

...
[info] - udf/udf-inner-join.sql - Scalar Pandas UDF (527 milliseconds)
...
[info] - udf/udf-special-values.sql - Scalar Pandas UDF (747 milliseconds)

SparkQA · 2020-07-24T07:05:02Z

Test build #126472 has finished for PR 29217 at commit d72c4c1.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-24T07:05:03Z

Test build #126468 has finished for PR 29217 at commit c70e29f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-24T07:06:12Z

retest this please

SparkQA · 2020-07-24T12:07:07Z

Test build #126483 has finished for PR 29217 at commit d72c4c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-24T15:18:30Z

sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala

  lazy val pythonExec: String = {
    val pythonExec = sys.env.getOrElse(
-      "PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python3.6"))
+      "PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python3"))


Since this is GitHub Action specific issue, shall we use export PYSPARK_PYTHON in GitHub Action side?

dongjoon-hyun

We had better isolate GitHub Action specific changes into master.yml. Otherwise, we will hit this failure when we backport GitHub Action into old branches.

HyukjinKwon · 2020-07-25T04:04:23Z

I think it's okay. Installing Python 3.8 in GitHub Actions overwrites python3. So, we won't face the tests being skipped even when we backport as long as there's the explicit Python 3 installation.

I wouldn't say this is only a GitHub specific issue. It was python3.6 in the codes for Jenkins environment when I added. It was to explicitly pick a Python version but we're already using python3 to pick Python 3 in many places like our dev scripts.

It fixes the issue in GitHub Actions but also keep the codes consistent with other places. Plus, it will reduce overhead of maintenance when we drop or deprecate minor Python version.

dongjoon-hyun

Got it. +1, LGTM. Thanks for the explanation.
Merged to master.

HyukjinKwon · 2020-07-25T10:28:24Z

Thank you @dongjoon-hyun

dongjoon-hyun · 2020-07-25T20:05:28Z

Just a question. Is this used in R test, too?

HyukjinKwon · 2020-07-26T10:47:58Z

Nope it isnt. I believe this is irrelacent with R test results.

HyukjinKwon · 2020-07-26T10:48:17Z

Nope it isnt. I believe this is irrelevant with R test results.

dongjoon-hyun · 2020-07-26T15:28:01Z

Thank you for confirmation~

… in IntegratedUDFTestUtils ### What changes were proposed in this pull request? This PR uses `python3` instead of `python3.6` executable as a fallback in `IntegratedUDFTestUtils`. ### Why are the changes needed? Currently, GitHub Actions skips pandas UDFs. Python 3.8 is installed explicitly but somehow `python3.6` looks available in GitHub Actions build environment by default. ``` [info] - udf/postgreSQL/udf-case.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!! ... [info] - udf/postgreSQL/udf-select_having.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!! ... ``` It was chosen as `python3.6` for Jenkins to pick one Python explicitly; however, looks we're already using `python3` here and there. It will also reduce the overhead to fix when we deprecate or drop Python versions. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? It should be tested in Jenkins and GitHub Actions environments here. Closes apache#29217 from HyukjinKwon/SPARK-32422. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

Use python3 executable instead of python3.6 in IntegratedUDFTestUtils

c70e29f

probot-autolabeler bot added the SQL label Jul 24, 2020

HyukjinKwon changed the title ~~[SPARK-32422][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils~~ [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils Jul 24, 2020

Revert "[SPARK-32253][INFRA] Show errors only for the sbt tests of gi…

826689a

…thub actions" This reverts commit 026b0b9.

probot-autolabeler bot added the BUILD label Jul 24, 2020

Revert "Revert "[SPARK-32253][INFRA] Show errors only for the sbt tes…

d72c4c1

…ts of github actions"" This reverts commit 826689a.

dongjoon-hyun reviewed Jul 24, 2020

View reviewed changes

dongjoon-hyun approved these changes Jul 25, 2020

View reviewed changes

dongjoon-hyun closed this in 277a406 Jul 25, 2020

HyukjinKwon deleted the SPARK-32422 branch July 27, 2020 07:43

[SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils #29217

[SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils #29217

Uh oh!

Conversation

HyukjinKwon commented Jul 24, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HyukjinKwon commented Jul 24, 2020

Uh oh!

HyukjinKwon commented Jul 24, 2020

Uh oh!

SparkQA commented Jul 24, 2020

Uh oh!

SparkQA commented Jul 24, 2020

Uh oh!

HyukjinKwon commented Jul 24, 2020

Uh oh!

SparkQA commented Jul 24, 2020

Uh oh!

dongjoon-hyun Jul 24, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jul 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jul 25, 2020

Uh oh!

dongjoon-hyun commented Jul 25, 2020

Uh oh!

HyukjinKwon commented Jul 26, 2020

Uh oh!

HyukjinKwon commented Jul 26, 2020

Uh oh!

dongjoon-hyun commented Jul 26, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HyukjinKwon commented Jul 25, 2020 •

edited

Loading