[SPARK-32451][R] Support Apache Arrow 1.0.0 #29252

dongjoon-hyun · 2020-07-27T00:05:19Z

What changes were proposed in this pull request?

Currently, GitHub Action is broken due to SparkR UT failure from new Apache Arrow 1.0.0.

This PR aims to update R code according to Apache Arrow 1.0.0 recommendation to pass R unit tests.

An alternative is pinning Apache Arrow version at 0.17.1 and I also created a PR to compare with this.

[SPARK-32451][R][TESTS] Pin Apache Arrow to 0.17.1 in GitHubAction/AppVeyor SparkR tests #29251

Why are the changes needed?

Apache Spark 3.1 supports Apache Arrow 0.15.1+.
Apache Arrow released 1.0.0 a few days ago and this causes GitHub Action SparkR test failures due to warnings.
- https://github.com/apache/spark/commits/master

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins ([SPARK-32451][R] Support Apache Arrow 1.0.0 #29252 (comment))
Pass the GitHub (https://github.com/apache/spark/runs/912656867)

dongjoon-hyun · 2020-07-27T00:11:33Z

Hi, @HyukjinKwon .
Could you review this?
This will recover GitHub Action.

dongjoon-hyun · 2020-07-27T00:56:22Z

appveyor.yml

  # This environment variable works around to test SparkR against a higher version.
  R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
+  # AppVeyor doesn't have python3 yet
+  PYSPARK_PYTHON: python


I piggy-backed this because I want to make AppVeyor success in this PR.

hm, so AppVeyor was broken now?

Yes. It has been broken for a while, so the recent Python3 change is not discovered correctly.

SparkQA · 2020-07-27T01:01:52Z

Test build #126593 has finished for PR 29252 at commit 2df5caf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-27T01:09:47Z

Hi, @viirya . Could you review this PR?

SparkQA · 2020-07-27T01:22:56Z

Test build #126595 has finished for PR 29252 at commit caacdee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2020-07-27T01:36:06Z

R/pkg/R/DataFrame.R

                output <- tryCatch({
                  doServerAuth(conn, authSecret)
-                  arrowTable <- arrow::read_arrow(readRaw(conn))
+                  arrowTable <- arrow::read_ipc_stream(readRaw(conn))


This looks good.

Quick question. When was this API added? R side currently supports Arrow 0.15.1+.

Jenkins has old Arrow, @HyukjinKwon , and this passed Jenkins~

Please note that the failure didn't occurred in Jenkins environment. Only GitHub Action and AppVeyor failed so far.

Oh.. Jenkins doesn't have arrow? Let me check then.

Ah, right, this was added since 0.17.0.

Oh.. It's not 0.15. Got it.

For this part, if we are not bump up our minimum, we need to have if .. else. Please make a follow up. Thanks, guys.

Maybe increasing minimum Arrow version to 0.17.1?

Let me just bump up the minimal version of Arrow in SparkR at #29253. Should be fine since such minimal version bump-up is already documented.

dongjoon-hyun · 2020-07-27T01:39:40Z

Thank you for review, @viirya . For AppVeyor, it seems to have another issue. I'm thinking of excluding AppVeyor from this PR.

[INFO] ------------------------------------------------------------------------
1906[INFO] BUILD SUCCESS
1907[INFO] ------------------------------------------------------------------------
1908[INFO] Total time:  22:38 min
1909[INFO] Finished at: 2020-07-27T01:13:32Z
1910[INFO] ------------------------------------------------------------------------
1911.\bin\spark-submit2.cmd --driver-java-options "-Dlog4j.configuration=file:///%CD:\=/%/R/log4j.properties" --conf spark.hadoop.fs.defaultFS="file:///" R\pkg\tests\run-all.R
1912"Presence of build for multiple Scala versions detected ( and )."
1913"Remove one of them or, set SPARK_SCALA_VERSION= in \spark-env.cmd."

viirya · 2020-07-27T01:48:45Z

I think we could just deal with Arrow in this PR.

dongjoon-hyun · 2020-07-27T01:49:41Z

Thanks.

This reverts commit caacdee.

dongjoon-hyun · 2020-07-27T01:50:44Z

I reverted AppVeyor part and updated the PR description. Can we proceed to merge since it's already verified?

dongjoon-hyun · 2020-07-27T01:50:59Z

Oh, thank you so much, @viirya ! Merged to master.

AmplabJenkins · 2020-07-27T01:51:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126599/
Test FAILed.

### What changes were proposed in this pull request? This PR ports back #29252 to support Arrow 1.0.0. Currently, SparkR with Arrow tests fails with the latest Arrow version in branch-3.0, see https://github.com/apache/spark/pull/29460/checks?check_run_id=996972267 ### Why are the changes needed? To support higher Arrow R version with SparkR. ### Does this PR introduce _any_ user-facing change? Yes, users will be able to use SparkR with Arrow 1.0.0+. ### How was this patch tested? Manually tested, GitHub Actions will test it. Closes #29462 from HyukjinKwon/SPARK-32451-3.0. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

a

2df5caf

probot-autolabeler bot added R SQL labels Jul 27, 2020

dongjoon-hyun requested a review from HyukjinKwon July 27, 2020 00:07

dongjoon-hyun changed the title ~~[SPARK-32451][R] Support Apache Arrow 1.0.0 in testing~~ [SPARK-32451][R] Support Apache Arrow 1.0.0 Jul 27, 2020

dongjoon-hyun mentioned this pull request Jul 27, 2020

[SPARK-32448][K8S][TESTS] Use single version for exec-maven-plugin/scalatest-maven-plugin #29248

Closed

a

caacdee

probot-autolabeler bot added the INFRA label Jul 27, 2020

dongjoon-hyun commented Jul 27, 2020

View reviewed changes

viirya reviewed Jul 27, 2020

View reviewed changes

Revert "a"

af2acdf

This reverts commit caacdee.

viirya approved these changes Jul 27, 2020

View reviewed changes

dongjoon-hyun closed this in 8153f56 Jul 27, 2020

dongjoon-hyun mentioned this pull request Jul 27, 2020

[SPARK-32451][R][TESTS] Pin Apache Arrow to 0.17.1 in GitHubAction/AppVeyor SparkR tests #29251

Closed

dongjoon-hyun deleted the SPARK-ARROW branch July 27, 2020 01:53

sarutak mentioned this pull request Jul 29, 2020

[SPARK-32175][CORE] Fix the order between initialization for ExecutorPlugin and starting heartbeat thread #29002

Closed

HyukjinKwon mentioned this pull request Aug 18, 2020

[SPARK-32451][R][3.0] Support Apache Arrow 1.0.0 #29462

Closed

[SPARK-32451][R] Support Apache Arrow 1.0.0 #29252

[SPARK-32451][R] Support Apache Arrow 1.0.0 #29252

Uh oh!

Conversation

dongjoon-hyun commented Jul 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Jul 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 27, 2020

Uh oh!

dongjoon-hyun commented Jul 27, 2020

Uh oh!

SparkQA commented Jul 27, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viirya commented Jul 27, 2020

Uh oh!

dongjoon-hyun commented Jul 27, 2020

Uh oh!

dongjoon-hyun commented Jul 27, 2020

Uh oh!

dongjoon-hyun commented Jul 27, 2020

Uh oh!

AmplabJenkins commented Jul 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dongjoon-hyun commented Jul 27, 2020 •

edited

Loading

dongjoon-hyun commented Jul 27, 2020 •

edited

Loading

dongjoon-hyun commented Jul 27, 2020 •

edited

Loading