Skip to content

[SPARK-29378][R] Upgrade SparkR to use Arrow 0.15 API #26555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 6 commits into from
Closed

[SPARK-29378][R] Upgrade SparkR to use Arrow 0.15 API #26555

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3519b23
AppVeyor test
dongjoon-hyun Nov 16, 2019
25c46a0
0.15.1 test
dongjoon-hyun Nov 16, 2019
b486093
Update API
dongjoon-hyun Nov 16, 2019
a4dc812
Revert unneeded
dongjoon-hyun Nov 16, 2019
ea5304d
Address comment
dongjoon-hyun Nov 16, 2019
96f20f8
Update RecordBatchStreamReader
dongjoon-hyun Nov 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address comment
  • Loading branch information
@dongjoon-hyun
dongjoon-hyun committed Nov 16, 2019
commit ea5304ded4e1082163adc7129b73fbc5c65ea611
3 changes: 1 addition & 2 deletions appveyor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,8 @@ install:
# Install maven and dependencies
- ps: .\dev\appveyor-install-dependencies.ps1
# Required package for R unit tests
- cmd: R -e "install.packages(c('knitr', 'rmarkdown', 'e1071', 'survival'), repos='https://cloud.r-project.org/')"
- cmd: R -e "install.packages(c('knitr', 'rmarkdown', 'e1071', 'survival', 'arrow'), repos='https://cloud.r-project.org/')"
- cmd: R -e "install.packages(c('assertthat', 'bit64', 'fs', 'purrr', 'R6', 'tidyselect'), repos='https://cloud.r-project.org/')"
Copy link
Member

@HyukjinKwon HyukjinKwon Nov 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line was to install Arrow dependencies manually (because devtools started to require testthat dependency as 2.0.0+; however, SparkR requires 1.0.2). So, I think we can remove this line too. Let me make a quick followup.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Nov 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Go ahead!

Copy link
Member

@HyukjinKwon HyukjinKwon Nov 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made a followup - #26566. For clarification It actually makes no diff but only less codes.

- cmd: R -e "install.packages('https://cran.r-project.org/src/contrib/Archive/arrow/arrow_0.15.1.tar.gz', repos=NULL, type='source')"
# Here, we use the fixed version of testthat. For more details, please see SPARK-22817.
# As of devtools 2.1.0, it requires testthat higher then 2.1.1 as a dependency. SparkR test requires testthat 1.0.2.
# Therefore, we don't use devtools but installs it directly from the archive including its dependencies.
Expand Down