Skip to content

[SPARK-16381][SQL][SparkR] Update SQL examples and programming guide for R language binding #14082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
keypointt wants to merge 13 commits into from
Closed

[SPARK-16381][SQL][SparkR] Update SQL examples and programming guide for R language binding #14082

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
e7def7c
[SPARK-16381] move example code to a separate R file
keypointt Jul 6, 2016
1af09f3
[SPARK-16381] some fixes, more to come
keypointt Jul 8, 2016
9ac6a70
[SPARK-16381] make schema merge example runnable
keypointt Jul 8, 2016
05ee46b
[SPARK-16381] make sql_query example runnable
keypointt Jul 8, 2016
828b2cf
[SPARK-16381] make load_programmatically example runnable
keypointt Jul 8, 2016
7dca42d
[SPARK-16381] replace last showDF()
keypointt Jul 8, 2016
34ca57c
Merge branch 'master' into SPARK-16381
keypointt Jul 8, 2016
af2365c
Merge branch 'SPARK-16381' of https://github.com/keypointt/spark into…
keypointt Jul 8, 2016
cd184b3
[SPARK-16381] minor fix
keypointt Jul 8, 2016
5e95fdd
[SPARK-16381] make it verbose
keypointt Jul 8, 2016
d5b0b7f
[SPARK-16381] remove code duplicate etc
keypointt Jul 8, 2016
a1eca2b
[SPARK-16381] style fix
keypointt Jul 8, 2016
7195750
[SPARK-16381] fix space style in other r examples
keypointt Jul 8, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-16381] make sql_query example runnable
  • Loading branch information
@keypointt
keypointt committed Jul 8, 2016
commit 05ee46bc46ddcb6855ab85ea79f256b1d6d27b90
5 changes: 5 additions & 0 deletions examples/src/main/r/RSparkSQLExample.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,11 @@ head(count(groupBy(df, "age")))
# $example off:dataframe_operations$


# Create a DataFrame from json file
path <- file.path(Sys.getenv("SPARK_HOME"), "examples/src/main/resources/people.json")
Copy link
Contributor

@shivaram shivaram Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use the same df from before or do we need to create a new one here ?

Copy link
Contributor Author

@keypointt keypointt Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can use the df from before, basically peopleDF here is the same as df before

but right after this line is df <- sql("SELECT * FROM table"), I think it's better use another name in this example for clarity.

what do you think?

Copy link
Contributor

@shivaram shivaram Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I see. since this is a select * query the contents are not changing. Would something like

createOrReplaceTempView(df, "table")
df <- sql("SELECT * FROM table")

work ?

Copy link
Contributor Author

@keypointt keypointt Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is great, I'll do it

peopleDF <- read.json(path)
# Register this DataFrame as a table.
createOrReplaceTempView(peopleDF, "table")
# $example on:sql_query$
df <- sql("SELECT * FROM table")
Copy link
Member

@felixcheung felixcheung Jul 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this example line is fine in doc but it won't run in an example R file - I think it does illustrate how to run a SQL query but there is no setup to create a temp view table before using it...

Copy link
Contributor Author

@keypointt keypointt Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here should I add more to create the table? or just leave it since it's only for demonstration purpose?

Copy link
Contributor

@shivaram shivaram Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets register df from above using createExternalTable and then run the query. We should aim for a case where this R file should be executable on its own

# $example off:sql_query$
Expand Down