Skip to content

[SPARK-25935][SQL] Prevent null rows from JSON parser #22938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 9 commits into from
Closed

[SPARK-25935][SQL] Prevent null rows from JSON parser #22938

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
31cb534
Eliminate producing nulls by JSON parser
MaxGekk Nov 3, 2018
0589d91
Updating the migration guide
MaxGekk Nov 4, 2018
0aa4f60
Addressing Hyukjin's review comments
MaxGekk Nov 4, 2018
d1bad7c
Removing not relevant test
MaxGekk Nov 4, 2018
c4d6a80
fix for R test
MaxGekk Nov 4, 2018
a7e016a
Addressing Attila's review comments
MaxGekk Nov 6, 2018
35b3013
Merge remote-tracking branch 'origin/master' into json-nulls
MaxGekk Nov 11, 2018
6a8cac3
Updating the migration guide
MaxGekk Nov 21, 2018
1ef2d5b
Taking into account JSON datasource behavior
MaxGekk Nov 21, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix for R test
  • Loading branch information
@MaxGekk
MaxGekk committed Nov 4, 2018
commit c4d6a8066031c4f1b0f9323f9998f0f0b10b74c7
2 changes: 1 addition & 1 deletion R/pkg/tests/fulltests/test_sparkSQL.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1674,7 +1674,7 @@ test_that("column functions", {

# check for unparseable
df <- as.DataFrame(list(list("a" = "")))
expect_equal(collect(select(df, from_json(df$a, schema)))[[1]][[1]], NA)
expect_equal(collect(select(df, from_json(df$a, schema)))[[1]][[1]]$a, NA)

# check if array type in string is correctly supported.
jsonArr <- "[{\"name\":\"Bob\"}, {\"name\":\"Alice\"}]"
Expand Down