Skip to content

[SPARK-19949][SQL] unify bad record handling in CSV and JSON #17315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cloud-fan wants to merge 8 commits into from
Closed

[SPARK-19949][SQL] unify bad record handling in CSV and JSON #17315

Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
adfde77
unify bad record handling in CSV and JSON
cloud-fan Mar 16, 2017
6326c9d
Fix from_json test in R to check NA not the exception
HyukjinKwon Mar 17, 2017
10e70fe
Merge pull request #4 from HyukjinKwon/r-test-fix
cloud-fan Mar 17, 2017
b5aee0e
improve
cloud-fan Mar 17, 2017
aa6736f
address comments
cloud-fan Mar 20, 2017
2df5d3b
Merge remote-tracking branch 'origin/master' into bad-record2
cloud-fan Mar 20, 2017
20ac52f
address comments
cloud-fan Mar 20, 2017
adf7d33
minor comments
cloud-fan Mar 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions R/pkg/inst/tests/testthat/test_sparkSQL.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1354,9 +1354,8 @@ test_that("column functions", {
# passing option
df <- as.DataFrame(list(list("col" = "{\"date\":\"21/10/2014\"}")))
schema2 <- structType(structField("date", "date"))
expect_error(tryCatch(collect(select(df, from_json(df$col, schema2))),
error = function(e) { stop(e) }),
paste0(".*(java.lang.NumberFormatException: For input string:).*"))
s <- collect(select(df, from_json(df$col, schema2)))
expect_equal(s[[1]][[1]], NA)
Copy link
Member

@gatorsmile gatorsmile Mar 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also sounds a behavior change. Could you add another test case here to trigger the exception?

Copy link
Member

@gatorsmile gatorsmile Mar 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or a bug fix?

Copy link
Contributor Author

@cloud-fan cloud-fan Mar 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea it's a minor bug fix, see cloud-fan#4

I'm not sure if it worth a ticket.

Copy link
Member

@gatorsmile gatorsmile Mar 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uh, I see.

s <- collect(select(df, from_json(df$col, schema2, dateFormat = "dd/MM/yyyy")))
expect_is(s[[1]][[1]]$date, "Date")
expect_equal(as.character(s[[1]][[1]]$date), "2014-10-21")
Expand Down