[WiP][SPARK-18699] SQL - parsing CSV should return null for certain numeric … #16319
Conversation
…fields - Sargis Dudaklayan and Kuba Tyszko - Zest Finance
|
Can one of the admins verify this patch? |
|
Can you describe which case this PR fixes (with reproducible codes)? I think you meant https://issues.apache.org/jira/browse/SPARK-18699 |
| case _: LongType => datum.toLong | ||
| case _: ShortType => Try(datum.toShort).getOrElse(null) | ||
| case _: IntegerType => Try(datum.toInt).getOrElse(null) | ||
| case _: LongType => Try(datum.toLong).getOrElse(null) |
There was a problem hiding this comment.
So, I guess you meant when the actual data is empty string and nullValue is other values such as NA, they should be translated into null when the mode is PERMISSIVE in this case.
Then, this is a exact duplicate of https://issues.apache.org/jira/browse/SPARK-18699. You could resolve the current JIRA as a duplicate and then fix the title of this PR to [SPARK-18699] ... with some changes suggested there.
There was a problem hiding this comment.
You're right, fixed the title. Thanks.
There was a problem hiding this comment.
Why do you use Try for each type here? In the JIRA, I proposed another approach (master...maropu:SPARK-18699), and I think the fix is more natural. Anyway, ISTM we do not decide yet it is worth fixing this issue...
There was a problem hiding this comment.
My patch was created before I even knew about SPARK-18699 (I did have another ticket open, then learned it was a duplicate).
I do agree that your solution may be more generic and intuitive, if others agree - why wasn't it merged into main tree yet ? (it has been over 10 days since your PR).
There was a problem hiding this comment.
Behavior changes are much arguable, so it basically takes much time to discuss.
kubatyszko
changed the title
kubatyszko
changed the title
|
Let's put Let me please cc @maropu and @falaki here who were in the JIRA. |
|
@HyukjinKwon Thanks for your pinging! I left some comments. |
kubatyszko
changed the title
|
Hi @kubatyszko, are you still working on this? If you are currently unable to proceed further, maybe it should be closed for now. It seems inactive for few months. |
## What changes were proposed in this pull request? This PR proposes to close stale PRs. What I mean by "stale" here includes that there are some review comments by reviewers but the author looks inactive without any answer to them more than a month. I left some comments roughly a week ago to ping and the author looks still inactive in these PR below These below includes some PR suggested to be closed and a PR against another branch which seems obviously inappropriate. Given the comments in the last three PRs below, they are probably worth being taken over by anyone who is interested in it. Closes apache#7963 Closes apache#8374 Closes apache#11192 Closes apache#11374 Closes apache#11692 Closes apache#12243 Closes apache#12583 Closes apache#12620 Closes apache#12675 Closes apache#12697 Closes apache#12800 Closes apache#13715 Closes apache#14266 Closes apache#15053 Closes apache#15159 Closes apache#15209 Closes apache#15264 Closes apache#15267 Closes apache#15871 Closes apache#15861 Closes apache#16319 Closes apache#16324 Closes apache#16890 Closes apache#12398 Closes apache#12933 Closes apache#14517 ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#16937 from HyukjinKwon/stale-prs-close.
4 participants
What changes were proposed in this pull request?
CSV parser changes allowing parsing of numeric fields to fail and return null in such case.
In conjunction with "nullValue" option that may be used elsewhere this allows handling of certain csv sources that may use empty string as indication of null in one column and another specific value indicating null in another.
Currently the option "nullValue" can only be provided once and we can't assume that a data source won't have a single "null" indicator.
This problem is very similar to the one discussed here: https://github.com/databricks/spark-csv/issues/239
Sargis Dudaklayan and Kuba Tyszko - Zest Finance
How was this patch tested?
The patch was tested using freshly compiled spark version 2.0.1 on a sample data source that has "null" values in 2 columns, one specified as "NA" and set using nullValue and another column with "" indicating no integer value.
Please review http://spark.apache.org/contributing.html before opening a pull request.