-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-26745][SQL] Skip empty lines in JSON-derived DataFrames when skipParsing optimization in effect #23665
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
446ae98
[WIP] filter out empty/whitespace JSON lines when skipping parsing
1544771
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
236227f
remove println/dumpStack
e4d9052
add test for non-parsed JSON count
105e5bb
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
e8e3189
Merge branch 'json_emptyline_count_test' of github.com:sumitsu/spark …
13942b8
fix scala import style errors
5f173d9
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
7a51764
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
91305ee
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
051d84a
push down non-parsed json record filter into FailureSafeParser
57d2c05
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
4fffe7f
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
2252045
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
532a83d
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
3cae4da
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
cd2f30c
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
add test for non-parsed JSON count
- Loading branch information
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| { "a" : 1 , "b" : 2 , "c" : 3 } | ||
|
|
||
| { "a" : 4 , "b" : 5 , "c" : 6 } | ||
|
|
||
| { "a" : 7 , "b" : 8 , "c" : 9 } | ||
|
|
||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2426,6 +2426,23 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData { | |
| countForMalformedJSON(0, Seq("")) | ||
| } | ||
|
|
||
| test("count() for non-multiline input with empty lines") { | ||
| val withEmptyLineData = Array(Map("a" -> 1, "b" -> 2, "c" -> 3), | ||
| Map("a" -> 4, "b" -> 5, "c" -> 6), | ||
| Map("a" -> 7, "b" -> 8, "c" -> 9)) | ||
| val df = spark.read.json("src/test/resources/test-data/with-empty-line.json") | ||
| // important to do this .count() first, prior to caching/persisting/computing/collecting, to | ||
| // test the non-parsed-count pathway | ||
| assert(df.count() === withEmptyLineData.length, | ||
| "JSON DataFrame unparsed-count should exclude whitespace-only lines") | ||
| // cache and collect to check that count stays stable under those operations | ||
| df.cache() | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we dont need this cache. |
||
| assert(df.count() === withEmptyLineData.length, | ||
| "JSON DataFrame parsed-count should exclude whitespace-only lines") | ||
| val collected = df.collect().map(_.getValuesMap(Seq("a", "b", "c"))) | ||
| assert(collected === withEmptyLineData) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. plz check |
||
| } | ||
|
|
||
| test("SPARK-25040: empty strings should be disallowed") { | ||
| def failedOnEmptyString(dataType: DataType): Unit = { | ||
| val df = spark.read.schema(s"a ${dataType.catalogString}") | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
plz use
testFile.