Skip to content

[SPARK-26745][SQL] Skip empty lines in JSON-derived DataFrames when skipParsing optimization in effect #23665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sumitsu wants to merge 17 commits into from
Closed

[SPARK-26745][SQL] Skip empty lines in JSON-derived DataFrames when skipParsing optimization in effect #23665

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
446ae98
[WIP] filter out empty/whitespace JSON lines when skipping parsing
Jan 21, 2019
1544771
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 21, 2019
236227f
remove println/dumpStack
Jan 21, 2019
e4d9052
add test for non-parsed JSON count
Jan 21, 2019
105e5bb
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 22, 2019
e8e3189
Merge branch 'json_emptyline_count_test' of github.com:sumitsu/spark …
Jan 22, 2019
13942b8
fix scala import style errors
Jan 22, 2019
5f173d9
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 22, 2019
7a51764
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 22, 2019
91305ee
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 23, 2019
051d84a
push down non-parsed json record filter into FailureSafeParser
Jan 23, 2019
57d2c05
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 24, 2019
4fffe7f
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 25, 2019
2252045
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 26, 2019
532a83d
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 27, 2019
3cae4da
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 27, 2019
cd2f30c
Merge branch 'master' of github.com:apache/spark into json_emptyline_…
Jan 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' of github.com:apache/spark into json_emptyline_… 
…count
  • Loading branch information
Branden Smith committed Jan 22, 2019
commit 105e5bb193176502945b3cfbc545fc6700a27f4c

This merge commit was added into this branch cleanly.

There are no new changes to show, but you can still view the diff.