Skip to content

[SPARK-18352][SQL] Support parsing multiline json files #16386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
NathanHowell wants to merge 23 commits into from
Closed

[SPARK-18352][SQL] Support parsing multiline json files #16386

Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
04c8cde
[SPARK-18352][SQL] Support parsing multiline json files
Dec 22, 2016
86e3cd2
JacksonParser.parseJsonToken should be explicit about nulls and boxing
Dec 22, 2016
f48fc66
Increase type safety of makeRootConverter, remove runtime type tests
Dec 21, 2016
361df08
Field converter lookups should be O(1)
Dec 23, 2016
06169f7
Support inference of tables with no columns
Dec 27, 2016
9b8a265
Improve failedRecord consistency with and without wholeFile mode enabled
Dec 29, 2016
42bee5e
Use withTempPath instead of withTempDir
Feb 8, 2017
b17809e
Simplify the corrupt document test
Feb 8, 2017
422f9b0
Always return a ByteBuffer from getByteBuffer
Feb 8, 2017
042d746
Add @Since annotations for all PortableDataStream public methods
Feb 9, 2017
e1a620a
Very verbosely test to see if a warning has already been printed
Feb 10, 2017
3e67473
Eagerly validate the corrupt column datatype
Feb 11, 2017
8ed787f
Avoid broadcasting JsonDataSource references
Feb 11, 2017
14d5f93
Remove name binding
Feb 11, 2017
5f5214b
Repartition by value instead of luck
Feb 11, 2017
7296f7e
Always provide a `T => UTF8String` conversion function
Feb 15, 2017
691fa2a
Reorder documentation to match the function parameter order
Feb 16, 2017
a629470
Fix build break in Python due to bad rebase
Feb 16, 2017
463062a
Fix constructor invocation style
Feb 16, 2017
24786d1
Check wholeFile roundtrip against all columns and the source RDD
Feb 17, 2017
e323317
Add tests for FAILFAST mode
Feb 17, 2017
58118f2
More style fixes
Feb 17, 2017
b801ab0
Missed one Javadoc parameter reordering
Feb 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix constructor invocation style
  • Loading branch information
Nathan Howell committed Feb 16, 2017
commit 463062ad731e661cd6036574052f56a0ad074f6d
3 changes: 2 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,8 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* @since 1.4.0
*/
def json(jsonRDD: RDD[String]): DataFrame = {
val parsedOptions = new JSONOptions(extraOptions.toMap,
val parsedOptions = new JSONOptions(
extraOptions.toMap,
sparkSession.sessionState.conf.sessionLocalTimeZone,
sparkSession.sessionState.conf.columnNameOfCorruptRecord)
val createParser = CreateJacksonParser.string _
Expand Down