Skip to content

[SPARK-23773][SQL] JacksonGenerator does not include keys that have null value for StructTypes #20884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
makagonov wants to merge 2 commits into from
Closed

[SPARK-23773][SQL] JacksonGenerator does not include keys that have null value for StructTypes #20884

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9faf853
[SPARK-23773][SQL] JacksonGenerator does not include keys that have n…
Mar 22, 2018
559c201
fixing broken unit tests
Mar 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[SPARK-23773][SQL] JacksonGenerator does not include keys that have n… 
…ull value for StructTypes
  • Loading branch information
Sergey Makagonov committed Mar 22, 2018
commit 9faf8533d044bd667bac6fb1925f2d38c4d281d4
6 changes: 4 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,8 +171,10 @@ private[sql] class JacksonGenerator(
var i = 0
while (i < row.numFields) {
val field = schema(i)
if (!row.isNullAt(i)) {
gen.writeFieldName(field.name)
gen.writeFieldName(field.name)
if (row.isNullAt(i)) {
gen.writeNull()
} else {
fieldWriters(i).apply(row, i)
}
i += 1
Expand Down
10 changes: 9 additions & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1229,7 +1229,7 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
val df2 = df1.toDF
val result = df2.toJSON.collect()
// scalastyle:off
assert(result(0) === "{\"f1\":1,\"f2\":\"A1\",\"f3\":true,\"f4\":[\"1\",\" A1\",\" true\",\" null\"]}")
assert(result(0) === "{\"f1\":1,\"f2\":\"A1\",\"f3\":true,\"f4\":[\"1\",\" A1\",\" true\",\" null\"],\"f5\":null}")
Copy link
Member

@HyukjinKwon HyukjinKwon Mar 24, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we go the current way, it'd write out every null with every field:

{"a":null,"b":null,"c":null}
{"a":null,"b":null,"c":1}
{"a":1,"b":null,"c":1}
{"a":1,"b":2,"c":3}

which I think's quite inefficient. Does that fix actually use case to be clear?

assert(result(3) === "{\"f1\":4,\"f2\":\"D4\",\"f3\":true,\"f4\":[\"4\",\" D4\",\" true\",\" 2147483644\"],\"f5\":2147483644}")
// scalastyle:on

Expand Down Expand Up @@ -1265,6 +1265,7 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
1.7976931348623157E308,
10,
21474836470L,
null,
"this is a simple string.")
)

Expand Down Expand Up @@ -2063,4 +2064,11 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
)
}
}

test("SPARK-23773: JacksonGenerator does not include keys " +
"that have null value for StructTypes") {
val df = sql("select NAMED_STRUCT('f1', 10, 'f2', null, 'f3', 15) as my_struct")
val result = df.toJSON.collect()(0)
assert(result === "{\"my_struct\":{\"f1\":10,\"f2\":null,\"f3\":15}}")
}
}