Skip to content

[SPARK-17763][SQL] JacksonParser silently parses null as 0 when the field is not nullable #15329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 2 commits into from
Closed

[SPARK-17763][SQL] JacksonParser silently parses null as 0 when the field is not nullable #15329

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8824186
JacksonParser silently parses null as 0 when the field is not nullable
HyukjinKwon Oct 3, 2016
cb5ece7
Consistent name
HyukjinKwon Oct 3, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Consistent name
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Oct 3, 2016
commit cb5ece70bc840820697fa73943bc28bd63c96b41
8 changes: 4 additions & 4 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -354,14 +354,14 @@ class JacksonParser(
parser: JsonParser,
dataType: DataType,
nullable: Boolean = true)(f: PartialFunction[JsonToken, Any]): Any = {
val nullParser = makeNullConverter(nullable)
val nullConverter = makeNullConverter(nullable)
parser.getCurrentToken match {
case FIELD_NAME =>
// There are useless FIELD_NAMEs between START_OBJECT and END_OBJECT tokens
parser.nextToken()
parseJsonToken(parser, dataType, nullable)(f)

case null | VALUE_NULL => nullParser.apply(parser)
case null | VALUE_NULL => nullConverter.apply(parser)

case other => f.applyOrElse(other, failedConversion(parser, dataType, nullable))
}
Expand All @@ -375,13 +375,13 @@ class JacksonParser(
parser: JsonParser,
dataType: DataType,
nullable: Boolean): PartialFunction[JsonToken, Any] = {
val nullParser = makeNullConverter(nullable)
val nullConverter = makeNullConverter(nullable)

{
case VALUE_STRING if parser.getTextLength < 1 =>
// If conversion is failed, this produces `null` rather than throwing exception.
// This will protect the mismatch of types.
nullParser.apply(parser)
nullConverter.apply(parser)

case token =>
// We cannot parse this token based on the given data type. So, we throw a
Expand Down