Skip to content

[SPARK-11753][SQL][test-hadoop2.2] Make allowNonNumericNumbers option work #9759

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 12 commits into from
Closed

[SPARK-11753][SQL][test-hadoop2.2] Make allowNonNumericNumbers option work #9759

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
2777677
Make allowNonNumericNumbers option work.
viirya Nov 17, 2015
b2a835d
Not to quote non numeric numbers when enabling allowNonNumericNumbers.
viirya Nov 17, 2015
186fa5e
Deal with quoted non-numeric number.
viirya Nov 20, 2015
dc9abc3
Merge remote-tracking branch 'upstream/master' into fix-json-nonnumric
viirya Nov 27, 2015
6d90b24
For comments.
viirya Nov 27, 2015
c74d715
Merge remote-tracking branch 'upstream/master' into fix-json-nonnumric
viirya May 16, 2016
6f668c3
Update doc.
viirya May 16, 2016
1cfd1dc
Fix dependency test.
viirya May 16, 2016
4fca52c
Address comment. Revert change of InferSchema.
viirya May 18, 2016
0e473fc
Support +INF, -INF.
viirya May 18, 2016
025edea
Manage paranamer up to 2.8.
viirya May 22, 2016
af1e3a1
Support INF and +Infinity when allowNonNumericNumbers is off.
viirya May 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Make allowNonNumericNumbers option work.
  • Loading branch information
@viirya
viirya committed Nov 17, 2015
commit 27776778b01c17cbb94f32a41eab858bb438b3e5
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONOptions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,6 @@ object JSONOptions {
allowNumericLeadingZeros =
parameters.get("allowNumericLeadingZeros").map(_.toBoolean).getOrElse(false),
allowNonNumericNumbers =
parameters.get("allowNonNumericNumbers").map(_.toBoolean).getOrElse(true)
parameters.get("allowNonNumericNumbers").map(_.toBoolean).getOrElse(false)
)
}
13 changes: 1 addition & 12 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,18 +100,7 @@ object JacksonParser {
parser.getFloatValue

case (VALUE_STRING, FloatType) =>
// Special case handling for NaN and Infinity.
val value = parser.getText
val lowerCaseValue = value.toLowerCase()
if (lowerCaseValue.equals("nan") ||
lowerCaseValue.equals("infinity") ||
lowerCaseValue.equals("-infinity") ||
lowerCaseValue.equals("inf") ||
lowerCaseValue.equals("-inf")) {
value.toFloat
} else {
sys.error(s"Cannot parse $value as FloatType.")
}
parser.getFloatValue
Copy link
Contributor

@rxin rxin Nov 27, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we removing the special handling for float types here?

Copy link
Member Author

@viirya viirya May 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea, should revert it back. BTW, do we actually test "inf" and "-inf" before? Because "inf".toFloat is not legal.


case (VALUE_NUMBER_INT | VALUE_NUMBER_FLOAT, DoubleType) =>
parser.getDoubleValue
Expand Down
6 changes: 2 additions & 4 deletions .../test/scala/org/apache/spark/sql/execution/datasources/json/JsonParsingOptionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,17 +93,15 @@ class JsonParsingOptionsSuite extends QueryTest with SharedSQLContext {
assert(df.first().getLong(0) == 18)
}

// The following two tests are not really working - need to look into Jackson's
// JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS.
ignore("allowNonNumericNumbers off") {
test("allowNonNumericNumbers off") {
val str = """{"age": NaN}"""
val rdd = sqlContext.sparkContext.parallelize(Seq(str))
val df = sqlContext.read.json(rdd)

assert(df.schema.head.name == "_corrupt_record")
}

ignore("allowNonNumericNumbers on") {
test("allowNonNumericNumbers on") {
val str = """{"age": NaN}"""
val rdd = sqlContext.sparkContext.parallelize(Seq(str))
val df = sqlContext.read.option("allowNonNumericNumbers", "true").json(rdd)
Expand Down