Skip to content

[SPARK-19949][SQL] unify bad record handling in CSV and JSON #17315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cloud-fan wants to merge 8 commits into from
Closed

[SPARK-19949][SQL] unify bad record handling in CSV and JSON #17315

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
adfde77
unify bad record handling in CSV and JSON
cloud-fan Mar 16, 2017
6326c9d
Fix from_json test in R to check NA not the exception
HyukjinKwon Mar 17, 2017
10e70fe
Merge pull request #4 from HyukjinKwon/r-test-fix
cloud-fan Mar 17, 2017
b5aee0e
improve
cloud-fan Mar 17, 2017
aa6736f
address comments
cloud-fan Mar 20, 2017
2df5d3b
Merge remote-tracking branch 'origin/master' into bad-record2
cloud-fan Mar 20, 2017
20ac52f
address comments
cloud-fan Mar 20, 2017
adf7d33
minor comments
cloud-fan Mar 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comments
  • Loading branch information
@cloud-fan
cloud-fan committed Mar 20, 2017
commit 20ac52f249477d17b0b07ff81bddeef444d5b546
6 changes: 3 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.spark.sql.types.StructType
import org.apache.spark.unsafe.types.UTF8String

class FailureSafeParser[IN](
func: IN => Seq[InternalRow],
rawParser: IN => Seq[InternalRow],
mode: String,
schema: StructType,
columnNameOfCorruptRecord: String) {
Expand All @@ -50,13 +50,13 @@ class FailureSafeParser[IN](
resultRow
}
} else {
(row, badRecord) => row.getOrElse(nullResult)
(row, _) => row.getOrElse(nullResult)
}
}

def parse(input: IN): Iterator[InternalRow] = {
try {
func(input).toIterator.map(row => toResultRow(Some(row), () => null))
rawParser.apply(input).toIterator.map(row => toResultRow(Some(row), () => null))
} catch {
case e: BadRecordException if ParseModes.isPermissiveMode(mode) =>
Iterator(toResultRow(e.partialResult(), e.record))
Expand Down
4 changes: 3 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,9 +212,11 @@ class UnivocityParser(
new RuntimeException("Malformed CSV record"))
Copy link
Member

@HyukjinKwon HyukjinKwon Mar 17, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. So, BadRecordException contains the information about which record is malformed so that users can access to and check this. Did I understand correctly?

Copy link
Contributor Author

@cloud-fan cloud-fan Mar 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

} else {
try {
for (i <- requiredSchema.indices) {
var i = 0
while (i < requiredSchema.length) {
val from = tokenIndexArr(i)
row(i) = valueConverters(from).apply(tokens(from))
i += 1
}
row
} catch {
Expand Down