Skip to content

[SPARK-2927][SQL] Add a conf to configure if we always read Binary columns stored in Parquet as String columns #1855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yhuai wants to merge 5 commits into from
Closed

[SPARK-2927][SQL] Add a conf to configure if we always read Binary columns stored in Parquet as String columns #1855

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5d436a1
The initial support of adding a conf to treat binary columns stored i…
yhuai Aug 8, 2014
9d3f199
Merge remote-tracking branch 'upstream/master' into parquetBinaryAsSt…
yhuai Aug 13, 2014
1765ca4
Use .toBoolean.
yhuai Aug 13, 2014
80827de
Unit test.
yhuai Aug 14, 2014
689ffa9
Add missing "=".
yhuai Aug 14, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Add missing "=".
  • Loading branch information
@yhuai
yhuai committed Aug 14, 2014
commit 689ffa9906990366190c7fbde406762fa059afd0
4 changes: 2 additions & 2 deletions sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ class ParquetQuerySuite extends QueryTest with FunSuiteLike with BeforeAndAfterA
val resultWithBinary = parquetFile(path).collect
range.foreach {
i =>
assert(resultWithBinary(i).getInt(0) == i)
assert(resultWithBinary(i).getInt(0) === i)
assert(resultWithBinary(i)(1) === s"val_$i".getBytes)
}

Expand All @@ -172,7 +172,7 @@ class ParquetQuerySuite extends QueryTest with FunSuiteLike with BeforeAndAfterA
val resultWithString = schemaRDD.collect
range.foreach {
i =>
assert(resultWithString(i).getInt(0) == i)
assert(resultWithString(i).getInt(0) === i)
assert(resultWithString(i)(1) === s"val_$i")
}

Expand Down