Skip to content

[SPARK-6776] [SQL] [WIP] More Parquet compatibility tests #7472

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liancheng wants to merge 6 commits into from
Closed

[SPARK-6776] [SQL] [WIP] More Parquet compatibility tests #7472

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3cb48ab
Tests for Avro enum values
liancheng Jul 17, 2015
e152ed6
Makes implicit conversions more explicit
liancheng Jul 17, 2015
3d69df3
Tests for nullable Avro complex types
liancheng Jul 17, 2015
4380470
Tests for Hive complex types
liancheng Jul 17, 2015
87b59ab
Tests for Avro array of struct
liancheng Jul 17, 2015
b1be549
Updates .gitignore and adds an Avro generated Java source file
liancheng Jul 18, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Tests for Hive complex types
  • Loading branch information
@liancheng
liancheng committed Jul 17, 2015
commit 43804700e82527afd3cfc731025ddef26c4dd380
10 changes: 8 additions & 2 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetHiveCompatibilitySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@ class ParquetHiveCompatibilitySuite extends ParquetCompatibilityTest {
| double_column DOUBLE,
|
| strings_column ARRAY<STRING>,
| int_to_string_column MAP<INT, STRING>
| int_to_string_column MAP<INT, STRING>,
| structs_column ARRAY<STRUCT<a: STRING, b: STRING>>,
| maps_column ARRAY<MAP<INT, STRING>>
|)
|STORED AS PARQUET
|LOCATION '${parquetStore.getCanonicalPath}'
Expand Down Expand Up @@ -86,7 +88,11 @@ class ParquetHiveCompatibilitySuite extends ParquetCompatibilityTest {
nullable(i.toFloat + 0.1f: java.lang.Float),
nullable(i.toDouble + 0.2d: java.lang.Double),
nullable(Seq.tabulate(3)(n => s"arr_${i + n}")),
nullable(Seq.tabulate(3)(n => (i + n: Integer) -> s"val_${i + n}").toMap))
nullable(Seq.tabulate(3)(n => (i + n: Integer) -> s"val_${i + n}").toMap),
nullable(Seq.tabulate(3)(n => Row(s"val_a_${i + n}", s"val_b_${i + n}"))),
nullable(Seq.tabulate(3) { n =>
Seq.tabulate(3)(m => (i + n + m: Integer) -> s"val_b_${i + n + m}").toMap
}))
}
}
}