Skip to content

[SPARK-8848] [SQL] [WIP] Refactors Parquet write path to follow Parquet format spec #7679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liancheng wants to merge 21 commits into from
Closed

[SPARK-8848] [SQL] [WIP] Refactors Parquet write path to follow Parquet format spec #7679

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
e9305bd
Refactors Parquet write support to follow Parquet format spec
liancheng Jul 26, 2015
b465661
Cleans up code only used by old RowWriteSupport
liancheng Jul 26, 2015
821e9ec
Fixes test failures
liancheng Jul 27, 2015
2a1e884
Fixes writing UDT
liancheng Jul 27, 2015
e9638f0
Optimizes writing structs
liancheng Jul 27, 2015
a2aeba5
Fixes writing empty arrays and maps
liancheng Jul 27, 2015
678ccd4
Migrates large decimal precision support
liancheng Jul 28, 2015
b9f93db
Fixes writing negative decimal values
liancheng Jul 28, 2015
0e0a957
Minor comment updates
liancheng Jul 28, 2015
2859132
Fixes array type conversion in legacy mode
liancheng Jul 29, 2015
62c4829
Minor refactoring
liancheng Jul 29, 2015
b37fe77
Fixes compilation error introduced while rebasing
liancheng Jul 31, 2015
0d61b3b
Writes arrays using ArrayData
liancheng Jul 31, 2015
f901a16
Writes maps using MapData
liancheng Aug 1, 2015
5127b8d
Retrieves data from SpecializedGetters
liancheng Aug 1, 2015
1f1d4af
Renames "Catalyst*" classes to "Parquet*"
liancheng Aug 2, 2015
0395e95
Renames root Parquet message name
liancheng Aug 2, 2015
23e523d
Makes implicit arguments in ParquetSchemaSuite explicit
liancheng Aug 2, 2015
66419b7
Renames followParquetFormatSpec to writeLegacyParquetFormat (its mean…
liancheng Aug 2, 2015
6bda94b
Renames analysisRequire to checkConversionRequirement
liancheng Aug 2, 2015
679888a
Simplifies ParquetSchemaConverter and updates outdated comments
liancheng Aug 3, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixes array type conversion in legacy mode
  • Loading branch information
@liancheng
liancheng committed Aug 1, 2015
commit 2859132e33a1d8d4e86016feca1d0876d162b380
14 changes: 8 additions & 6 deletions sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystSchemaConverter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,29 +439,31 @@ private[parquet] class CatalystSchemaConverter(
// ===================================================

// Spark 1.4.x and prior versions convert ArrayType with nullable elements into a 3-level
// LIST structure. This behavior mimics parquet-hive (1.6.0rc3). Note that this case is
// covered by the backwards-compatibility rules implemented in `isElementType()`.
// LIST structure. This behavior is somewhat a hybrid of parquet-hive and parquet-avro
// (1.6.0rc3): the 3-level structure is similar to parquet-hive while the 3rd level anonymous
// field name "array" is from parquet-avro. Note that this case is covered by the backwards-
// compatibility rules implemented in `isElementType()`.
case ArrayType(elementType, nullable @ true) if !followParquetFormatSpec =>
// <list-repetition> group <name> (LIST) {
// optional group bag {
// repeated <element-type> element;
// repeated <element-type> array;
// }
// }
ConversionPatterns.listType(
repetition,
field.name,
Types
.buildGroup(REPEATED)
// "array_element" is the name chosen by parquet-hive (1.7.0 and prior version)
.addField(convertField(StructField("array_element", elementType, nullable)))
// "array" is the name chosen by Spark SQL 1.4.0 and prior versions
.addField(convertField(StructField("array", elementType, nullable)))
.named("bag"))

// Spark 1.4.x and prior versions convert ArrayType with non-nullable elements into a 2-level
// LIST structure. This behavior mimics parquet-avro (1.6.0rc3). Note that this case is
// covered by the backwards-compatibility rules implemented in `isElementType()`.
case ArrayType(elementType, nullable @ false) if !followParquetFormatSpec =>
// <list-repetition> group <name> (LIST) {
// repeated <element-type> element;
// repeated <element-type> array;
// }
ConversionPatterns.listType(
repetition,
Expand Down
6 changes: 3 additions & 3 deletions sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetSchemaSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ class ParquetSchemaInferenceSuite extends ParquetSchemaTest {
|message root {
| optional group _1 (LIST) {
| repeated group bag {
| optional int32 array_element;
| optional int32 array;
| }
| }
|}
Expand Down Expand Up @@ -267,7 +267,7 @@ class ParquetSchemaInferenceSuite extends ParquetSchemaTest {
| optional binary _1 (UTF8);
| optional group _2 (LIST) {
| repeated group bag {
| optional group array_element {
| optional group array {
| required int32 _1;
| required double _2;
| }
Expand Down Expand Up @@ -616,7 +616,7 @@ class ParquetSchemaSuite extends ParquetSchemaTest {
"""message root {
| optional group f1 (LIST) {
| repeated group bag {
| optional int32 array_element;
| optional int32 array;
| }
| }
|}
Expand Down