Skip to content

[SPARK-8848] [SQL] [WIP] Refactors Parquet write path to follow Parquet format spec #7679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liancheng wants to merge 21 commits into from
Closed

[SPARK-8848] [SQL] [WIP] Refactors Parquet write path to follow Parquet format spec #7679

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
e9305bd
Refactors Parquet write support to follow Parquet format spec
liancheng Jul 26, 2015
b465661
Cleans up code only used by old RowWriteSupport
liancheng Jul 26, 2015
821e9ec
Fixes test failures
liancheng Jul 27, 2015
2a1e884
Fixes writing UDT
liancheng Jul 27, 2015
e9638f0
Optimizes writing structs
liancheng Jul 27, 2015
a2aeba5
Fixes writing empty arrays and maps
liancheng Jul 27, 2015
678ccd4
Migrates large decimal precision support
liancheng Jul 28, 2015
b9f93db
Fixes writing negative decimal values
liancheng Jul 28, 2015
0e0a957
Minor comment updates
liancheng Jul 28, 2015
2859132
Fixes array type conversion in legacy mode
liancheng Jul 29, 2015
62c4829
Minor refactoring
liancheng Jul 29, 2015
b37fe77
Fixes compilation error introduced while rebasing
liancheng Jul 31, 2015
0d61b3b
Writes arrays using ArrayData
liancheng Jul 31, 2015
f901a16
Writes maps using MapData
liancheng Aug 1, 2015
5127b8d
Retrieves data from SpecializedGetters
liancheng Aug 1, 2015
1f1d4af
Renames "Catalyst*" classes to "Parquet*"
liancheng Aug 2, 2015
0395e95
Renames root Parquet message name
liancheng Aug 2, 2015
23e523d
Makes implicit arguments in ParquetSchemaSuite explicit
liancheng Aug 2, 2015
66419b7
Renames followParquetFormatSpec to writeLegacyParquetFormat (its mean…
liancheng Aug 2, 2015
6bda94b
Renames analysisRequire to checkConversionRequirement
liancheng Aug 2, 2015
679888a
Simplifies ParquetSchemaConverter and updates outdated comments
liancheng Aug 3, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Minor comment updates
  • Loading branch information
@liancheng
liancheng committed Aug 1, 2015
commit 0e0a957d25b94b8121867d24730490b7b4bde2be
8 changes: 5 additions & 3 deletions sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystWriteSupport.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,9 @@ private[parquet] class CatalystWriteSupport extends WriteSupport[InternalRow] wi
recordConsumer.addBinary(Binary.fromByteArray(row.getUTF8String(ordinal).getBytes))

case TimestampType =>
// TODO Writes `TimestampType` values as `TIMESTAMP_MICROS` once parquet-mr implements it
(row: InternalRow, ordinal: Int) => {
// Actually Spark SQL `TimestampType` only has microsecond precision.
val (julianDay, timeOfDayNanos) = DateTimeUtils.toJulianDay(row.getLong(ordinal))
val buf = ByteBuffer.wrap(timestampBuffer)
buf.order(ByteOrder.LITTLE_ENDIAN).putLong(timeOfDayNanos).putInt(julianDay)
Expand Down Expand Up @@ -239,12 +241,12 @@ private[parquet] class CatalystWriteSupport extends WriteSupport[InternalRow] wi
// Standard mode, writes decimals with precision <= 18 as INT64
case true if precision <= MAX_PRECISION_FOR_INT64 => int64Writer

// Legacy mode, writes decimals with precision <= 18 as BINARY
// Legacy mode, writes decimals with precision <= 18 as FIXED_LEN_BYTE_ARRAY
case false if precision <= MAX_PRECISION_FOR_INT64 => binaryWriterUsingUnscaledLong

// All other cases:
// - Standard mode, writes decimals with precision > 18 as BINARY
// - Legacy mode, writes decimals with all precision as BINARY
// - Standard mode, writes decimals with precision > 18 as FIXED_LEN_BYTE_ARRAY
// - Legacy mode, writes decimals with all precision as FIXED_LEN_BYTE_ARRAY
case _ => binaryWriterUsingUnscaledBytes
}
}
Expand Down