Skip to content

[SPARK-8848] [SQL] Refactors Parquet write path to follow parquet-format #8988

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liancheng wants to merge 12 commits into from
Closed

[SPARK-8848] [SQL] Refactors Parquet write path to follow parquet-format #8988

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
d1583f8
Refactors Parquet write path to follow parquet-format
liancheng Oct 5, 2015
6fd20f7
Optimizes writing low precision decimals
liancheng Oct 6, 2015
d395bfd
Fixes bug in reading UDTs under standard mode
liancheng Oct 6, 2015
a680f09
Stops using MutableRow when writing maps to avoid boxing
liancheng Oct 6, 2015
5b08a20
Removes more dead code
liancheng Oct 7, 2015
f03ef93
Fixes a bug when writing small decimals coming from rows that are not…
liancheng Oct 7, 2015
2bc5ebc
Addresses comments
liancheng Oct 8, 2015
c542ae9
Fixes comment typo and removes an unused import
liancheng Oct 8, 2015
af50f9c
One more dead code :)
liancheng Oct 8, 2015
e67d0b1
Fixes test cases
liancheng Oct 8, 2015
db79fb6
Addresses comments
liancheng Oct 8, 2015
fb6ee9f
Removes redundant test case
liancheng Oct 8, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Optimizes writing low precision decimals
  • Loading branch information
@liancheng
liancheng committed Oct 6, 2015
commit 6fd20f70baa535b1772c1c30a3f651ea673560f2
12 changes: 4 additions & 8 deletions .../main/scala/org/apache/spark/sql/execution/datasources/parquet/CatalystWriteSupport.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,16 +197,12 @@ private[parquet] class CatalystWriteSupport extends WriteSupport[InternalRow] wi
val numBytes = minBytesForPrecision(precision)

val int32Writer =
(row: SpecializedGetters, ordinal: Int) => {
val unscaledInt = row.getDecimal(ordinal, precision, scale).toUnscaledLong.toInt
recordConsumer.addInteger(unscaledInt)
}
(row: SpecializedGetters, ordinal: Int) =>
recordConsumer.addInteger(row.getLong(ordinal).toInt)

val int64Writer =
(row: SpecializedGetters, ordinal: Int) => {
val unscaledLong = row.getDecimal(ordinal, precision, scale).toUnscaledLong
recordConsumer.addLong(unscaledLong)
}
(row: SpecializedGetters, ordinal: Int) =>
recordConsumer.addLong(row.getLong(ordinal))

val binaryWriterUsingUnscaledLong =
(row: SpecializedGetters, ordinal: Int) => {
Expand Down