Skip to content

[SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0. #21070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rdblue wants to merge 11 commits into from
Closed

[SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0. #21070

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
26336a9
SPARK-23972: Update Parquet to 1.10.0.
rdblue Dec 1, 2017
a599515
SPARK-23972: Update dependencies for Parquet 1.10.0.
rdblue Apr 16, 2018
3e1f84c
SPARK-23972: Fix method signatures for IOExceptions.
rdblue Apr 16, 2018
93b8ee8
SPAKR-23972: Bulk write to ColumnVector when Parquet data is on the h…
rdblue Apr 16, 2018
93f2a92
SPARK-23972: Fix test failures and review comments.
rdblue Apr 18, 2018
64907bc
SPARK-23972: Add lz4, brotli, and zstd compression for Parquet.
rdblue Apr 18, 2018
66df4b6
SPARK-23972: Update InMemoryColumnarQuerySuite for new Parquet version.
rdblue Apr 18, 2018
ee2ffdd
SPARK-23972: Regenerate golden outputs for Parquet size changes.
rdblue Apr 19, 2018
270540f
SPARK-23972: Update Hadoop 3.1 dependency profile.
rdblue May 7, 2018
6c9d47b
SPARK-23972: Update sql-programming-guide.md.
rdblue May 7, 2018
95ecde0
Fix buffer unpacking offset in VectorizedRleValuesReader.
rdblue May 8, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SPARK-23972: Add lz4, brotli, and zstd compression for Parquet.
  • Loading branch information
@rdblue
rdblue committed May 7, 2018
commit 64907bc3f97c9a59da1e3bee1ec57585ecc3ac41
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,7 @@ object SQLConf {
"snappy, gzip, lzo.")
.stringConf
.transform(_.toLowerCase(Locale.ROOT))
.checkValues(Set("none", "uncompressed", "snappy", "gzip", "lzo"))
.checkValues(Set("none", "uncompressed", "snappy", "gzip", "lzo", "lz4", "brotli", "zstd"))
Copy link
Member

@dongjoon-hyun dongjoon-hyun May 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you update sql-programming-guide.md together?

Copy link
Contributor Author

@rdblue rdblue May 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

.createWithDefault("snappy")

val PARQUET_FILTER_PUSHDOWN_ENABLED = buildConf("spark.sql.parquet.filterPushdown")
Expand Down
5 changes: 4 additions & 1 deletion ...re/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOptions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,10 @@ object ParquetOptions {
"uncompressed" -> CompressionCodecName.UNCOMPRESSED,
"snappy" -> CompressionCodecName.SNAPPY,
"gzip" -> CompressionCodecName.GZIP,
"lzo" -> CompressionCodecName.LZO)
"lzo" -> CompressionCodecName.LZO,
"lz4" -> CompressionCodecName.LZ4,
"brotli" -> CompressionCodecName.BROTLI,
"zstd" -> CompressionCodecName.ZSTD)

def getParquetCompressionCodecName(name: String): String = {
shortParquetCompressionCodecNames(name).name()
Expand Down