Skip to content

[SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0. #21070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rdblue wants to merge 11 commits into from
Closed

[SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0. #21070

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
26336a9
SPARK-23972: Update Parquet to 1.10.0.
rdblue Dec 1, 2017
a599515
SPARK-23972: Update dependencies for Parquet 1.10.0.
rdblue Apr 16, 2018
3e1f84c
SPARK-23972: Fix method signatures for IOExceptions.
rdblue Apr 16, 2018
93b8ee8
SPAKR-23972: Bulk write to ColumnVector when Parquet data is on the h…
rdblue Apr 16, 2018
93f2a92
SPARK-23972: Fix test failures and review comments.
rdblue Apr 18, 2018
64907bc
SPARK-23972: Add lz4, brotli, and zstd compression for Parquet.
rdblue Apr 18, 2018
66df4b6
SPARK-23972: Update InMemoryColumnarQuerySuite for new Parquet version.
rdblue Apr 18, 2018
ee2ffdd
SPARK-23972: Regenerate golden outputs for Parquet size changes.
rdblue Apr 19, 2018
270540f
SPARK-23972: Update Hadoop 3.1 dependency profile.
rdblue May 7, 2018
6c9d47b
SPARK-23972: Update sql-programming-guide.md.
rdblue May 7, 2018
95ecde0
Fix buffer unpacking offset in VectorizedRleValuesReader.
rdblue May 8, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SPARK-23972: Fix method signatures for IOExceptions.
  • Loading branch information
@rdblue
rdblue committed May 7, 2018
commit 3e1f84cf25ec60a62a21e3545a7f7a86ffbff2fc
13 changes: 7 additions & 6 deletions .../main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,15 +390,16 @@ private void decodeDictionaryIds(
* is guaranteed that num is smaller than the number of values left in the current page.
*/

private void readBooleanBatch(int rowId, int num, WritableColumnVector column) {
private void readBooleanBatch(int rowId, int num, WritableColumnVector column)
throws IOException {
if (column.dataType() != DataTypes.BooleanType) {
throw constructConvertNotSupportedException(descriptor, column);
}
defColumn.readBooleans(
num, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
}

private void readIntBatch(int rowId, int num, WritableColumnVector column) {
private void readIntBatch(int rowId, int num, WritableColumnVector column) throws IOException {
// This is where we implement support for the valid type conversions.
// TODO: implement remaining type conversions
if (column.dataType() == DataTypes.IntegerType || column.dataType() == DataTypes.DateType ||
Expand All @@ -416,7 +417,7 @@ private void readIntBatch(int rowId, int num, WritableColumnVector column) {
}
}

private void readLongBatch(int rowId, int num, WritableColumnVector column) {
private void readLongBatch(int rowId, int num, WritableColumnVector column) throws IOException {
// This is where we implement support for the valid type conversions.
if (column.dataType() == DataTypes.LongType ||
DecimalType.is64BitDecimalType(column.dataType()) ||
Expand All @@ -436,7 +437,7 @@ private void readLongBatch(int rowId, int num, WritableColumnVector column) {
}
}

private void readFloatBatch(int rowId, int num, WritableColumnVector column) {
private void readFloatBatch(int rowId, int num, WritableColumnVector column) throws IOException {
// This is where we implement support for the valid type conversions.
// TODO: support implicit cast to double?
if (column.dataType() == DataTypes.FloatType) {
Expand All @@ -447,7 +448,7 @@ private void readFloatBatch(int rowId, int num, WritableColumnVector column) {
}
}

private void readDoubleBatch(int rowId, int num, WritableColumnVector column) {
private void readDoubleBatch(int rowId, int num, WritableColumnVector column) throws IOException {
// This is where we implement support for the valid type conversions.
// TODO: implement remaining type conversions
if (column.dataType() == DataTypes.DoubleType) {
Expand All @@ -458,7 +459,7 @@ private void readDoubleBatch(int rowId, int num, WritableColumnVector column) {
}
}

private void readBinaryBatch(int rowId, int num, WritableColumnVector column) {
private void readBinaryBatch(int rowId, int num, WritableColumnVector column) throws IOException {
// This is where we implement support for the valid type conversions.
// TODO: implement remaining type conversions
VectorizedValuesReader data = (VectorizedValuesReader) dataColumn;
Expand Down