Skip to content

Update matrix for parquet-cpp and parquet-java #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wgtmac merged 5 commits into from
Feb 13, 2025
Merged

Update matrix for parquet-cpp and parquet-java #100

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
33d9bce
update matrix for cpp and java
wgtmac Feb 5, 2025
ca02ad6
fix brotli
wgtmac Feb 6, 2025
7b57262
address comment
wgtmac Feb 7, 2025
77614fa
mark all types supported for java
wgtmac Feb 7, 2025
c6573ff
use asterisk to explain supported logical type
wgtmac Feb 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 59 additions & 57 deletions content/en/docs/File Format/implementationstatus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,14 @@ Implementations:

| Data type | C++ | Java | Go | Rust | cuDF |
| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- |
| BOOLEAN | | | | | ✅ |
| INT32 | | | | | ✅ |
| INT64 | | | | | ✅ |
| INT96 (1) | | | | | ✅ |
| FLOAT | | | | | ✅ |
| DOUBLE | | | | | ✅ |
| BYTE_ARRAY | | | | | ✅ |
| FIXED_LEN_BYTE_ARRAY | | | | | ✅ |
| BOOLEAN | ✅ | ✅ | | | ✅ |
| INT32 | ✅ | ✅ | | | ✅ |
| INT64 | ✅ | ✅ | | | ✅ |
| INT96 (1) | ✅ | ✅ | | | ✅ |
| FLOAT | ✅ | ✅ | | | ✅ |
| DOUBLE | ✅ | ✅ | | | ✅ |
| BYTE_ARRAY | ✅ | ✅ | | | ✅ |
| FIXED_LEN_BYTE_ARRAY | ✅ | ✅ | | | ✅ |

* \(1) This type is deprecated, but as of 2024 it's common in currently produced parquet files

Expand All @@ -45,64 +45,66 @@ Implementations:

| Data type | C++ | Java | Go | Rust | cuDF |
| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- |
| STRING | | | | | ✅ |
| ENUM | | | | | ❌ |
| UUID | | | | | ❌ |
| 8, 16, 32, 64 bit signed and unsigned INT | | | | | ✅ |
| DECIMAL (INT32) | | | | | ✅ |
| DECIMAL (INT64) | | | | | ✅ |
| DECIMAL (BYTE_ARRAY) | | | | | ✅ |
| DECIMAL (FIXED_LEN_BYTE_ARRAY) | | | | | ✅ |
| DATE | | | | | ✅ |
| TIME (INT32) | | | | | ✅ |
| TIME (INT64) | | | | | ✅ |
| TIMESTAMP (INT64) | | | | | ✅ |
| INTERVAL | | | | | ❌ |
| JSON | | | | | ❌ |
| BSON | | | | | ❌ |
| LIST | | | | | ✅ |
| MAP | | | | | ✅ |
| UNKNOWN (always null) | | | | | ✅ |
| FLOAT16 | | | | | ✅ |
| STRING | ✅ | ✅ | | | ✅ |
| ENUM | ❌ | ✅ | | | ❌ |
| UUID | ❌ | ✅ | | | ❌ |
| 8, 16, 32, 64 bit signed and unsigned INT | ✅ | ✅ | | | ✅ |
| DECIMAL (INT32) | ✅ | ✅ | | | ✅ |
| DECIMAL (INT64) | ✅ | ✅ | | | ✅ |
| DECIMAL (BYTE_ARRAY) | ✅ | ✅ | | | ✅ |
| DECIMAL (FIXED_LEN_BYTE_ARRAY) | ✅ | ✅ | | | ✅ |
| DATE | ✅ | ✅ | | | ✅ |
| TIME (INT32) | ✅ | ✅ | | | ✅ |
| TIME (INT64) | ✅ | ✅ | | | ✅ |
| TIMESTAMP (INT64) | ✅ | ✅ | | | ✅ |
Copy link
Contributor

@Fokko Fokko Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we split this out on the unit as well?

Suggested change
| TIMESTAMP (INT64) | ✅ | ✅ | | | ✅ |
| TIMESTAMP (INT64, MICROS) | ✅ | ✅ | | | ✅ |

Copy link
Member Author

@wgtmac wgtmac Feb 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is that? Any unsupported unit in parquet-java?

Copy link
Contributor

@Fokko Fokko Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but I'm not sure if all the other implementations support nanos since that was added later on.

Copy link
Member Author

@wgtmac wgtmac Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was added almost 7 years ago: apache/parquet-format@b879065. We can be explicit if we really see an unsupported implementation then.

| INTERVAL | ✅ | ✅(*)| | | ❌ |
| JSON | ✅ | ✅(*)| | | ❌ |
| BSON | ❌ | ✅(*)| | | ❌ |
| LIST | ✅ | ✅ | | | ✅ |
| MAP | ✅ | ✅ | | | ✅ |
| UNKNOWN (always null) | ✅ | ✅ | | | ✅ |
| FLOAT16 | ✅ | ✅(*)| | | ✅ |

(*): Only supported to use its annotated physical type

### Encodings

| Encoding | C++ | Java | Go | Rust | cuDF |
| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- |
| PLAIN | | | | | ✅ |
| PLAIN_DICTIONARY | | | | | ✅ |
| RLE_DICTIONARY | | | | | ✅ |
| RLE | | | | | ✅ |
| BIT_PACKED (deprecated) | | | | | (R) |
| DELTA_BINARY_PACKED | | | | | ✅ |
| DELTA_LENGTH_BYTE_ARRAY | | | | | ✅ |
| DELTA_BYTE_ARRAY | | | | | ✅ |
| BYTE_STREAM_SPLIT | | | | | ✅ |
| PLAIN | ✅ | ✅ | | | ✅ |
| PLAIN_DICTIONARY | ✅ | ✅ | | | ✅ |
| RLE_DICTIONARY | ✅ | ✅ | | | ✅ |
| RLE | ✅ | ✅ | | | ✅ |
| BIT_PACKED (deprecated) | ✅ | ✅ | | | (R) |
| DELTA_BINARY_PACKED | ✅ | ✅ | | | ✅ |
| DELTA_LENGTH_BYTE_ARRAY | ✅ | ✅ | | | ✅ |
| DELTA_BYTE_ARRAY | ✅ | ✅ | | | ✅ |
| BYTE_STREAM_SPLIT | ✅ | ✅ | | | ✅ |

### Compressions

| Compression | C++ | Java | Go | Rust | cuDF |
| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- |
| UNCOMPRESSED | | | | | ✅ |
| BROTLI | | | | | (R) |
| GZIP | | | | | (R) |
| LZ4 (deprecated) | | | | | ❌ |
| LZ4_RAW | | | | | ✅ |
| LZO | | | | | ❌ |
| SNAPPY | | | | | ✅ |
| ZSTD | | | | | ✅ |
| UNCOMPRESSED | ✅ | ✅ | | | ✅ |
| BROTLI | ✅ | ✅ | | | (R) |
| GZIP | ✅ | ✅ | | | (R) |
| LZ4 (deprecated) | ✅ | ❌ | | | ❌ |
| LZ4_RAW | ✅ | ✅ | | | ✅ |
| LZO | ❌ | ❌ | | | ❌ |
| SNAPPY | ✅ | ✅ | | | ✅ |
| ZSTD | ✅ | ✅ | | | ✅ |

### Other format level features

| | C++ | Java | Go | Rust | cuDF |
| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- |
| xxHash-based bloom filters | | | | | (R) |
| Bloom filter length (1) | | | | | (R) |
| Statistics min_value, max_value | | | | | ✅ |
| Page index | | | | | ✅ |
| Page CRC32 checksum | | | | | ❌ |
| Modular encryption | | | | | ❌ |
| Size statistics (2) | | | | | ✅ |
| xxHash-based bloom filters | (R) | ✅ | | | (R) |
| Bloom filter length (1) | (R) | ✅ | | | (R) |
| Statistics min_value, max_value | ✅ | ✅ | | | ✅ |
| Page index | ✅ | ✅ | | | ✅ |
| Page CRC32 checksum | ✅ | ✅ | | | ❌ |
| Modular encryption | ✅ | ✅ | | | ❌ |
| Size statistics (2) | ✅ | ✅ | | | ✅ |


* \(1) In parquet.thrift: ColumnMetaData->bloom_filter_length
Expand All @@ -113,12 +115,12 @@ Implementations:

| Format | C++ | Java | Go | Rust | cuDF |
| -------------------------------------------- | ----- | ----- | ----- | ----- | ----- |
| External column data (1) | | | | | (W) |
| Row group "Sorting column" metadata (2) | | | | | (W) |
| Row group pruning using statistics | | | | | ✅ |
| Row group pruning using bloom filter | | | | | ✅ |
| Reading select columns only | | | | | ✅ |
| Page pruning using statistics | | | | | ❌ |
| External column data (1) | ✅ | ✅ | | | (W) |
| Row group "Sorting column" metadata (2) | ✅ | ❌ | | | (W) |
| Row group pruning using statistics | ❌ | ✅ | | | ✅ |
| Row group pruning using bloom filter | ❌ | ✅ | | | ✅ |
| Reading select columns only | ✅ | ✅ | | | ✅ |
| Page pruning using statistics | ❌ | ✅ | | | ❌ |


* \(1) In parquet.thrift: ColumnChunk->file_path
Expand Down