Skip to content

[thrift-remodel] Complete decoding of FileMetaData and RowGroupMetaData #8111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
etseidl merged 23 commits into from
Aug 15, 2025
Merged

[thrift-remodel] Complete decoding of FileMetaData and RowGroupMetaData #8111

Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
502c679
first cut at reading straight to ParquetMetaData
etseidl Aug 11, 2025
e6cfce1
metadata got a little smaller
etseidl Aug 11, 2025
6072693
use macro for SchemaElement
etseidl Aug 11, 2025
575a7cf
make names a bit clearer
etseidl Aug 11, 2025
434c5e5
refactor schema_from_thrift_input
etseidl Aug 11, 2025
0c3a5b0
add first test
etseidl Aug 12, 2025
ae069af
make file_path a slice
etseidl Aug 12, 2025
c9284f2
add macro for mixed unions
etseidl Aug 12, 2025
52ad234
cleanup and fix union bug
etseidl Aug 13, 2025
cba0561
list changes and move OrderedF64
etseidl Aug 13, 2025
6c5d58f
oops
etseidl Aug 13, 2025
36450c5
replace KeyValue with macro
etseidl Aug 13, 2025
b24b358
Go back to first fully decoding thrift, and then transforming
etseidl Aug 14, 2025
cec9815
clippy
etseidl Aug 14, 2025
6538dd4
start moving generated structs to their own module
etseidl Aug 14, 2025
8b1cfdb
Merge remote-tracking branch 'origin/main' into redo_filemetadata
etseidl Aug 14, 2025
4fa5167
Merge remote-tracking branch 'origin/gh5854_thrift_remodel' into redo…
etseidl Aug 14, 2025
4a1384e
fix some comments
etseidl Aug 14, 2025
e497133
add some explanation and todos
etseidl Aug 14, 2025
8a26462
add try_from for ParquetMetaData
etseidl Aug 14, 2025
cefeba4
oof
etseidl Aug 14, 2025
d0b6ffc
take ownership of schema elements to avoid clone
etseidl Aug 15, 2025
390e883
allow for list of binary with lifetime
etseidl Aug 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
make file_path a slice
  • Loading branch information
@etseidl
etseidl committed Aug 12, 2025
commit ae069af36f60497791b9abc9d794a01e19cb5bec
2 changes: 1 addition & 1 deletion parquet/src/file/metadata/reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1236,7 +1236,7 @@ struct RowGroup<'a> {
#[cfg(feature = "encryption")]
Copy link
Contributor

@alamb alamb Aug 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I suggest moving the code that does the thrift decoding into a different module from the ParquetMetaDataReader (which is basically a state machine for reading parts of the file into local buffers and decoding it)

Maybe parquet/src/file/metadata/from_thrift.rs 🤔

Copy link
Contributor Author

@etseidl etseidl Aug 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I want to consolidate at some point...I'm just not sure where it should live. There's basic already for the low level enums and unions, with structs scattered about where they're used. Pulling them all into a single module (or maybe two) is a good idea.

Copy link
Contributor Author

@etseidl etseidl Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now (largely) done. There are some exceptions that I've left as TODOs.

thrift_struct!(
struct ColumnChunk<'a> {
1: optional string file_path
1: optional string<'a> file_path
2: required i64 file_offset = 0
3: optional ColumnMetaData<'a> meta_data
4: optional i64 offset_index_offset
Expand Down
Loading