Skip to content

Write Bloom filters between row groups instead of the end #5860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 16 commits into from
Jun 21, 2024
Merged

Write Bloom filters between row groups instead of the end #5860

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
759767b
Add example script to write Parquet files with a Bloom filter
progval Jun 10, 2024
a417f01
Write Bloom filters between row groups instead of the end
progval Jun 10, 2024
5daf96f
Merge branch 'master' into interleave-bloom
progval Jun 10, 2024
6effa7f
Add WriterProperties::bloom_filter_position
progval Jun 10, 2024
f23759a
Mutate the right row group metadata
progval Jun 10, 2024
83b475e
Add a test for Bloom Filters written at the end
progval Jun 10, 2024
3f810b5
Update async writer accordingly
progval Jun 10, 2024
ad0c40e
Undo accidental commit
progval Jun 10, 2024
74c40ee
Clippy
progval Jun 10, 2024
f237e8c
Apply suggestions from code review
progval Jun 13, 2024
d2a7ab8
Rewrite example with constants as parameters and fewer dependencies
progval Jun 13, 2024
b434eea
Merge branch 'master' into interleave-bloom
progval Jun 13, 2024
942c6ab
rustfmt
progval Jun 13, 2024
e4a588d
Clippy
progval Jun 13, 2024
ed9e576
Fix MSRV
progval Jun 13, 2024
abd81ef
Merge remote-tracking branch 'apache/master' into interleave-bloom
alamb Jun 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clippy
  • Loading branch information
@progval
progval committed Jun 10, 2024
commit 74c40ee8dae1db70e15eb818e3da53ac4a1b93a1
2 changes: 1 addition & 1 deletion parquet/src/file/writer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -434,7 +434,7 @@ impl<W: Write + Send> SerializedFileWriter<W> {
/// and returns the updated row group metadata.
fn write_bloom_filters<W: Write + Send>(
buf: &mut TrackedWrite<W>,
bloom_filters: &mut Vec<Vec<Option<Sbbf>>>,
bloom_filters: &mut [Vec<Option<Sbbf>>],
row_group: &mut RowGroupMetaData,
) -> Result<()> {
// iter row group
Expand Down