Skip to content

GC StringViewArray in CoalesceBatchesStream #11587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 10 commits into from
Jul 25, 2024
Merged

GC StringViewArray in CoalesceBatchesStream #11587

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0fb5f7b
gc string view when appropriate
XiangpengHao Jul 21, 2024
1340a63
make clippy happy
XiangpengHao Jul 21, 2024
a55810e
address comments
XiangpengHao Jul 22, 2024
f886d50
make doc happy
XiangpengHao Jul 22, 2024
b397944
update style
XiangpengHao Jul 23, 2024
1fac94a
Add comments and tests for gc_string_view_batch
alamb Jul 24, 2024
de0c84a
better herustic
XiangpengHao Jul 25, 2024
2c882d6
Merge pull request #1 from alamb/alamb/string-view-gc-tests
XiangpengHao Jul 25, 2024
8ab087c
update test
XiangpengHao Jul 25, 2024
8e0eaa6
Update datafusion/physical-plan/src/coalesce_batches.rs
XiangpengHao Jul 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
make doc happy
  • Loading branch information
@XiangpengHao
XiangpengHao committed Jul 22, 2024
commit f886d5032dc68a6a2fbb42be5d378485c8b2ab26
2 changes: 1 addition & 1 deletion datafusion/physical-plan/src/coalesce_batches.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ pub fn concat_batches(
arrow::compute::concat_batches(schema, batches)
}

/// [`StringViewArray`] reference to the raw parquet decoded buffer, which reduces copy but prevents those buffer from being released.
/// `StringViewArray` reference to the raw parquet decoded buffer, which reduces copy but prevents those buffer from being released.
/// When `StringViewArray`'s cardinality significantly drops (e.g., after `FilterExec` or `HashJoinExec` or many others),
/// we should consider consolidating it so that we can release the buffer to reduce memory usage and improve string locality for better performance.
fn gc_string_view_batch(batch: &RecordBatch) -> RecordBatch {
Expand Down