Repository navigation
Get rid of double range copy for aggregation operations over ranges - #352
Merged
Merged
Conversation
h-a-n-a-s-h-i
pushed a commit
that referenced
this pull request
Oct 11, 2026
Get rid of double range copy for aggregation operations over ranges # Conflicts: # crates/calc/src/tests.rs
Member
|
Merged — thank you for the contribution! If you haven't already, come join us on the ArtCraft Discord: https://discord.gg/artcraft |
Contributor
Author
|
@echelon thanks for quick merge! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Aggregations such as
SUM,COUNTandAVERAGEnow add up a range cell by cell instead of first copying the whole range into an array. This removes the 16M-cell limit for them and saves about 24 to 39 bytes of peak memory per range cell. The benchmark behind the numbers below is kept in this description, not in the changeset.Problem
On a dense 5000 x 5000 sheet (25M cells),
=SUM(A1:GJH5000)returned#NUM!, while the status bar showed Sum, Count and Average for the same selection.A formula's range is copied into an in-memory array before the function sees it. That copy is refused above
MAX_CELLS(16M cells,crates/calc/src/eval.rs:15).SUMthen copies the numbers a second time innumbers(). The status bar skips both copies because it reads the stored cells in place.The cap bites only when the sheet's used range itself is above 16M cells:
area_valuefirst trims large ranges to the used range, so sparse sheets were not affected.What changed
Evaluator,
crates/calc/src/eval.rsHostgets a default methodvisit_range()that returnsfalse, so other hosts behave as before.builtinhandsSUM,COUNTandAVERAGEto the newEvaluator::aggregate.Aggnames the function andAcccarries a sum and a count. Adding another aggregate means a newAggvariant and, if needed, another field inAcc.visit_range. Single cells, blocks the host declines, and non-reference arguments are added by calling the realSUMandCOUNTon them (AVERAGEof an argument is itsSUMover itsCOUNT), so coercion and error rules stay in one place.SUMandAVERAGEstop at the first error.COUNTskips errors, as the real functions do.B2 = RAND()*10,=SUM(B1:B2, RAND())must readB2before it draws the second random number. An earlier version of this change evaluated every argument first and got the draws the other way round.Calculation,
crates/calc/src/recalc.rsPassHost::visit_rangewalks the stored cells of a block in two passes and holds no per-cell data of its own (memory stays constant however many cells the block has):range_valuesdoes. Reading a cell calculates it and the pass remembers the value; this one only discards the result.range_values). The check comes after pass 1 because evaluating a formula in the block can itself produce a spill over the block.Results
Release build, 3000 x 3000 sheet of repeating digits 0..9 (9M cells), measured with the benchmark in the spoiler below. The array-path numbers come from
HEAD(before this change).Peak heap (heaptrack 1.5.0, one benchmark case per run,
HEADagainst the final code):SUMCOUNTAVERAGEvec![Value::Empty; h * w]inrange_values(216 MB in one call), plus, forSUMandAVERAGE, theVec<f64>thatnumbers()grows withpush(134 MB at its peak).COUNTdoesn't build that second vector, which is why it saves less.Benchmark used for the numbers (not part of this changeset)
crates/engine/benches/sum.rs(Divan) and how it was runAdd to
crates/engine/Cargo.toml:crates/engine/benches/sum.rs:Timings:
cargo bench -p gridcraft-engine --bench sum(add a name to filter).Peak heap, one case per run, once on the final code and once on an unmodified
HEADin a separate git worktree (withsum.rs, theCargo.tomllines andCargo.lockcopied in):Not changed
display::stats) don't use the evaluator.Please look at
PassHost::visit_range. It relies oniter_rangebeing row-major, so the first error and the order of float additions match the array path.Try it
Generate and ppen a sheet with more than 16M used cells and enter, e.g.,
=SUM(B2:GZA5500)in A1. Before this change it shows#NUM!; now it shows the sum.