You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(memory): Atomic Memory reconciliation re-sends the whole source evidence at full price for every candidate #2414
Each Atomic Memory candidate is reconciled in its own model call. The input is AtomicMemoryReconciliationInput with fields proposal, related, evidence, serialized in that order. evidence holds the full Source content: for a coding session, the whole conversation transcript. It is identical for every candidate from the same Source, but it comes after proposal, which differs per call. So the provider can never cache it, and every call pays for the full evidence again.
Problem and proposed solution
In SWE-bench Pro sessions extracted through the gateway (cache-aware units: uncached input 1, cached 0.1, output 8):
Session
Extraction call
Reconciliation
ansible, 6 candidates
8.1k units
48.8k units (5 calls, 6,015 of 44,724 input tokens cached)
qutebrowser, 11 candidates
188k units for extraction and reconciliation together (1 + 11 calls)
Reconciliation cost grows linearly with the candidate count and dominated the cost of memory for these sessions. These two sessions used an extraction prompt that produced many candidates, but each candidate costs the same full evidence regardless of the prompt.
Proposal, cheapest first:
Serialize evidence before proposal and related, so all reconciliation calls for one Source share a prefix: Core's instructions plus the evidence. Providers with prefix caching then bill it as cache reads after the first call. This changes no semantics, only field order in the input.
Optionally, reconcile several candidates from the same Source in one call when the budget allows, as is already done for related items.
Alternatives considered
Capping the number of candidates: it reduces cost, but also loses memories. A better extraction prompt (#2412) also reduces low-value candidates.
Activity