Skip to content

perf(memory): Atomic Memory reconciliation re-sends the whole source evidence at full price for every candidate #2414

Open

Description

@jasper-yl

Feature description

Each Atomic Memory candidate is reconciled in its own model call. The input is AtomicMemoryReconciliationInput with fields proposal, related, evidence, serialized in that order. evidence holds the full Source content: for a coding session, the whole conversation transcript. It is identical for every candidate from the same Source, but it comes after proposal, which differs per call. So the provider can never cache it, and every call pays for the full evidence again.

Problem and proposed solution

In SWE-bench Pro sessions extracted through the gateway (cache-aware units: uncached input 1, cached 0.1, output 8):

Session Extraction call Reconciliation
ansible, 6 candidates 8.1k units 48.8k units (5 calls, 6,015 of 44,724 input tokens cached)
qutebrowser, 11 candidates 188k units for extraction and reconciliation together (1 + 11 calls)

Reconciliation cost grows linearly with the candidate count and dominated the cost of memory for these sessions. These two sessions used an extraction prompt that produced many candidates, but each candidate costs the same full evidence regardless of the prompt.

Proposal, cheapest first:

  1. Serialize evidence before proposal and related, so all reconciliation calls for one Source share a prefix: Core's instructions plus the evidence. Providers with prefix caching then bill it as cache reads after the first call. This changes no semantics, only field order in the input.
  2. Optionally, reconcile several candidates from the same Source in one call when the budget allows, as is already done for related items.

Alternatives considered

Capping the number of candidates: it reduces cost, but also loses memories. A better extraction prompt (#2412) also reduces low-value candidates.

Are you willing to submit a PR for this feature?

  • Yes, I would like to submit a PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions