Repository navigation
Description
Feature description
In gateway mode, memory extraction can reuse the provider cache that the agent's own requests just wrote, instead of paying full price to re-read the conversation.
The agent's last request already holds the whole conversation as a cached prefix. An extraction request that is that exact request plus one appended user message (Core's extraction instruction) is billed almost entirely as cache reads.
Problem and proposed solution
Measured on OpenAI Responses with Codex 0.145 requests:
- Appending one user message to the agent's last request hit 100% of the prefix in the cache.
tool_choice: "none"andmax_output_tokensdo not affect the hit.- These break it entirely (0% hit): structured output (
text.format), a differentreasoning.effort, a differentprompt_cache_key, and removing the tool definitions. - In one run, the cache still hit after 5, 10 and 20 minutes idle. Providers do not guarantee this.
Real flow (extraction right after a SWE-bench Pro session, same account; extraction call only):
| Session | Appended to the cached prefix | Same prompt over the normalized transcript, full price |
|---|---|---|
| ansible | 6.8k units (31,625 of 32,269 input tokens cached) | 15.2k |
| qutebrowser | 7.8k units (37,630/38,269) | 14.8k |
| openlibrary | 9.0k units (32,413/33,633) | 13.9k |
Units are cache-aware: uncached input 1, cached 0.1, output 8. With the same instruction, both ways extracted nearly the same memories. The memory itself then saved 0.81× on later tasks (see #2412 for that measurement and the extraction categories).
Proposed design:
- Capture. At the end of an agent turn (a response without tool calls), the gateway captures a Source. Its content is the normalized conversation, which is the evidence (bug(gateway): sedimented gateway turns keep the raw response body and lose tool calls and outputs #2407). Its metadata carries the exact upstream request and response output of that last step, plus the provider-reported
input_tokens. - Extraction. When Core's extraction runs through the gateway, Core builds the request as that raw prefix plus its own extraction instruction. Core changes only
tool_choiceandmax_output_tokens, and states the output format in the message, not intext.format. Core sends it with a marker header, so the gateway skips injection, sedimentation and capture for it; without the marker, Core's own calls get sedimented and extracted again in a loop, which we observed. The request still passes the gateway key and Core'scheck. Core owns the instruction, the parsing, validation, reconciliation and writes. - Fallback. If anything fails (no prefix, context overflow, rejected request), Core removes the prefix and extracts from the normalized transcript as today.
Same account and model. The cache is per provider account and model, so this only helps when Core's extraction uses the agent's model and account. RFC 2199 keeps Core's background inference on its own boundary, not forced through the gateway, and this proposal keeps that as the default. It adds an opt-in: Core may use the gateway as its generation endpoint, for example when no generation model is configured in a combined deployment, and only then uses the cached-prefix path.
Caveats we hit:
- The raw prefix must not reach the regular evidence path. Core materializes Source metadata into the evidence
content, and a prefix there blew the 24k reconciliation budget. - Reconciliation cost dominates when extraction yields many candidates (perf(memory): Atomic Memory reconciliation re-sends the whole source evidence at full price for every candidate #2414).
Alternatives considered
- Extracting from the normalized transcript only, as today: simpler, but 1.6–2.2× the extraction cost in these sessions, and it never benefits from the cache.
- Having the gateway run extraction itself: rejected, because the prompts, schema and validation belong in Core.
Are you willing to submit a PR for this feature?
- Yes, I would like to submit a PR.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsIn progress
Activity
jasper-yl commentedon Nov 3, 2026
@frf12 @hidb4ai With #2407, #2416 and #2419 merged, the gateway now has what this proposal needs: one capture per user turn, a normalized transcript, and a trusted conversation identity. Before writing code I'd like to agree on the design, since it touches RFC 2199's inference boundary.
Opt-in only. Core's background inference keeps its own boundary by default. The cached-prefix path is used only when an operator configures Core to use the gateway as its generation endpoint and the extraction target equals the agent's target (same endpoint, model and account). Otherwise extraction runs exactly as today.
The prefix is a temporary cache hint, never evidence.
Authority comes from the binding, not a header.
before_send/before_releasechecks like any other request. It is never injected, sedimented or captured.Core's structured-output contract stays enforced. The instruction states the output format in the message, because
text.formatbreaks the cache. Core parses and validates the reply with the same schema and rules as the current extraction. If parsing or validation fails, Core discards the reply and extracts the normal way.Fallbacks, all to today's extraction, each with a reason code:
A cache miss is not a failure; it is reported.
Metrics. Prefixed-extraction outcome (
prefixed,fallback:<reason>), and the cached share of input tokens per extraction.Does this match what you'd accept? If so I'll implement it in that order: gateway handle and endpoint, Core opt-in configuration and fallback, then metrics. I'll measure cost against today's extraction on the same sessions.
frf12 commentedon Nov 3, 2026
@jasper-yl Agreed, please go ahead with your latest proposal, assuming Gateway and Core are deployed together on the same host. We will handle acceptance validation on our side.