Skip to content

feat(gateway): extract memory from a gateway conversation by reusing the agent's prompt cache #2413

Open

Description

@jasper-yl

Feature description

In gateway mode, memory extraction can reuse the provider cache that the agent's own requests just wrote, instead of paying full price to re-read the conversation.

The agent's last request already holds the whole conversation as a cached prefix. An extraction request that is that exact request plus one appended user message (Core's extraction instruction) is billed almost entirely as cache reads.

Problem and proposed solution

Measured on OpenAI Responses with Codex 0.145 requests:

  • Appending one user message to the agent's last request hit 100% of the prefix in the cache.
  • tool_choice: "none" and max_output_tokens do not affect the hit.
  • These break it entirely (0% hit): structured output (text.format), a different reasoning.effort, a different prompt_cache_key, and removing the tool definitions.
  • In one run, the cache still hit after 5, 10 and 20 minutes idle. Providers do not guarantee this.

Real flow (extraction right after a SWE-bench Pro session, same account; extraction call only):

Session Appended to the cached prefix Same prompt over the normalized transcript, full price
ansible 6.8k units (31,625 of 32,269 input tokens cached) 15.2k
qutebrowser 7.8k units (37,630/38,269) 14.8k
openlibrary 9.0k units (32,413/33,633) 13.9k

Units are cache-aware: uncached input 1, cached 0.1, output 8. With the same instruction, both ways extracted nearly the same memories. The memory itself then saved 0.81× on later tasks (see #2412 for that measurement and the extraction categories).

Proposed design:

  1. Capture. At the end of an agent turn (a response without tool calls), the gateway captures a Source. Its content is the normalized conversation, which is the evidence (bug(gateway): sedimented gateway turns keep the raw response body and lose tool calls and outputs #2407). Its metadata carries the exact upstream request and response output of that last step, plus the provider-reported input_tokens.
  2. Extraction. When Core's extraction runs through the gateway, Core builds the request as that raw prefix plus its own extraction instruction. Core changes only tool_choice and max_output_tokens, and states the output format in the message, not in text.format. Core sends it with a marker header, so the gateway skips injection, sedimentation and capture for it; without the marker, Core's own calls get sedimented and extracted again in a loop, which we observed. The request still passes the gateway key and Core's check. Core owns the instruction, the parsing, validation, reconciliation and writes.
  3. Fallback. If anything fails (no prefix, context overflow, rejected request), Core removes the prefix and extracts from the normalized transcript as today.

Same account and model. The cache is per provider account and model, so this only helps when Core's extraction uses the agent's model and account. RFC 2199 keeps Core's background inference on its own boundary, not forced through the gateway, and this proposal keeps that as the default. It adds an opt-in: Core may use the gateway as its generation endpoint, for example when no generation model is configured in a combined deployment, and only then uses the cached-prefix path.

Caveats we hit:

Alternatives considered

  • Extracting from the normalized transcript only, as today: simpler, but 1.6–2.2× the extraction cost in these sessions, and it never benefits from the cache.
  • Having the gateway run extraction itself: rejected, because the prompts, schema and validation belong in Core.

Are you willing to submit a PR for this feature?

  • Yes, I would like to submit a PR.

Activity

  1. jasper-yl commented on Nov 3, 2026

    @jasper-yl
    ContributorAuthor

    @frf12 @hidb4ai With #2407, #2416 and #2419 merged, the gateway now has what this proposal needs: one capture per user turn, a normalized transcript, and a trusted conversation identity. Before writing code I'd like to agree on the design, since it touches RFC 2199's inference boundary.

    Opt-in only. Core's background inference keeps its own boundary by default. The cached-prefix path is used only when an operator configures Core to use the gateway as its generation endpoint and the extraction target equals the agent's target (same endpoint, model and account). Otherwise extraction runs exactly as today.

    The prefix is a temporary cache hint, never evidence.

    • At the terminal response of a turn, the gateway keeps the exact released request and response output in its own bounded memory, with a short TTL, keyed by an opaque handle. The handle is bound to the Scope, target, conversation and the release binding of that response.
    • The Source that Core receives (fix(gateway): capture each user turn once with a normalized transcript #2423) carries only the normalized transcript as content and the handle in its metadata. The raw prefix never enters Source content, metadata materialized into evidence, or reconciliation evidence.

    Authority comes from the binding, not a header.

    • Core asks the gateway for a prefixed extraction over the internal channel, presenting the handle and its own extraction instruction.
    • The gateway admits it only if the handle exists, has not expired, and matches the Scope, target and conversation.
    • The resulting request (prefix plus one appended user message) goes through Core's normal before_send/before_release checks like any other request. It is never injected, sedimented or captured.

    Core's structured-output contract stays enforced. The instruction states the output format in the message, because text.format breaks the cache. Core parses and validates the reply with the same schema and rules as the current extraction. If parsing or validation fails, Core discards the reply and extracts the normal way.

    Fallbacks, all to today's extraction, each with a reason code:

    • handle missing or expired;
    • target or account mismatch;
    • prefix plus instruction over the context window;
    • check withheld or blocked;
    • provider error;
    • output that fails validation.

    A cache miss is not a failure; it is reported.

    Metrics. Prefixed-extraction outcome (prefixed, fallback:<reason>), and the cached share of input tokens per extraction.

    Does this match what you'd accept? If so I'll implement it in that order: gateway handle and endpoint, Core opt-in configuration and fallback, then metrics. I'll measure cost against today's extraction on the same sessions.

  2. frf12 commented on Nov 3, 2026

    @frf12
    Member

    @jasper-yl Agreed, please go ahead with your latest proposal, assuming Gateway and Core are deployed together on the same host. We will handle acceptance validation on our side.

  3. moved this from Todo to In progress in PowerContexton Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions