Skip to content

fix(examples): key the cache hook on what the answer depends on - #590

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:fix-cache-key-question-order
Sep 27, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:fix-cache-key-question-order

Conversation

@aashish254

@aashish254 aashish254 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

What

Two things are wrong with the cache hook the docs teach, and the second one is bigger than the first.

(1) Its key folds together two questions the engine refuses to treat as one. A start hook that
calls ctx.skip(results) hands back whatever its cache key matched, so the key is part of the hook
contract. The shipped example and the two docs pages that teach the pattern
(examples/hooks/cache.py, docs/hooks/patterns.md, docs/hooks/examples.md) built that key with
json.dumps(..., sort_keys=True). Sorting the payload makes two criteria orders of the same
choice question hash to one entry. The core deliberately does the opposite: _question_schema
(laya/router.py:135) keeps them apart because "option order is positional", which is how
predict_batch decides which states may share a forward pass (#166), and laya/evals.py:304 uses
the same sort_keys=False rule for its batch grouping.

(2) It covers one state of a call that can carry a whole batch. Hooks fire once per call
(laya/agent.py:866), and predict_batch calls them with every state at once, so keying on
ctx.states[0] and skipping with one element answers a batch with one state's result.
docs/hooks/api.md already says the skip list holds "per-state results, same shape as
predict_batch's return" -- the pattern it introduces does not do that.

Change

examples/hooks/cache.py, the two docs blocks and the ctx.skip snippet in docs/hooks/api.md now
share one shape:

  • What a key covers. cache_key(ctx, index) takes the state, the questions in the caller's own
    order, and ctx.model / ctx.max_len / ctx.head_max_len. The checkpoint and the token budget
    are in the key for the same reason order is -- on a Router one hook set serves three checkpoints,
    and a smaller max_len truncates the state -- and the example's comment names the rule so nobody
    reintroduces the sort.
  • How many states it covers. read looks up a key per state and skips only when every state of
    the call hits; write stores each state's own result under its own key. A reordered batch is
    still a hit, and the results come back in the caller's order.

laya/hooks.py gains one docstring paragraph: what skip() replaces, and that a call can carry
many states. No behaviour in laya/ changes.

One limit worth stating: PredictContext carries no lang, so a caller that varies the language
per call cannot key on it from a hook. The key covers what the context exposes.

Measured

(1) Criteria order is answer-bearing. laya-bench/probe_cache_key_ordering_real.py, English
checkpoint, cpu, offline cache, gate at confidence >= 0.80. Same script on both trees; log at
laya-bench/cache_key_before_after.log. Decisions are compared at the level a caller reads: the
label, and whether a confidence gate opens.

Same question, criteria written in the other order, five states:

state label A -> B confidence A -> B gate A -> B
"already told the agent last week... payment still came out" billing -> billing 0.661 -> 0.496 escalate -> escalate
"please cancel my subscription" cancel -> cancel 0.822 -> 0.786 ship -> escalate
"support never answered my ticket and I want a refund" billing -> cancel 0.090 -> 0.113 escalate -> escalate
"the invoice doubled after I changed plans" billing -> billing 0.904 -> 0.885 ship -> ship
"I am moving abroad next month" billing -> billing 0.424 -> 0.267 escalate -> escalate

Order alone moves the label on 1 of 5 and the gate decision on 1 of 5; every one of the five
confidences moves. So the claim is not "the model is unreliable", it is that criteria order is
answer-bearing, which is exactly the assumption the cache key has to match.

The documented cache over three calls -- question A, A again, then B (the reordered one):

before after
cache entries 1 2
repeat of A still a hit yes yes
B answered with A's entry (conf 0.6611) its own pass (conf 0.4960)
B from a fresh pass 0.4960 0.4960
cache answered a different question yes no

The example's own print goes from cache entries: 1 (two identical calls) to
cache entries: 2 (two questions and one repeat). A hit on an identical call is unchanged, which is
the property the fix must not break.

(2) A warm batch returns one result for five states. laya-bench/cache_batch_check.py, English
checkpoint, cpu, one predict_batch call of 5 states driven through the example's own cache_read
/ cache_write:

before after
cache entries after a cold 5-state batch 1 5
results back from a warm 5-state batch 1 5
states whose served answer is their own 1 of 5 5 of 5

Before, results[1] onward did not exist: the call was served by ctx.skip([hit]) with state 0's
answer, and the entry it stored was keyed on state 0 while its usage block was the whole call's
aggregate (180 tokens recorded against a 34-token decision). After, each state is stored and served
under its own key, and the batch keeps its length:

results for a 2-state batch: 2
one answer per state, in the caller's order: ['other', 'refund']

Gate

laya-bench/mutate_cache_key_gate.py puts one property back into the wrong state at a time, in one
file at a time, and prints the checks each variant breaks. tests/test_hooks_api.py grows 43 checks
(131 -> 174): the key builder and the read/write pair are exec'd out of each of the three files' own
text -- the example calls laya.load() at module scope, and a docs block is not importable -- and
run against a real PredictContext, because what a hook reads off the context is the contract.

Per file: reorders stay distinct, and are distinct exactly when _question_schema says so; another
checkpoint, max_len or head_max_len is a new entry; the key takes the state it describes; two
states of one call are two entries; write stores one entry per state; a warm batch comes back per
state in the caller's order; a batch with one uncached state still runs; a single-state call is one
entry; an empty batch skips to []. Then the two docs pages are required to hash like the example,
at index 0 and at index 1, so a page that drifts fails a suite CI already runs.

0  as shipped                            exit=0  174 passed  failures: none
m1 example sorted again                  exit=1  168  reordered criteria is a new entry + 4
m2 patterns.md sorted again              exit=1  170  that page's two order checks + 2
m3 examples.md sorted again              exit=1  170  same three, that page
m4 key drops the checkpoint              exit=1  169  another checkpoint is a new entry + 4
m5 key drops the token budget            exit=1  172  another max_len / another head_max_len
m6 keyed on the object, not its contents exit=1  165  the same call is a hit + 8
m7 key pinned to the first state         exit=1  167  two states of one call are two entries, write
                                                      stores one entry per state: got 1 want 3, a
                                                      warm batch is served back per state, a batch
                                                      with one uncached state still runs, + 2
m8 write: one entry per call             exit=1  171  write stores one entry per state: got 1 want 3,
                                                      a warm batch is served back per state, + 1
m9 read: one key for a whole batch       exit=1  171  a warm batch is served back per state, in the
                                                      caller's order
m10 read: skips when any state hits      exit=1  172  a batch with one uncached state still runs, an
                                                      empty batch skips to an empty list
m11 patterns.md write: one per call      exit=1  171  that page's three batch checks
m12 patterns.md read: one key for a batch exit=1  172  a warm batch is served back per state, in the
                                                      caller's order, + a batch with one uncached state
m13 examples.md write: one per call      exit=1  171  that page's three batch checks
   restored: True

Full transcript with every failure name: laya-bench/cache_key_mutations2.log.

Gates run locally

ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 -> All checks passed;
python -m compileall -q laya/ tests/ -> clean; python tests/test_hooks.py -> 162 passed, 0
failed; python tests/test_hooks_api.py -> 174 passed, 0 failed; python -m zensical build --strict --clean -> "No issues found", and the new sentences are in the built site/hooks/*/index.html.

Overlap

Checked every open PR's diff for these files. docs/hooks/examples.md, docs/hooks/api.md and
examples/hooks/cache.py are otherwise untouched; #577 edits a different section of api.md (the
predict_long events), and no open PR other than this one changes PredictContext.skip.

One collision to name: my own #532 appends a paragraph right after the closing note of the
patterns.md Caching block, which this commit rewrites, so whichever of the two lands second needs
a four-line rebase. #424 (hooks teardown) and #515 (per-call hook args) are nearby in laya/hooks.py
but neither edits skip.

A start hook that calls `ctx.skip()` hands back whatever its cache key matched, so the key is
part of the hook contract. The shipped example and the two docs pages that teach it keyed on
`json.dumps(..., sort_keys=True)`, which folds two criteria orders of the same choice question
into one entry -- while `_question_schema` in `laya/router.py`, and `laya/evals.py`'s batch
grouping, keep them apart on purpose because option order is positional.

Measured on the English checkpoint: of five states, one changed label and one changed a 0.8
confidence gate on order alone, and the reordered call was served 0.661 confidence where its own
fresh pass gave 0.4960. The key now takes the context, so the checkpoint and the per-call token
budget are covered too, and `tests/test_hooks_api.py` checks all three copies against the core's
own schema identity.
Hooks fire once per call, and predict_batch calls them with the whole list of
states, so keying and skipping on one state answers a batch with one state's
result: a warm 5-state batch came back with 1 result, and the entry it did write
was keyed on states[0] while carrying the whole call's aggregated usage.

PredictContext.skip now says what it replaces, and the three taught copies and
the API snippet look up and store one entry per state, in the caller's order.
@aashish254

aashish254 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Short answer: one more commit here (8b332fe), because writing the key comment made me run the same hooks against predict_batch.

The pattern only covered one state of a call that can carry a whole batch: hooks fire once per call, ctx.skip() replaces every result that call would have returned, and a warm 5-state batch came back with 1 result - state 0's answer, and results[4] an IndexError. That is a bigger consequence than the sort order, so the fix went in here rather than a competing PR on the same three files.

Now: one key per state, read skips only when the whole call hits, write stores each state's own result, and PredictContext.skip's docstring says what it replaces. 131 -> 174 checks, 13 mutation variants all named. The body has the before/after tables.

Heads-up on a collision: my own #532 appends four lines after the closing note of the patterns.md Caching block, which this commit rewrites - whichever lands second needs a small rebase.

@Bruce-Yii

Copy link
Copy Markdown
Contributor

The three artifacts no longer share one shape. docs/hooks/api.md:69 teaches

hits = [CACHE.get(key(s, ctx.questions)) for s in ctx.states]

but key doesn't exist anywhere in the tree, and the two-argument (state, questions) form is exactly what this replaces: examples/hooks/cache.py:20 now takes (ctx, index) and folds ctx.model, ctx.max_len and ctx.head_max_len in alongside the state and questions. Copying the api.md snippet yields a key that treats one state as a hit across checkpoints and token budgets — the pre-#590 behaviour, from the page documenting the new one. The new drift checks cover the example and the two patterns/examples docs but not api.md, so nothing flags it.

Relatedly, examples/hooks/cache.py:63 says "both states are cached, so the whole call is served", but OTHER first appears on line 64 — lines 57-59 only warm STATE, and line 60 confirms two entries, neither of them OTHER. cache_read's all(...) is therefore False, the batch call runs a full forward pass, and the two answers it prints are fresh ones. One agent.system_one(OTHER, QUESTIONS) before line 64 would make the demo show what its comment says it shows.

docs/hooks/api.md still showed the pre-fix two-argument key, and no drift
check covered that page, so the copy a reader makes from the API page
dropped the checkpoint and token budget from the cache key. Rewrite the
block as the taught pattern, add the page to the drift checks, and warm
the second state in the example's batch so its comment about a served
call is true. Add checks that run the demo against a stub agent.
@aashish254

aashish254 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Short answer: both points confirmed and fixed in fbe27d1.

  1. docs/hooks/api.md taught the removed shape, and nothing could see it. The page's ctx.skip() block is now the full pattern the example uses — CACHE, key(ctx, index), cache_read, cache_write, and the laya.load(...) wiring — and tests/test_hooks_api.py execs it alongside the other copies. The gap is closed the structural way, not with an api.md-specific assertion: taught_cache() slices the block at its closing fence, the page is a fourth entry in TAUGHT, so every existing check runs against it. Failing-first, measured: adding the page to TAUGHT while leaving it untouched kills the suite with ValueError: substring not found at text.index("CACHE = {") in taught_cache — the right outcome for a snippet referencing a key and a CACHE the page never defines. After the rewrite: 193 passed, 0 failed. The same file at 8b332fe was 174 passed, so this adds 19 checks.

  2. The example's batch was not served — you were right that the comment described a state the demo never reached. Fixed as you suggested: one agent.system_one(OTHER, QUESTIONS) before the predict_batch. The claim is now checkable instead of asserted. SKIPS = [] next to CACHE is appended by cache_read when it skips, and the demo prints calls served without a forward pass: 2. tests/test_hooks_api.py runs it against a stub agent that counts its own forward passes, so no weights are needed: SKIPS == [1, 2], forwards [1, 1, 1], and the batch answers still come back in the caller's order. Mutation, your scenario exactly: delete the agent.system_one(OTHER, QUESTIONS) line and the suite reports 191 passed, 2 failed with SKIPS: got [1], want [1, 2] and forwards: got [1, 1, 2], want [1, 1, 1] — the batch running a real forward.

What I could not measure: cache entries: stays at 2 (the print sits before the new warm-up). Real weights abort on this machine inside torch's MPS backend (failed assertion 'Placeholder shape mismatches (expected 1 vs got tensorData with 16)') at the first forward pass, and the unmodified example at 8b332fe aborts identically with exit 134 — so the printouts above are from the stub harness, not the weights.

Adjacent suites: python tests/test_hooks.py -> 162 passed, python tests/test_doc_tables.py -> 109 passed, python tests/test_evals_api.py unaffected. CI's ruff check laya/ has nothing to lint here: the diff touches docs/, examples/ and tests/ only.

@NandhaKishorM
NandhaKishorM merged commit 1e28b6e into NandhaKishorM:main Sep 27, 2026
22 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254, merged. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants