Repository navigation
fix(examples): key the cache hook on what the answer depends on - #590
Conversation
A start hook that calls `ctx.skip()` hands back whatever its cache key matched, so the key is part of the hook contract. The shipped example and the two docs pages that teach it keyed on `json.dumps(..., sort_keys=True)`, which folds two criteria orders of the same choice question into one entry -- while `_question_schema` in `laya/router.py`, and `laya/evals.py`'s batch grouping, keep them apart on purpose because option order is positional. Measured on the English checkpoint: of five states, one changed label and one changed a 0.8 confidence gate on order alone, and the reordered call was served 0.661 confidence where its own fresh pass gave 0.4960. The key now takes the context, so the checkpoint and the per-call token budget are covered too, and `tests/test_hooks_api.py` checks all three copies against the core's own schema identity.
Hooks fire once per call, and predict_batch calls them with the whole list of states, so keying and skipping on one state answers a batch with one state's result: a warm 5-state batch came back with 1 result, and the entry it did write was keyed on states[0] while carrying the whole call's aggregated usage. PredictContext.skip now says what it replaces, and the three taught copies and the API snippet look up and store one entry per state, in the caller's order.
|
Short answer: one more commit here (8b332fe), because writing the key comment made me run the same hooks against predict_batch. The pattern only covered one state of a call that can carry a whole batch: hooks fire once per call, ctx.skip() replaces every result that call would have returned, and a warm 5-state batch came back with 1 result - state 0's answer, and results[4] an IndexError. That is a bigger consequence than the sort order, so the fix went in here rather than a competing PR on the same three files. Now: one key per state, read skips only when the whole call hits, write stores each state's own result, and PredictContext.skip's docstring says what it replaces. 131 -> 174 checks, 13 mutation variants all named. The body has the before/after tables. Heads-up on a collision: my own #532 appends four lines after the closing note of the patterns.md Caching block, which this commit rewrites - whichever lands second needs a small rebase. |
|
The three artifacts no longer share one shape. hits = [CACHE.get(key(s, ctx.questions)) for s in ctx.states]but Relatedly, |
docs/hooks/api.md still showed the pre-fix two-argument key, and no drift check covered that page, so the copy a reader makes from the API page dropped the checkpoint and token budget from the cache key. Rewrite the block as the taught pattern, add the page to the drift checks, and warm the second state in the example's batch so its comment about a served call is true. Add checks that run the demo against a stub agent.
|
Short answer: both points confirmed and fixed in fbe27d1.
What I could not measure: Adjacent suites: |
|
Thanks @aashish254, merged. It ships in 0.3.21. |
What
Two things are wrong with the cache hook the docs teach, and the second one is bigger than the first.
(1) Its key folds together two questions the engine refuses to treat as one. A start hook that
calls
ctx.skip(results)hands back whatever its cache key matched, so the key is part of the hookcontract. The shipped example and the two docs pages that teach the pattern
(
examples/hooks/cache.py,docs/hooks/patterns.md,docs/hooks/examples.md) built that key withjson.dumps(..., sort_keys=True). Sorting the payload makes two criteria orders of the samechoicequestion hash to one entry. The core deliberately does the opposite:_question_schema(
laya/router.py:135) keeps them apart because "option order is positional", which is howpredict_batchdecides which states may share a forward pass (#166), andlaya/evals.py:304usesthe same
sort_keys=Falserule for its batch grouping.(2) It covers one state of a call that can carry a whole batch. Hooks fire once per call
(
laya/agent.py:866), andpredict_batchcalls them with every state at once, so keying onctx.states[0]and skipping with one element answers a batch with one state's result.docs/hooks/api.mdalready says the skip list holds "per-state results, same shape aspredict_batch's return" -- the pattern it introduces does not do that.
Change
examples/hooks/cache.py, the two docs blocks and thectx.skipsnippet indocs/hooks/api.mdnowshare one shape:
cache_key(ctx, index)takes the state, the questions in the caller's ownorder, and
ctx.model/ctx.max_len/ctx.head_max_len. The checkpoint and the token budgetare in the key for the same reason order is -- on a Router one hook set serves three checkpoints,
and a smaller
max_lentruncates the state -- and the example's comment names the rule so nobodyreintroduces the sort.
readlooks up a key per state and skips only when every state ofthe call hits;
writestores each state's own result under its own key. A reordered batch isstill a hit, and the results come back in the caller's order.
laya/hooks.pygains one docstring paragraph: whatskip()replaces, and that a call can carrymany states. No behaviour in
laya/changes.One limit worth stating:
PredictContextcarries nolang, so a caller that varies the languageper call cannot key on it from a hook. The key covers what the context exposes.
Measured
(1) Criteria order is answer-bearing.
laya-bench/probe_cache_key_ordering_real.py, Englishcheckpoint, cpu, offline cache, gate at confidence >= 0.80. Same script on both trees; log at
laya-bench/cache_key_before_after.log. Decisions are compared at the level a caller reads: thelabel, and whether a confidence gate opens.
Same question, criteria written in the other order, five states:
Order alone moves the label on 1 of 5 and the gate decision on 1 of 5; every one of the five
confidences moves. So the claim is not "the model is unreliable", it is that criteria order is
answer-bearing, which is exactly the assumption the cache key has to match.
The documented cache over three calls -- question A, A again, then B (the reordered one):
The example's own print goes from
cache entries: 1(two identical calls) tocache entries: 2(two questions and one repeat). A hit on an identical call is unchanged, which isthe property the fix must not break.
(2) A warm batch returns one result for five states.
laya-bench/cache_batch_check.py, Englishcheckpoint, cpu, one
predict_batchcall of 5 states driven through the example's owncache_read/
cache_write:Before,
results[1]onward did not exist: the call was served byctx.skip([hit])with state 0'sanswer, and the entry it stored was keyed on state 0 while its
usageblock was the whole call'saggregate (180 tokens recorded against a 34-token decision). After, each state is stored and served
under its own key, and the batch keeps its length:
Gate
laya-bench/mutate_cache_key_gate.pyputs one property back into the wrong state at a time, in onefile at a time, and prints the checks each variant breaks.
tests/test_hooks_api.pygrows 43 checks(131 -> 174): the key builder and the read/write pair are exec'd out of each of the three files' own
text -- the example calls
laya.load()at module scope, and a docs block is not importable -- andrun against a real
PredictContext, because what a hook reads off the context is the contract.Per file: reorders stay distinct, and are distinct exactly when
_question_schemasays so; anothercheckpoint,
max_lenorhead_max_lenis a new entry; the key takes the state it describes; twostates of one call are two entries;
writestores one entry per state; a warm batch comes back perstate in the caller's order; a batch with one uncached state still runs; a single-state call is one
entry; an empty batch skips to
[]. Then the two docs pages are required to hash like the example,at index 0 and at index 1, so a page that drifts fails a suite CI already runs.
Full transcript with every failure name:
laya-bench/cache_key_mutations2.log.Gates run locally
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120-> All checks passed;python -m compileall -q laya/ tests/-> clean;python tests/test_hooks.py-> 162 passed, 0failed;
python tests/test_hooks_api.py-> 174 passed, 0 failed;python -m zensical build --strict --clean-> "No issues found", and the new sentences are in the builtsite/hooks/*/index.html.Overlap
Checked every open PR's diff for these files.
docs/hooks/examples.md,docs/hooks/api.mdandexamples/hooks/cache.pyare otherwise untouched; #577 edits a different section ofapi.md(thepredict_longevents), and no open PR other than this one changesPredictContext.skip.One collision to name: my own #532 appends a paragraph right after the closing note of the
patterns.mdCaching block, which this commit rewrites, so whichever of the two lands second needsa four-line rebase. #424 (hooks teardown) and #515 (per-call hook args) are nearby in
laya/hooks.pybut neither edits
skip.