Repository navigation
feat(agent): per-call prediction hooks on predict_long, and no window for a skipped answer - #577
Conversation
…ped answer predict_long was the only inference entry point that accepted none of the per-call hook arguments, so a hook installed per call had to be promoted to agent state just to instrument a long-document scan. The hooks wrap the inference that answers the state: one call on the fits-in-one-window path, one shared predict_batch over the windows otherwise. A start hook that calls ctx.skip(...) is answering the document, not a window, so its payload is returned unattributed with usage["windows"] = 0 instead of being aggregated into a deciding-window entry for windows that never scored it.
|
Thanks for picking this up — One consequence of adding The short-state path short-circuits to
So a caller who writes the natural check for "was this actually scanned" — if result["usage"]["windows"] == 0: # KeyError on every short stategets a Two ways out, and I do not have a strong preference:
(1) changes an output shape that has shipped, so it is your call rather than mine; if you want it, it belongs in this PR rather than a follow-up, since this is the change that makes the distinction observable. Nothing else from my side: forwarding the hooks into the short-circuit |
A state that fits one window returned no windows key at all, so the natural gate (usage["windows"] == 0) raised KeyError on the shortest, most common inputs and could not tell "a hook answered it" from "it was scanned". predict_long now reports 1 there, keeping 0 for a hook that answered a multi-window document and N for a scan, and the copy-then-write means a start hook's own payload dict is never mutated.
|
Short answer: took (1). Measured on the
The answers didn't move — both runs choose One consequence the short-circuit makes necessary: a start hook that answers there returns the caller's own payload dict, so The README's "identical output" sentence and the |
|
The count check compares Driving the real
That reads against the docstring at |
|
@Bruce-Yii is right, and the root cause is worth naming precisely: The real signal is reachable without changing the hook API, using the same trick #497 uses on the router path: append an internal start hook last and let it observe what
So the observer records
The middle row is the one I would not silently aggregate with attribution even once the count check is fixed: Happy to send this as a patch against your branch rather than a follow-up PR, if you would rather not rework it yourself — it is the observer hook, the two fields it records, and dropping the attribution on the rewritten path. Your call; it is your PR and you have done the harder part. |
…t count A start hook that answered a long state and a start hook that replaced the window list both return fewer results than there were windows, and the count could not tell them apart: the rewrite was reported to the caller as "a hook answered before the state was scanned", and the tokens it did book were dropped. predict_long now appends its own recorder after the caller's start hooks and reads ctx.results from the engine's own context, so only a real answer takes the unattributed path and a rewritten list raises with the split it could not attribute. The same rule covers the fits-in-one-window path, where a hook answer now reports 0 windows like every other hook answer, and the window list is copied before the hooks see it, so an in-place append can no longer move the split the answers are attributed to.
The check that read a count of one as a hook answer still used the count as evidence. A start hook may replace `ctx.states` (`docs/hooks/api.md`), so narrowing the scan to one window is a supported outcome and not an answer: a forward ran and booked tokens, and raising for two windows discarded work the caller had already paid for. `predict_long` now asks the observer appended after the caller's own start hooks what reached inference, which makes the three cases total: a `ctx.skip()` payload comes back with no attribution and 0 windows; an unchanged scan aggregates with the deciding window named; a rewritten scan aggregates over the states that were scored, with `usage["windows"]` as their count and no `answer["window"]` -- the token offsets describe the windows this method built, not the text the model read. A count the observer cannot explain (no hook chain ran) stays an error.
… count it reports The aliasing bug is visible only as a type: without the copy of the split, `ctx.states.append()` grows both lists, the counts agree, and the attribution indexes a shorter `starts`. So the check asserts the call returns at all before it asserts what it returned.
|
Short answer: adopted the observer, dropped the raise. The start hook records both things the reviewers named:
The last two differ only by a list comparison against the split this method computed, so a text-only rewrite would attribute answers to windows that were not scored. Driving the real
Two deliberate differences, easy to reverse:
One judgement call worth your eye: a hook that empties Verification, all stubbed-forward, so it's about branching and shape, not decision quality:
The observer and the two fields are the reviewers' design. No patch needed — but do say if you'd rather the empty scan raise. |
|
I’d keep the empty-scan behavior as it is now. Once the observer has
established that the hook chain ran and left `ctx.states` empty, that
is an explainable hook outcome rather than an unexplained result-count
mismatch. `windows: 0` + no window attribution + a `RuntimeWarning`
reports what actually happened without inventing inference. I’d keep
the `ValueError` for the cases the observer cannot explain, rather
than restoring it for an empty rewritten scan.
…On Sat, 26 Sep 2026 22:15:57 -0700, Aashish Mahato ***@***.***> wrote:
aashish254 left a comment [(NandhaKishorM/laya#577)](#577 (comment))
Adopted the observer, and dropped the raise. 9ae2318 (plus c12f701, one more check) is on the
branch now; the result count is no longer read as evidence anywhere in predict_long.
What the appended start hook records is both things ***@***.***(https://github.com/AlKor13) named — answered = ctx.results is not None, and a snapshot of ctx.states — and the three rows of your table are the shape of the code:
what the start hook did
usage["windows"]
answer["window"]
answered with ctx.skip([result])
0
absent
left the scan as built
N
present
replaced the scan, in any way
states scored
absent
The difference between the last two rows is a list comparison against the split this method
computed, so a text-only rewrite drops the attribution too: the offsets name windows that were not
scored, and reporting them would be the same lie in a smaller form. docs/hooks/api.md now carries
that table, the README bullet says attribution survives only an unchanged scan, and the docstring
enumerates the three outcomes.
Against ***@***.***(https://github.com/Bruce-Yii)'s two bullets, driving the real predict_batch with only _encode_state,
_forward and _decode_answers stubbed:
- narrowing ctx.states to one window returns windows: 1 with the tokens that forward booked and
no RuntimeWarning. On 9a52a22 the two checks that pin this fail as reported
(narrow/the count is the state scored, not the split made: got 0 want 1,
narrow/a rewrite is not warned about as a hook answer: got ['RuntimeWarning'] want []).
- narrowing to two returns windows: 2, and nothing raises after the forwards have run
(narrow/two states left behind are counted as two fails on every earlier revision).
Two deliberate differences from the proposal, both easy to reverse if you disagree:
- The ValueError stays for one case: a count the observer cannot explain. If no hook chain ran
at all — predict_batch replaced or overridden, which is what the other harness in
tests/test_predict_long.py does — then a result count that is not the split is a bug rather
than a hook outcome, and guessing over it would put the original failure in a new place. The
message now says which of the two it saw (no start hook chain ran, so nothing rewrote the scan).
- predict_batch receives list(windows) rather than windows. Without that copy the observer
compares a list against itself: an in-place ctx.states.append(...) grew the split too, so the
counts agreed and the attribution indexed a shorter starts. On 9a52a22 that surfaces as
grow/scoring one invented state is not an error: got 'IndexError' want None — an
IndexError: list index out of range from inside predict_long, at the token_start line.
With the copy the invented state is scored, counted as N+1, and attributed to nothing.
One judgement call worth your eye: a hook that empties ctx.states now returns
{"answers": {}, "usage": {"input_tokens": 0, ..., "windows": 0}} with a RuntimeWarning, where
every earlier revision raised ValueError. Nothing was scored, so 0 is the honest count, and an
empty answer set the caller is warned about seemed better than a crash that invents nothing — but
that is the one place where I would take the opposite instruction.
Verification, all of it stubbed-forward so no weights are involved; this is a claim about branching
and shape, not about decision quality:
- python tests/test_predict_long.py: 79 checks, 0 failed (42 on 9a52a22). Run against
9a52a22 — the code ***@***.***(https://github.com/Bruce-Yii) tested — the new checks fail 14; run against the intermediate
bd18c47, which kept the raise, they fail 15. The tests gate the design, not just the diff.
- Narrow mutation gate, 8 single-line mutants of laya/agent.py, each killed and each by a distinct
named check: count-only heuristic → narrow/the count is the state scored, not the split made;
observer records no answer → skip/one payload for the document is never refused as a count mismatch; observer prepended before the caller's hooks → hooks/the caller's start hook still runs first; rewritten forced false → narrow/one state left behind is a scan, not a rejected count; rewritten forced true → scan/the deciding window is the last; the list(windows) copy
removed → grow/scoring one invented state is not an error; usage["windows"] taken from the
split instead of the results → narrow/two states left behind are counted as two; windows: 1 on
the short path → short skip/a hook answer is not counted as a window read.
- Adjacent lanes after the change, all 0 failed: test_hooks.py 162, test_hooks_api.py 140,
test_router.py 546, test_doc_tables.py 109, test_structured.py 58, test_runtime_fixes.py
33, test_cli.py 38, test_portability.py 26, test_predict_batch.py OK.
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 is clean.
***@***.***(https://github.com/AlKor13) the observer and the two fields are your design, so the shape here is yours; the part that
was left was agreeing the count had to stop being the witness, and that was ***@***.***(https://github.com/Bruce-Yii)'s message. No
patch needed — but do say if you would rather the empty scan raise.
—
Reply to this email directly, [view it on GitHub](#577?email_source=notifications&email_token=CHDJZC5U26Z23ZUOULCO6KD5RCPA3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOBVGI4TAMRUGA22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5852902405), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/CHDJZC5VW67HR6XKPPXXBST5RCPA3AVCNFSNUABGKJSXA33TNF2G64TZHMYTGNZVGM4DEMRVGM5US43TOVSTWNJVHE2TIMRYGQ2DHILWAI).
You are receiving this because you were mentioned.
|
|
Settled, and that is what the head already does — nothing to change.
So the judgement call is now a reviewer decision rather than mine, and the branch matches it. Thanks |
|
Checked out The remaining one is the
def answer_per_window(ctx): # what a cache hook naturally does
ctx.skip([cached for _ in ctx.states])
agent.predict_long({"body": "x" * 400}, Q, on_predict_start=answer_per_window)
# ValueError: predict_long: a start hook answered this state with 11 results;
# ctx.skip() takes one result for the document, not one per window
def answer_one_document(ctx):
ctx.skip([cached])
# -> windows=0, the document answer, RuntimeWarning as designedSo a hook installed on the Agent — telemetry-plus-cache is the obvious one, and #590 is about exactly that kind of hook — works on The observer already carries what settles it, since
Treating the first row like scored results is what makes a cache useful: the served answer and the cached answer come back in the same shape, which is the whole point of caching the call. And the ambiguity at Happy to send this as a patch to your branch; it is a few lines in the same block plus two cases in |
|
Agreed. The important invariant is that the hook contract should not
change just because the same logical call crosses the long-document
scan threshold. If `len(results) == len(evidence["states"])`, treating
that as the documented per-state `ctx.skip()` shape keeps cache hooks
reusable across `predict`, `predict_batch`, and `predict_long`;
`len(results) == 1` on a multi-window scan can remain the
document-level answer, and anything else can stay a `ValueError`. That
split is cleaner than making hook authors branch on `predict_long`
internals.
…On Sat, 26 Sep 2026 22:51:34 -0700, AlKor13 ***@***.***> wrote:
AlKor13 left a comment [(NandhaKishorM/laya#577)](#577 (comment))
Checked out c12f701 and drove it — your suite is 79/79, the observer does what the three rows say, and the single-window skip reporting 0 rather than 1 is a nicer call than anything in my table. Two of the three rows I could not fault.
The remaining one is the ValueError, and it is not about counting any more: it is that predict_long gives ctx.skip() a different contract from the call it is implemented on, and a hook cannot tell which contract it is under.
ctx.skip() is documented as "set cached results" (hooks.py:48-50) for a call whose results are one per state. A cache hook therefore answers with one result per state in ctx.states — correct against predict_batch, and correct against predict — and the same hook object raises the moment the state is long enough to be scanned. Driving your make_real_agent harness:
def answer_per_window(ctx): # what a cache hook naturally does
ctx.skip([cached for _ in ctx.states])
agent.predict_long({"body": "x" * 400}, Q, on_predict_start=answer_per_window)
# ValueError: predict_long: a start hook answered this state with 11 results;
# ctx.skip() takes one result for the document, not one per window
def answer_one_document(ctx):
ctx.skip([cached])
# -> windows=0, the document answer, RuntimeWarning as designed
So a hook installed on the Agent — telemetry-plus-cache is the obvious one, and [#590](#590) is about exactly that kind of hook — works on predict, works on predict_batch, and raises on predict_long for a long input only. That is a sharp edge for the caller who least expects it: the failure appears when a document grows, not when the hook is written.
The observer already carries what settles it, since evidence["states"] is the list the hook actually saw:
len(results)
meaning
windows
answer["window"]
len(evidence["states"])
per-state results, the documented skip() shape
that many
present only if the scan is unchanged
1, when the split made more than one window
one answer for the document
0
absent
anything else
genuinely inconsistent
ValueError, as now
Treating the first row like scored results is what makes a cache useful: the served answer and the cached answer come back in the same shape, which is the whole point of caching the call. And the ambiguity at len(windows) == 1 disappears on its own, because that case never reaches here — the short-circuit above handles it, and it already distinguishes the two by evidence["answered"].
Happy to send this as a patch to your branch; it is a few lines in the same block plus two cases in test_predict_long.py. Your PR, your call — and either way the count-as-evidence problem is gone, which was the thing worth fixing.
—
Reply to this email directly, [view it on GitHub](#577?email_source=notifications&email_token=CHDJZCY777FR4XGDBKWT6ZD5RCTGNA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOBVGMYTGNRQGIZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5853136022), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/CHDJZC23UWRFYVFUZSPXZQ35RCTGNAVCNFSNUABGKJSXA33TNF2G64TZHMYTGNZVGM4DEMRVGM5US43TOVSTWNJVHE2TIMRYGQ2DHILWAI).
You are receiving this because you were mentioned.
|
…he batch merges - ONNXAgent.predict_long takes the per-call hooks Agent.predict_long gained in #577, with the same usage["windows"] contract (1 for one window, 0 for a hook's answer, no window attribution on a rewritten scan). tests/test_onnx_long.py's parity check caught the gap. - decide_batch takes min_confidence like decide (#361), and ONNXAgent gets decide_batch now that it has predict_batch (#489). The docs no longer say ONNXAgent cannot batch. - test_structured_docs follows decide's signature, and the API table names callables that exist (agent./router.decide_batch). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Thanks @aashish254, merged. I ported the same hook contract to |
Router.route_batch takes hooks_timeout (router.py:911, docstring 925-926) and applies it per-request during on_route dispatch. The batch sibling laya_predict_batch has always had the override plumbed; laya_route_batch did not. The tool layer accepted only `requests`, so an operator running a routing sweep over a workload cannot protect against a slow or hung operator hook the way they can on the predict path. - New optional `hooks_timeout` arg on laya_route_batch; forwarded to Router.route_batch only when the caller actually asked for one, so a pre-#577 attached Router keeps working. - _validate_hooks_timeout wraps core's validate_timeout and turns its ValueError into a ToolError code (`invalid_hooks_timeout`) at the MCP boundary. Bools rejected here (not by core, which coerces True -> 1.0). - Server wrapper exposes hooks_timeout; the tool description now mentions it so clients discover the knob. - Tests: BatchRouter.route_batch records the timeout; six new checks cover forwarding (float + coerced int), default-stays-unset, validation errors, and a strict-signature Router stub as the witness against always-kwarg. Four mutations each produce a named FAIL.
What
laya.agent.Agent.predict_longis the only inference entry point in the package that acceptsnone of the per-call hook arguments.
predict,system_oneandpredict_batchall takehooks,on_predict_start,on_predict_end,hooks_raiseandhooks_timeout; thelong-document path takes
langand nothing else, so the same call that works on every siblingis a
TypeErrorthere:That is #489's shape: the core has the control, one entry point does not forward it. The hooks
are the documented place for the request-scoped work a long-document caller most wants —
PII redaction before inference, a cache read via
ctx.skip(...), audit logging — and an Agentthat builds them per call cannot reach the scan at all. Instance-level hooks (
Agent(hooks=[…]))already fire inside
predict_long, so this is not new machinery; it is the same dispatch point,made reachable per call, plus the contract written down where it was missing.
Second thing the probe turned up. Because the split into windows happens after
on_predict_start, a start hook that answers the document returns one result where the scanexpects one per window, and
predict_longaggregated that payload anyway:input_tokens: 1proves inference never ran, whilewindownamed a deciding window andwindows: 3counted windows — so the response asserted where in the document the model lookedfor a decision it never made. A skipped payload of exactly one result is now passed through
unattributed with a
RuntimeWarning, and a count matching neither the document nor the windowsraises instead of picking between answers that were never scored. Both halves are one logical
change — hooks on the scan path, and what a hook's answer is allowed to claim — so they are one
commit.
Measured, on the real checkpoint
laya(English), CPU, M-series, upstreammain@4066d5d. A 3,441-character state (twoquestions, one scanned path,
max_len512 /head_max_len192 / state budget 312 tokens → thesplit produced 3 windows):
predict_long(state, q, hooks=[h])TypeErrorlen(ctx.states) == 3, decoded window texts, lengths 1928 / 1922 / 1513ctx.skipanswer'swindow{'index': 0, 'token_start': 0, 'token_end': 312, 'count': 3}usage["windows"]on that answer30{'cancel': True, 'department': 'billing'}{'cancel': True, 'department': 'billing'}— identicalpredict_longDecisions compared at the level a caller acts on (
choicelabel,bool(noul >= 0.5)), not rawfloats. The two medians are the same call; the ordering between them is noise, and forwarding
five keyword arguments onto an existing call site is not a hot path.
Tests
tests/test_predict_long.py: 13 → 32 checks, weight-free on the file's existing stub(
Agent.__new__+ a token-count-equals-character-count tokenizer), covering: the five controlsreach the scan and the single-window branch; window texts (not the caller's state) are what the
hook layer gets; forwarding is decision-neutral; a skipped document answer comes back with no
windowandwindows: 0, and warns; a mismatching count raises.tests/test_hooks_api.py: 131 → 140 checks.Agent.predict_longjoins the predict-surfaceloop that pins the five hook parameters, and
window/strideplus the absence ofmax_len/head_max_lenare pinned (the scan sizes windows from the checkpoint budget, so aper-call token override would mean something different there).
Six mutations of
laya/agent.py, each named by the tests it fails — nothing here rests on apassing count:
hooks/forwarded to system_one,hooks/the hook list reaches it there,hooks/hooks_timeout reaches it therehooks/forwarded to the scan,hooks/the hook list reaches it,hooks/hooks_raise reaches it,hooks/hooks_timeout reaches itskip/no window was scored,skip/no deciding window is claimed,skip/the caller is told,skip/rejects a count matching nothingwindowsreports the split instead of 0skip/no window was scoredskip/rejects a count matching nothingskip/the caller is toldGates
Docs
docs/hooks/api.mdgains theagent.predict_long(...)line, which the Agent block did not have,and states what a start hook sees on the scanned path. The README's
predict_longsection gainsthe matching bullet and a
hooks=example line.Checked, and left alone
laya.agent.Agent.predict_long(the otherpredict_longchanges are feat(onnx): add ONNXAgent.predict_long, the long-document API the torch Agent has #494/feat(onnx): port predict_batch sort_by_length to ONNXAgent #507's ONNX method and feat(router): predict_long — route, then scan the whole state in windows #497's router entry, neither of which touches this
function).
Router.predictwith a start hook that wraps route + scan,so a router-level
hooks=already applies to the whole call. Different level, same intent, nooverlap with this one.
ONNXAgent.predict_long, so it has the same hole. Oncefeat(onnx): add ONNXAgent.predict_long, the long-document API the torch Agent has #494 lands I will carry the same five arguments across; flagging it rather than leaving it
silent.
exactly as before, and the fast path only forwards arguments that the callee already accepts.
Nearest issue by title: #549 reports the served
/v1/systemonepath answering a long state fromits first 312 tokens. That lives in
laya/serve.pyand is what #550/#566 work on, not thisfunction — so no issue is closed here.
Added after @AlKor13's review:
usage["windows"]is now totalThe review found that adding
windows = 0for the skipped case made the key tri-state, because thesingle-window path never had it at all -- so
usage["windows"] == 0, the natural check for "a hookanswered this", raised
KeyErroron the shortest and most common inputs. Option (1) from thatreview is implemented here: the short-circuit reports
1.KeyError('windows')windows=1; same gate returnsFalsewindows=8windows=0Measured on the
layacheckpoint, cpu, same states and questions both sides; the answers areidentical (0.5810 / 0.6496 / 0.4200), so the change adds a key on one path and moves no decision.
The write goes through a copy, because a start hook that answers returns the caller's own payload
dict and that object may be cached -- the probe shows
same dict object: True -> False.README's "identical output" bullet and
docs/hooks/api.mdteach all three counts, andtests/test_predict_long.pypins them (32 -> 42 weight-free checks). A 9-variant mutation run(count not written, count
0, write in place, scanned count hardcoded, hook count counted as ascan,
system_oneusage dropped, README value changed, README totality sentence removed, docstringreverted) fails on every variant with the failing check named.