Skip to content

feat(agent): per-call prediction hooks on predict_long, and no window for a skipped answer - #577

Merged
NandhaKishorM merged 7 commits into
NandhaKishorM:mainfrom
aashish254:feat-predict-long-hooks
Sep 27, 2026
Merged

NandhaKishorM merged 7 commits into
NandhaKishorM:mainfrom
aashish254:feat-predict-long-hooks

Conversation

@aashish254

@aashish254 aashish254 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

What

laya.agent.Agent.predict_long is the only inference entry point in the package that accepts
none of the per-call hook arguments. predict, system_one and predict_batch all take
hooks, on_predict_start, on_predict_end, hooks_raise and hooks_timeout; the
long-document path takes lang and nothing else, so the same call that works on every sibling
is a TypeError there:

predict        hook args: 5      system_one     hook args: 5      predict_batch  hook args: 5
predict_long   hook args: 0      decide         hook args: 0 (**predict_kwargs, so it forwards)

predict        accepted hooks= (hook fired 1 time(s))
system_one     accepted hooks= (hook fired 1 time(s))
predict_batch  accepted hooks= (hook fired 1 time(s))
predict_long   TypeError: Agent.predict_long() got an unexpected keyword argument 'hooks'   # before
predict_long   accepted hooks= (hook fired 1 time(s))                                       # after

That is #489's shape: the core has the control, one entry point does not forward it. The hooks
are the documented place for the request-scoped work a long-document caller most wants —
PII redaction before inference, a cache read via ctx.skip(...), audit logging — and an Agent
that builds them per call cannot reach the scan at all. Instance-level hooks (Agent(hooks=[…]))
already fire inside predict_long, so this is not new machinery; it is the same dispatch point,
made reachable per call, plus the contract written down where it was missing.

Second thing the probe turned up. Because the split into windows happens after
on_predict_start, a start hook that answers the document returns one result where the scan
expects one per window, and predict_long aggregated that payload anyway:

agent = Agent("convaiinnovations/laya", device="cpu", hooks=[CachingHook()])   # ctx.skip -> 1 result
result = agent.predict_long(state, questions)
before: answers.department.window = {'index': 0, 'token_start': 0, 'token_end': 312, 'count': 3}
        usage = {'input_tokens': 1, 'output_tokens': 0, 'windows': 3}
after:  no 'window' key at all
        usage = {'input_tokens': 1, 'output_tokens': 0, 'windows': 0}

input_tokens: 1 proves inference never ran, while window named a deciding window and
windows: 3 counted windows — so the response asserted where in the document the model looked
for a decision it never made. A skipped payload of exactly one result is now passed through
unattributed with a RuntimeWarning, and a count matching neither the document nor the windows
raises instead of picking between answers that were never scored. Both halves are one logical
change — hooks on the scan path, and what a hook's answer is allowed to claim — so they are one
commit.

Measured, on the real checkpoint

laya (English), CPU, M-series, upstream main @ 4066d5d. A 3,441-character state (two
questions, one scanned path, max_len 512 / head_max_len 192 / state budget 312 tokens → the
split produced 3 windows):

before after
predict_long(state, q, hooks=[h]) TypeError accepted, hook fires once
what the hook receives (instance hooks only) 1 dispatch, len(ctx.states) == 3, decoded window texts, lengths 1928 / 1922 / 1513 same, and now per-call too
ctx.skip answer's window {'index': 0, 'token_start': 0, 'token_end': 312, 'count': 3} absent
usage["windows"] on that answer 3 0
decision, no hook vs per-call hook {'cancel': True, 'department': 'billing'} {'cancel': True, 'department': 'billing'} — identical
median predict_long 1238.2 ms (7 runs, no hooks) 1185.2 ms (7 runs, one no-op per-call hook)

Decisions compared at the level a caller acts on (choice label, bool(noul >= 0.5)), not raw
floats. The two medians are the same call; the ordering between them is noise, and forwarding
five keyword arguments onto an existing call site is not a hot path.

Tests

  • tests/test_predict_long.py: 13 → 32 checks, weight-free on the file's existing stub
    (Agent.__new__ + a token-count-equals-character-count tokenizer), covering: the five controls
    reach the scan and the single-window branch; window texts (not the caller's state) are what the
    hook layer gets; forwarding is decision-neutral; a skipped document answer comes back with no
    window and windows: 0, and warns; a mismatching count raises.

  • tests/test_hooks_api.py: 131 → 140 checks. Agent.predict_long joins the predict-surface
    loop that pins the five hook parameters, and window/stride plus the absence of
    max_len/head_max_len are pinned (the scan sizes windows from the checkpoint budget, so a
    per-call token override would mean something different there).

  • Six mutations of laya/agent.py, each named by the tests it fails — nothing here rests on a
    passing count:

    mutation named failures
    fast-path call drops the controls hooks/forwarded to system_one, hooks/the hook list reaches it there, hooks/hooks_timeout reaches it there
    scan call drops the controls hooks/forwarded to the scan, hooks/the hook list reaches it, hooks/hooks_raise reaches it, hooks/hooks_timeout reaches it
    alignment guard disabled skip/no window was scored, skip/no deciding window is claimed, skip/the caller is told, skip/rejects a count matching nothing
    windows reports the split instead of 0 skip/no window was scored
    mismatching count passes silently skip/rejects a count matching nothing
    warning removed skip/the caller is told

Gates

ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   All checks passed!
python -m compileall -q laya/ tests/                                  clean
python tests/test_predict_long.py   32 passed, 0 failed
python tests/test_hooks_api.py      140 passed, 0 failed
python tests/test_doc_tables.py     109 passed, 0 failed
python tests/test_hooks.py / test_router.py / test_batch.py           all passed
zensical build --strict --clean                                       No issues found

Docs

docs/hooks/api.md gains the agent.predict_long(...) line, which the Agent block did not have,
and states what a start hook sees on the scanned path. The README's predict_long section gains
the matching bullet and a hooks= example line.

Checked, and left alone

Nearest issue by title: #549 reports the served /v1/systemone path answering a long state from
its first 312 tokens. That lives in laya/serve.py and is what #550/#566 work on, not this
function — so no issue is closed here.

Added after @AlKor13's review: usage["windows"] is now total

The review found that adding windows = 0 for the skipped case made the key tri-state, because the
single-window path never had it at all -- so usage["windows"] == 0, the natural check for "a hook
answered this", raised KeyError on the shortest and most common inputs. Option (1) from that
review is implemented here: the short-circuit reports 1.

path before after
short state (fits one window) key absent; caller's gate raises KeyError('windows') windows=1; same gate returns False
scanned document windows=8 unchanged
hook answered before the scan windows=0 unchanged

Measured on the laya checkpoint, cpu, same states and questions both sides; the answers are
identical (0.5810 / 0.6496 / 0.4200), so the change adds a key on one path and moves no decision.
The write goes through a copy, because a start hook that answers returns the caller's own payload
dict and that object may be cached -- the probe shows same dict object: True -> False.

README's "identical output" bullet and docs/hooks/api.md teach all three counts, and
tests/test_predict_long.py pins them (32 -> 42 weight-free checks). A 9-variant mutation run
(count not written, count 0, write in place, scanned count hardcoded, hook count counted as a
scan, system_one usage dropped, README value changed, README totality sentence removed, docstring
reverted) fails on every variant with the failing check named.

…ped answer

predict_long was the only inference entry point that accepted none of the
per-call hook arguments, so a hook installed per call had to be promoted to
agent state just to instrument a long-document scan.

The hooks wrap the inference that answers the state: one call on the
fits-in-one-window path, one shared predict_batch over the windows
otherwise. A start hook that calls ctx.skip(...) is answering the document,
not a window, so its payload is returned unattributed with
usage["windows"] = 0 instead of being aggregated into a deciding-window
entry for windows that never scored it.
@AlKor13

AlKor13 commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

Thanks for picking this up — predict_long taking lang and nothing else was an oversight in #363, and the ctx.skip case is handled more carefully here than I would have guessed: a ValueError for any count mismatch other than one, rather than aggregating over a payload that no window produced, is the right call.

One consequence of adding usage["windows"] = 0 that is worth deciding deliberately, because it makes the key tri-state. Measured on main today, laya checkpoint:

short state (fits one window)  usage {"input_tokens": 40, "output_tokens": 0}          <- no "windows" key at all
long state  (scanned)          usage {"input_tokens": 5087, "output_tokens": 0, "windows": 15}

The short-state path short-circuits to system_one, which has no reason to report a window count, so the key is simply absent — that is mine from #363 and nobody noticed because there were only two cases. With this PR there are three:

case usage["windows"]
state fits one window absent
a hook answered before the scan 0
scanned N

So a caller who writes the natural check for "was this actually scanned" —

if result["usage"]["windows"] == 0:      # KeyError on every short state

gets a KeyError on exactly the inputs that are most common in a mixed workload, and one who writes usage.get("windows", 0) == 0 cannot tell "a hook answered it" from "it fit in one window", which are different things: the first means no model saw it, the second means the model saw all of it.

Two ways out, and I do not have a strong preference:

  1. Report 1 on the short-circuit path — it is literally one window, the key becomes total, and 0 keeps its new meaning of "no window scored this". One line in predict_long, and it makes windows monotone with how much of the document was read.
  2. Leave it absent and say so in the same docstring paragraph this PR adds, so the three cases are at least written down.

(1) changes an output shape that has shipped, so it is your call rather than mine; if you want it, it belongs in this PR rather than a follow-up, since this is the change that makes the distinction observable.

Nothing else from my side: forwarding the hooks into the short-circuit system_one as well as the windowed predict_batch is what keeps the two paths honest, and documenting that ctx.states holds decoded windows rather than the caller's state is the part I would have got wrong.

A state that fits one window returned no windows key at all, so the natural
gate (usage["windows"] == 0) raised KeyError on the shortest, most common
inputs and could not tell "a hook answered it" from "it was scanned".
predict_long now reports 1 there, keeping 0 for a hook that answered a
multi-window document and N for a scan, and the copy-then-write means a
start hook's own payload dict is never mutated.
@aashish254

aashish254 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Short answer: took (1). usage["windows"] is now 1 on the single-window path, so the key is total over all three cases and 0 means exactly one thing: a hook answered a multi-window document before the model read any of it. Option (2) left the KeyError on the shortest, most common inputs.

Measured on the laya checkpoint, cpu, same states and questions before and after:

  • short state (fits one window): before, keys=input_tokens,output_tokens and the gate usage["windows"] > 1 raises KeyError('windows'); after, windows=1 and the same gate returns False
  • scanned document (windows=8, gate True) and hook answered before the scan (windows=0, gate False): unchanged

The answers didn't move — both runs choose billing at 0.5810 / 0.6496 / 0.4200 — the only claim available for a usage field: one path gains a key, the two that already carried it are byte-identical.

One consequence the short-circuit makes necessary: a start hook that answers there returns the caller's own payload dict, so predict_long now copies before it writes. In the probe the result used to be the caller's object (same dict object: True, now False) and its usage stays {'input_tokens': 9} — writing the count would otherwise mutate a dict the caller may have cached.

The README's "identical output" sentence and the docs/hooks/api.md bullet now teach all three counts, and tests/test_predict_long.py pins them (32 -> 42 checks, still weight-free). A 9-variant mutation run fails on every variant, with the failing check named rather than crashing.

@Bruce-Yii

Copy link
Copy Markdown
Contributor

The count check compares len(results) (after the start hook ran) against len(windows) (before it), so a count of 1 isn't by itself evidence that a hook answered the document.

Driving the real predict_batch with a start hook that only narrows the scan:

  • Narrowing ctx.states to one window returns usage["windows"] = 0 plus the RuntimeWarning "a hook answered the state before it was scanned", and the answer comes back with no window attribution — but a forward pass did run and booked input_tokens. The output is indistinguishable from the ctx.skip() case apart from the tokens spent.
  • Narrowing to two windows raises ValueError: 2 results returned; a start hook that answers a long state returns one result for the document, after two forward passes have already run.

That reads against the docstring at laya/agent.py:989 ("A start hook that rewrites ctx.states therefore rewrites windows") and docs/hooks/api.md:47 ("A start hook may replace the list"). Comparing against the post-hook state count, or checking whether the hook set ctx.results, would keep the two cases apart.

@AlKor13

AlKor13 commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

@Bruce-Yii is right, and the root cause is worth naming precisely: ctx.skip() does nothing but assign ctx.results (laya/hooks.py:48-50) — there is no flag — so from outside predict_batch there is no signal that says "a hook answered", and the count comparison is a proxy for it that misfires in both directions. Narrowing to one window looks like an answer; narrowing to two is rejected outright, after both forwards have already run. usage["input_tokens"] > 0 alongside windows: 0 is the self-contradiction that gives it away.

The real signal is reachable without changing the hook API, using the same trick #497 uses on the router path: append an internal start hook last and let it observe what predict_batch is about to act on.

  • compose_hooks returns defaults + installed + per-call (hooks.py:220), so an observer placed after the caller's per-call hooks is last in the list.
  • dispatch iterates that list in order (hooks.py:470) — sequentially, whatever hooks_concurrent is set to, since the flag chooses a lock rather than parallelism.
  • predict_batch then reads exactly the two things the observer just saw: states = ctx.states and if ctx.results is None (agent.py:867-869).

So the observer records answered = ctx.results is not None and scanned = len(ctx.states), and the three cases become total without guessing:

what the hook did windows answer["window"]
answered with ctx.skip(...) 0 absent — no window scored it
rewrote the scan to n windows n absent, and this is the case the count check gets wrong
left the scan alone n present

The middle row is the one I would not silently aggregate with attribution even once the count check is fixed: predict_long computed token_start/token_end for its windows, and a hook that replaced the list has invalidated those offsets, so reporting them would name spans that no longer describe the text that was scored. Aggregating over whatever came back is right; claiming which span decided it is not. That also removes the ValueError entirely — two windows back from a rewritten scan is a legitimate outcome, not a contract violation.

Happy to send this as a patch against your branch rather than a follow-up PR, if you would rather not rework it yourself — it is the observer hook, the two fields it records, and dropping the attribution on the rewritten path. Your call; it is your PR and you have done the harder part.

…t count

A start hook that answered a long state and a start hook that replaced the window list
both return fewer results than there were windows, and the count could not tell them
apart: the rewrite was reported to the caller as "a hook answered before the state was
scanned", and the tokens it did book were dropped.

predict_long now appends its own recorder after the caller's start hooks and reads
ctx.results from the engine's own context, so only a real answer takes the unattributed
path and a rewritten list raises with the split it could not attribute. The same rule
covers the fits-in-one-window path, where a hook answer now reports 0 windows like every
other hook answer, and the window list is copied before the hooks see it, so an in-place
append can no longer move the split the answers are attributed to.
The check that read a count of one as a hook answer still used the count as evidence. A
start hook may replace `ctx.states` (`docs/hooks/api.md`), so narrowing the scan to one
window is a supported outcome and not an answer: a forward ran and booked tokens, and
raising for two windows discarded work the caller had already paid for.

`predict_long` now asks the observer appended after the caller's own start hooks what
reached inference, which makes the three cases total: a `ctx.skip()` payload comes back
with no attribution and 0 windows; an unchanged scan aggregates with the deciding window
named; a rewritten scan aggregates over the states that were scored, with
`usage["windows"]` as their count and no `answer["window"]` -- the token offsets describe
the windows this method built, not the text the model read. A count the observer cannot
explain (no hook chain ran) stays an error.
… count it reports

The aliasing bug is visible only as a type: without the copy of the split, `ctx.states.append()`
grows both lists, the counts agree, and the attribution indexes a shorter `starts`. So the check
asserts the call returns at all before it asserts what it returned.
@aashish254

aashish254 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Short answer: adopted the observer, dropped the raise. 9ae2318 (plus c12f701, one more check) is on the branch; the result count is no longer read as evidence anywhere in predict_long.

The start hook records both things the reviewers named: answered = ctx.results is not None, and a snapshot of ctx.states. The three outcomes your table showed:

  • answered with ctx.skip([result]): usage["windows"] is 0, no answer["window"]
  • scan left as built: usage["windows"] is N, answer["window"] present
  • scan replaced in any way: states are scored, answer["window"] absent

The last two differ only by a list comparison against the split this method computed, so a text-only rewrite would attribute answers to windows that were not scored. docs/hooks/api.md carries the table now, and the README bullet says attribution survives only an unchanged scan.

Driving the real predict_batch with only _encode_state, _forward and _decode_answers stubbed:

  • narrowing ctx.states to one window returns windows: 1 and no RuntimeWarning. On 9a52a22 the pinning checks fail as reported: narrow/the count is the state scored, not the split made: got 0 want 1, and the matching RuntimeWarning check.
  • narrowing to two returns windows: 2 and nothing raises after the forwards have run.

Two deliberate differences, easy to reverse:

  1. The ValueError stays for one case: a count the observer cannot explain. If no hook chain ran at all — predict_batch replaced, which is what the other harness in tests/test_predict_long.py does — a wrong count is a bug, not a hook outcome. The message now says no start hook chain ran, so nothing rewrote the scan.
  2. predict_batch receives list(windows), a copy. Without it an in-place ctx.states.append(...) grows the split too, and on 9a52a22 you get grow/scoring one invented state is not an error: got 'IndexError' want None. With the copy the invented state is scored, counted as N+1, and attributed to nothing.

One judgement call worth your eye: a hook that empties ctx.states now returns {"answers": {}, "usage": {"input_tokens": 0, ..., "windows": 0}} with a RuntimeWarning, where every earlier revision raised ValueError. 0 is the honest count — but that's the one place I'd take the opposite instruction.

Verification, all stubbed-forward, so it's about branching and shape, not decision quality:

  • python tests/test_predict_long.py: 79 checks, 0 failed (42 on 9a52a22; the new checks fail 14 there, and 15 against the intermediate bd18c47, which kept the raise).
  • 8 single-line mutants of laya/agent.py, each killed by a distinct named check.
  • Adjacent lanes, all 0 failed: test_hooks.py 162, test_hooks_api.py 140, test_router.py 546, test_doc_tables.py 109, test_structured.py 58, test_runtime_fixes.py 33, test_cli.py 38, test_portability.py 26, test_predict_batch.py OK. ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 clean.

The observer and the two fields are the reviewers' design. No patch needed — but do say if you'd rather the empty scan raise.

@Bruce-Yii

Bruce-Yii commented Sep 27, 2026 via email

Copy link
Copy Markdown
Contributor

@aashish254

Copy link
Copy Markdown
Contributor Author

Settled, and that is what the head already does — nothing to change.

9ae2318 returns {"answers": {}, "usage": {..., "windows": 0}} behind a RuntimeWarning once the
observer has seen the chain run and leave no states, and keeps the ValueError only for the case the
observer cannot explain. Four checks pin that split: empty/nothing scored returns instead of raising,
empty/no answer is invented, empty/the count says so, empty/the caller is told for the hook
outcome, and count/rejects a result count matching nothing for the unexplained one.

So the judgement call is now a reviewer decision rather than mine, and the branch matches it. Thanks
for the read.

@AlKor13

AlKor13 commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Checked out c12f701 and drove it — your suite is 79/79, the observer does what the three rows say, and the single-window skip reporting 0 rather than 1 is a nicer call than anything in my table. Two of the three rows I could not fault.

The remaining one is the ValueError, and it is not about counting any more: it is that predict_long gives ctx.skip() a different contract from the call it is implemented on, and a hook cannot tell which contract it is under.

ctx.skip() is documented as "set cached results" (hooks.py:48-50) for a call whose results are one per state. A cache hook therefore answers with one result per state in ctx.states — correct against predict_batch, and correct against predict — and the same hook object raises the moment the state is long enough to be scanned. Driving your make_real_agent harness:

def answer_per_window(ctx):                      # what a cache hook naturally does
    ctx.skip([cached for _ in ctx.states])

agent.predict_long({"body": "x" * 400}, Q, on_predict_start=answer_per_window)
# ValueError: predict_long: a start hook answered this state with 11 results;
#             ctx.skip() takes one result for the document, not one per window

def answer_one_document(ctx):
    ctx.skip([cached])
# -> windows=0, the document answer, RuntimeWarning as designed

So a hook installed on the Agent — telemetry-plus-cache is the obvious one, and #590 is about exactly that kind of hook — works on predict, works on predict_batch, and raises on predict_long for a long input only. That is a sharp edge for the caller who least expects it: the failure appears when a document grows, not when the hook is written.

The observer already carries what settles it, since evidence["states"] is the list the hook actually saw:

len(results) meaning windows answer["window"]
len(evidence["states"]) per-state results, the documented skip() shape that many present only if the scan is unchanged
1, when the split made more than one window one answer for the document 0 absent
anything else genuinely inconsistent ValueError, as now

Treating the first row like scored results is what makes a cache useful: the served answer and the cached answer come back in the same shape, which is the whole point of caching the call. And the ambiguity at len(windows) == 1 disappears on its own, because that case never reaches here — the short-circuit above handles it, and it already distinguishes the two by evidence["answered"].

Happy to send this as a patch to your branch; it is a few lines in the same block plus two cases in test_predict_long.py. Your PR, your call — and either way the count-as-evidence problem is gone, which was the thing worth fixing.

@Bruce-Yii

Bruce-Yii commented Sep 27, 2026 via email

Copy link
Copy Markdown
Contributor

@NandhaKishorM
NandhaKishorM merged commit d6f1f3e into NandhaKishorM:main Sep 27, 2026
22 checks passed
NandhaKishorM added a commit that referenced this pull request Sep 27, 2026
…he batch merges

- ONNXAgent.predict_long takes the per-call hooks Agent.predict_long gained in #577, with
  the same usage["windows"] contract (1 for one window, 0 for a hook's answer, no window
  attribution on a rewritten scan). tests/test_onnx_long.py's parity check caught the gap.
- decide_batch takes min_confidence like decide (#361), and ONNXAgent gets decide_batch now
  that it has predict_batch (#489). The docs no longer say ONNXAgent cannot batch.
- test_structured_docs follows decide's signature, and the API table names callables that
  exist (agent./router.decide_batch).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254, merged. I ported the same hook contract to ONNXAgent.predict_long so the two backends keep one signature. It ships in 0.3.21.

NandhaKishorM pushed a commit that referenced this pull request Oct 1, 2026
Router.route_batch takes hooks_timeout (router.py:911, docstring 925-926)
and applies it per-request during on_route dispatch. The batch sibling
laya_predict_batch has always had the override plumbed; laya_route_batch
did not. The tool layer accepted only `requests`, so an operator running a
routing sweep over a workload cannot protect against a slow or hung
operator hook the way they can on the predict path.

- New optional `hooks_timeout` arg on laya_route_batch; forwarded to
  Router.route_batch only when the caller actually asked for one, so a
  pre-#577 attached Router keeps working.
- _validate_hooks_timeout wraps core's validate_timeout and turns its
  ValueError into a ToolError code (`invalid_hooks_timeout`) at the MCP
  boundary. Bools rejected here (not by core, which coerces True -> 1.0).
- Server wrapper exposes hooks_timeout; the tool description now mentions
  it so clients discover the knob.
- Tests: BatchRouter.route_batch records the timeout; six new checks cover
  forwarding (float + coerced int), default-stays-unset, validation errors,
  and a strict-signature Router stub as the witness against always-kwarg.
  Four mutations each produce a named FAIL.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants