Skip to content

feat(onnx): add ONNXAgent.predict_long, the long-document API the torch Agent has - #494

Merged
NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
aashish254:feat-onnx-predict-long
Sep 27, 2026
Merged

NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
aashish254:feat-onnx-predict-long

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

Stacked on #489 — this branch is cut from it, so the diff carries the predict_batch commit until that one merges; review the second commit only.

The gap, found by diffing the two backends rather than any issue: Agent.predict_long is the documented long-state recipe — window an over-budget state, score the windows through predict_batch, aggregate per question — and ONNXAgent had no equivalent. So an ONNX deployment gets the batch API (with #489) but still silently truncates long states to one window, exactly the failure predict_long exists to avoid.

The port mirrors the PyTorch implementation rule for rule: same signature (window/stride/aggregate/batch_size/lang), the one-window fast path delegated to system_one byte-identically, noul max-over-windows and choice/score most-confident-window aggregation, the answer["window"] span fields, generic usage aggregation plus usage["windows"], and the same "not a calibrated number for the whole document" warning in the docstring. It is defined on top of ONNXAgent.predict_batch (as Agent.predict_long is on Agent.predict_batch), so windows share ONNX Runtime session runs — one run for all of them, or one per batch_size chunk — and the single/multi-window paths cannot drift apart. No new knobs, no behavior change to anything existing.

Tests (tests/test_onnx_long.py, wired into both CI lanes next to test_onnx_batch.py): same weight-free technique as that suite — __new__ the agent, fake tokenizer, stub session whose logits derive from each row's own tokens, so a window decoded for the wrong span changes the answer. 26 checks: contract/parity of parameter names against Agent.predict_long, short-state delegation, the deciding window equals that window's own predict_batch answer and is the argmax on each type's rule, span fields index the original state, shared-vs-chunked session runs produce identical results, explicit window/stride, aggregate rejection, and empty questions. The fixture asserts it is non-vacuous (window confidences genuinely differ), mirroring the batch suite.

Gates: ruff --select=E9,F63,F7,F82,F401,F811 and compileall clean; test_onnx_batch.py 49 passed, test_onnx_lang_parity.py 10 passed, test_hooks_api.py all passed (predict_long carries no hook knobs on either backend, matching Agent.predict_long); zensical build --strict --clean no issues (the docstring renders into the API reference).

README: one paragraph under "Long documents: predict_long" stating the ONNX contract, in the style of the predict_batch parity note above it.

…nfidence) (NandhaKishorM#361)

- Add opt-in min_confidence parameter to Agent, Router, ONNXAgent, and decide
- Read answer_confidence (calibrated max(p), invariant to option count k), falling back to confidence
- Flag low_confidence: True on answers below threshold while keeping raw predictions and probabilities intact
- Project low-confidence fields to None in decide() schema output
- Isolate validation and flagging in pure-Python laya/confidence.py so Router and decide stay torch-free
- Update LangChain router to read answer_confidence first before falling back to confidence
- Verify default behavior is 100% unchanged when min_confidence is omitted or None
- Add comprehensive test coverage across test_confidence.py, test_structured.py, test_router_batch.py, and test_langchain.py
…KishorM#361)

Addresses the review on NandhaKishorM#456.

- Router.predict: keeps the original narrow `unexpected keyword argument 'lang'`
  fallback and does not forward min_confidence to agent.system_one; the Router
  flags its own results. Rebased onto main, which had already folded NandhaKishorM#435's
  skip-defaults guard into the same block.
- low_confidence is set before on_predict_end runs, on Agent.predict_batch,
  ONNXAgent.system_one, Router.predict and Router.predict_batch, so audit hooks
  see exactly what the caller gets. test_batch.py and test_router_batch.py
  assert the hook-time snapshot on both Router paths.
- README: new "Opt-in abstention: min_confidence" section, incl. that
  LayaRouter's confidence_threshold reads answer_confidence; docs/langchain.md
  notes it beside the threshold example.
- common.py: drop the two blank lines the feature diff carried.
…t has (NandhaKishorM#323)

ONNXAgent exposed only system_one/predict, so an ONNX deployment had no batch path: a caller
that treats backends interchangeably (Router.predict_batch calls agent.predict_batch
unconditionally; the evals runner protocol probes hasattr(runner, "predict_batch")) raised
AttributeError instead of returning results, and N states cost N session runs.

- predict_batch(states, questions, batch_size=..., lang=..., min_confidence=..., hooks...)
  mirrors Agent.predict_batch: the same validation (bare str/dict -> TypeError), the same empty
  cases, the same hook lifecycle, the same per-state usage and result order, and
  ceil(N / batch_size) ONNX Runtime session runs. ONNX Runtime parallelises inside a run, so
  several states' rows belong in one call.
- system_one now delegates to it, as Agent.system_one delegates to Agent.predict_batch, so the
  single and batched paths cannot drift apart. _infer stays as the unhooked single-state entry
  point the existing suites drive.
- _infer's body is split into _encode_state (tokenize the state once, build its question rows)
  and _decode_answers (decode the len(ids) rows starting at an offset), which the batch path
  reuses instead of re-deriving them; _infer_batch validates once, then encodes, collates, runs
  and decodes per chunk.

Tests: tests/test_onnx_batch.py, 49 checks and no onnxruntime or checkpoint (a stub session whose
logits derive from each row's own tokens, so a row decoded for the wrong state fails a check).
It covers batched == sequential, chunking, result order, usage parity, the hook lifecycle
(rewrite, skip, on_error), min_confidence including that on_predict_end already sees the flag,
lang override parity, and Router.predict_batch with an attached ONNX agent -- the reported
failure. Wired into both CI lanes.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254. Merged as part of the #489/#507 stack. ONNXAgent.predict_long now also takes the per-call hooks Agent.predict_long got in #577, with the same usage["windows"] rules. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants