Repository navigation
feat(onnx): add ONNXAgent.predict_long, the long-document API the torch Agent has - #494
Merged
NandhaKishorM merged 5 commits intoSep 27, 2026
Merged
Conversation
…nfidence) (NandhaKishorM#361) - Add opt-in min_confidence parameter to Agent, Router, ONNXAgent, and decide - Read answer_confidence (calibrated max(p), invariant to option count k), falling back to confidence - Flag low_confidence: True on answers below threshold while keeping raw predictions and probabilities intact - Project low-confidence fields to None in decide() schema output - Isolate validation and flagging in pure-Python laya/confidence.py so Router and decide stay torch-free - Update LangChain router to read answer_confidence first before falling back to confidence - Verify default behavior is 100% unchanged when min_confidence is omitted or None - Add comprehensive test coverage across test_confidence.py, test_structured.py, test_router_batch.py, and test_langchain.py
…KishorM#361) Addresses the review on NandhaKishorM#456. - Router.predict: keeps the original narrow `unexpected keyword argument 'lang'` fallback and does not forward min_confidence to agent.system_one; the Router flags its own results. Rebased onto main, which had already folded NandhaKishorM#435's skip-defaults guard into the same block. - low_confidence is set before on_predict_end runs, on Agent.predict_batch, ONNXAgent.system_one, Router.predict and Router.predict_batch, so audit hooks see exactly what the caller gets. test_batch.py and test_router_batch.py assert the hook-time snapshot on both Router paths. - README: new "Opt-in abstention: min_confidence" section, incl. that LayaRouter's confidence_threshold reads answer_confidence; docs/langchain.md notes it beside the threshold example. - common.py: drop the two blank lines the feature diff carried.
…t has (NandhaKishorM#323) ONNXAgent exposed only system_one/predict, so an ONNX deployment had no batch path: a caller that treats backends interchangeably (Router.predict_batch calls agent.predict_batch unconditionally; the evals runner protocol probes hasattr(runner, "predict_batch")) raised AttributeError instead of returning results, and N states cost N session runs. - predict_batch(states, questions, batch_size=..., lang=..., min_confidence=..., hooks...) mirrors Agent.predict_batch: the same validation (bare str/dict -> TypeError), the same empty cases, the same hook lifecycle, the same per-state usage and result order, and ceil(N / batch_size) ONNX Runtime session runs. ONNX Runtime parallelises inside a run, so several states' rows belong in one call. - system_one now delegates to it, as Agent.system_one delegates to Agent.predict_batch, so the single and batched paths cannot drift apart. _infer stays as the unhooked single-state entry point the existing suites drive. - _infer's body is split into _encode_state (tokenize the state once, build its question rows) and _decode_answers (decode the len(ids) rows starting at an offset), which the batch path reuses instead of re-deriving them; _infer_batch validates once, then encodes, collates, runs and decodes per chunk. Tests: tests/test_onnx_batch.py, 49 checks and no onnxruntime or checkpoint (a stub session whose logits derive from each row's own tokens, so a row decoded for the wrong state fails a check). It covers batched == sequential, chunking, result order, usage parity, the hook lifecycle (rewrite, skip, on_error), min_confidence including that on_predict_end already sees the flag, lang override parity, and Router.predict_batch with an attached ONNX agent -- the reported failure. Wired into both CI lanes.
This was referenced Sep 25, 2026
This was referenced Sep 25, 2026
Merged
Owner
|
Thanks @aashish254. Merged as part of the #489/#507 stack. |
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #489 — this branch is cut from it, so the diff carries the
predict_batchcommit until that one merges; review the second commit only.The gap, found by diffing the two backends rather than any issue:
Agent.predict_longis the documented long-state recipe — window an over-budget state, score the windows throughpredict_batch, aggregate per question — andONNXAgenthad no equivalent. So an ONNX deployment gets the batch API (with #489) but still silently truncates long states to one window, exactly the failurepredict_longexists to avoid.The port mirrors the PyTorch implementation rule for rule: same signature (
window/stride/aggregate/batch_size/lang), the one-window fast path delegated tosystem_onebyte-identically, noul max-over-windows and choice/score most-confident-window aggregation, theanswer["window"]span fields, generic usage aggregation plususage["windows"], and the same "not a calibrated number for the whole document" warning in the docstring. It is defined on top ofONNXAgent.predict_batch(asAgent.predict_longis onAgent.predict_batch), so windows share ONNX Runtime session runs — one run for all of them, or one perbatch_sizechunk — and the single/multi-window paths cannot drift apart. No new knobs, no behavior change to anything existing.Tests (
tests/test_onnx_long.py, wired into both CI lanes next totest_onnx_batch.py): same weight-free technique as that suite —__new__the agent, fake tokenizer, stub session whose logits derive from each row's own tokens, so a window decoded for the wrong span changes the answer. 26 checks: contract/parity of parameter names againstAgent.predict_long, short-state delegation, the deciding window equals that window's ownpredict_batchanswer and is the argmax on each type's rule, span fields index the original state, shared-vs-chunked session runs produce identical results, explicitwindow/stride,aggregaterejection, and empty questions. The fixture asserts it is non-vacuous (window confidences genuinely differ), mirroring the batch suite.Gates:
ruff --select=E9,F63,F7,F82,F401,F811andcompileallclean;test_onnx_batch.py49 passed,test_onnx_lang_parity.py10 passed,test_hooks_api.pyall passed (predict_long carries no hook knobs on either backend, matchingAgent.predict_long);zensical build --strict --cleanno issues (the docstring renders into the API reference).README: one paragraph under "Long documents:
predict_long" stating the ONNX contract, in the style of thepredict_batchparity note above it.