Skip to content

docs(serve): document the abstention gate's report on every answer, and hold the answer table to the code - #824

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:docs-http-abstention-gate
Oct 2, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:docs-http-abstention-gate

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

Two commits, no behaviour change:

  1. docs(http): document the abstention gate's report on each answer — the ### Response answer-key
    table in docs/http-api.md gains a | gate | row naming abstention, abstention_threshold and
    low_confidence, plus the prose that says when those three appear and what each state means: a
    request that sets min_confidence gets a state on every answer, one that does not gets none of
    the three keys, low_confidence is true on exactly the abstained answers, 0.0 is a threshold
    that was set (so states are reported and all read passed), and the answer is kept in every state —
    the gate marks, it does not drop.
  2. test(serve): hold the documented answer keys to the keys the agents and the gate build —
    tests/test_serve.py gains test_http_api_page_documents_the_gate_report_on_an_answer and three
    helpers: _answer_literal_keys() reads the answers[qid] = {...} dict literals out of an agent's
    own source, _gate_written_keys() reads the a[...] = ... assignments out of
    laya/confidence.py, _md_answer_table() parses the | answer type | keys | table. Nothing is
    transcribed by hand, so the test is not a third copy of the contract.

Why

The page already sells the gate: min_confidence is in the request-control table, so a caller can pay
for it. It did not document what the caller then receives. apply_confidence_gate (#361) writes three
keys onto the very answers the table describes, and the table named none of them — so there was no
documented name for the state, no documented name for the threshold echo, and no documented way to tell
"the gate ran and this answer cleared it" from "this run had no gate at all". That last distinction is
the gate's whole contract (laya/confidence.py:92-96: absence, not a fourth value), and it was exactly
the thing the page left a reader to guess.

This is the answer-level counterpart of #816/#819, which hold the decision response key set and the
usage shapes on the same page to the code that builds them. The answer table was the one table on
### Response left out — and it had already absorbed a new always-on key (answer_confidence, #126)
with no test noticing, which is the drift a checked table cannot have.

What the two writers actually produce, printed by the helpers themselves rather than transcribed:

laya/agent.py      choice ['action', 'answer_confidence', 'choice', 'confidence', 'probabilities', 'type']
                   score  ['action', 'answer_confidence', 'confidence', 'legend', 'probabilities', 'score', 'type']
                   noul   ['action', 'answer_confidence', 'confidence', 'noul', 'type']
laya/onnx_agent.py  identical to laya/agent.py for all three types
confidence.py      gate writes ['abstention', 'abstention_threshold', 'low_confidence']

docs/http-api.md   row all    ['action', 'answer_confidence', 'confidence']
after this PR      row choice ['choice', 'probabilities']
                   row score  ['legend', 'probabilities', 'score']
                   row noul   ['noul']
                   row gate   ['abstention', 'abstention_threshold', 'low_confidence']

Three deliberate choices:

  • Both directions, and against the printed sample. Not just "every documented key exists": the test
    asserts documented == built | gate per question type, so a key the code stops writing fails too. And
    the sample response is checked against the request the page itself prints — that request sends no
    threshold, so its answers must carry the always-on keys and no gate report. A sample that showed an
    abstention on an ungated call would teach the wrong contract to everyone who copies it.
  • The gate is a separate row, not extra keys in all. all describes what every answer carries;
    the gate's three keys are conditional on the request. The test keeps them apart, which is what makes
    arm 4 below fail instead of pass.
  • torch_site == onnx_site is asserted, not assumed. One row set on the page describes both
    runtimes, so the page can only be correct if the two agents agree key-for-key. That is the same parity
    claim docs(http-api): report every key a decision response actually carries #816 makes at the response level, checked here at the answer level.

How it was verified

$ python -m pytest tests/test_serve.py -q
196 passed in 4.16s                     # was 195 on upstream/main; +1 test, nothing removed
$ ruff check tests/test_serve.py laya/agent.py laya/onnx_agent.py laya/confidence.py \
      --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q tests laya && echo ok
ok
$ python -m pytest tests/test_serve.py -k gate_report -q
1 passed, 195 deselected

The gate's teeth, by breaking each claim it makes — 12 arms, each a single-line edit, tree committed
first (git status --porcelain empty) because the harness restores with git checkout, run with -B
so a same-length mutant cannot be masked by a live .pyc:

$ python /tmp/mut823/run.py
RED  the gate report is undocumented again                 the answer table rows are ['all','choice','noul','score']; this gate reads one row per question type, ...
RED  a documented answer key has no code behind it         the choice rows say an answer carries [...,'ghost_field',...]
RED  a key the agents build drops out of the shared row     the choice rows say an answer carries [...], the agents build [...]
RED  a request control is counted as an answer key          the gate report says [...,'min_confidence'], `apply_confidence_gate` and `flag_low_confidence` write [...]
RED  the gate report renames one of its keys                the gate report says ['abstain_state',...], ... write ['abstention',...]
RED  the type row stops naming the key the sample prints     the noul rows say an answer carries [...]
RED  the score row stops documenting its distribution       the score rows say an answer carries [...]
RED  the gate grows a fourth key nobody documented          the gate report says [...], ... write ['abstention',...,'gate_reason']
RED  one agent drifts from the other on a key               the two agents build different answers, so the page cannot describe both: torch {...}
RED  the two agents answer with different keys              the two agents build different answers, ...
RED  the printed ungated sample carries a gate report       queue is ungated in the printed request but carries the gate key ['abstention']
RED  the printed request sets a threshold, so the sample must show the gate  the printed request now sets a threshold, so the sample below it has to show the gate report

12/12 arms red

0 survivors, FAIL: 0, tree clean afterwards (git status --porcelain empty). The first two arms prove
the old page was under-documented and that a made-up key would not have slipped in; arms 9-10 prove the
parity claim is real rather than decorative.

Checklist

  • Focused on one change (split unrelated work into another PR) — the gate row on the answer table and
    the test that holds it; no SDK, no schema, no behaviour.
  • Rebased on the latest main (git merge-base --is-ancestor upstream/main HEAD passes on 4aa6761,
    which is still upstream/main at push time).
  • Tests pass locally (196 passed, ruff and compileall clean, mutation 12/12 red).
  • Docs or examples updated when the public API changed — this PR is the docs change; the test is
    what keeps it from drifting again.

Note for review: tests/test_serve.py is also touched by #716 and #667. There is no overlap — their
hunks land at lines 375 and 845-867 of the current file, and everything here appends at EOF (lines
2558-2686). The docs (strict build) job is the inherited red every
main-based PR shows until #794 lands; the docs change here adds a table row and prose to an existing
page, so it adds no nav or link surface.

Fixes #

apply_confidence_gate writes abstention and abstention_threshold onto every answer of a gated request and flag_low_confidence writes low_confidence onto the ones below the threshold (NandhaKishorM#361). The page named the request control and said an answer comes back marked, but never said what the mark is called, what states it takes, or what threshold produced them -- so the only way to learn the report existed was to read the gate's docstring.
@NandhaKishorM
NandhaKishorM merged commit 3ff1278 into NandhaKishorM:main Oct 2, 2026
25 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.24.

Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, compileall clean, both TypeScript packages green (laya-ts 597 tests, the SDK 26), and a decision diff against 0.3.23 on the real english and multilingual checkpoints (74 answers over choice, score and noul; short and long states; budgeted calls and predict_long) showing zero decision flips and a max probability delta of 0.000000.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants