Repository navigation
docs(serve): document the abstention gate's report on every answer, and hold the answer table to the code - #824
Merged
NandhaKishorM merged 2 commits intoOct 2, 2026
Conversation
apply_confidence_gate writes abstention and abstention_threshold onto every answer of a gated request and flag_low_confidence writes low_confidence onto the ones below the threshold (NandhaKishorM#361). The page named the request control and said an answer comes back marked, but never said what the mark is called, what states it takes, or what threshold produced them -- so the only way to learn the report existed was to read the gate's docstring.
…nd the gate build
Owner
|
Merged for 0.3.24. Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, Thank you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two commits, no behaviour change:
docs(http): document the abstention gate's report on each answer— the### Responseanswer-keytable in
docs/http-api.mdgains a| gate |row namingabstention,abstention_thresholdandlow_confidence, plus the prose that says when those three appear and what each state means: arequest that sets
min_confidencegets a state on every answer, one that does not gets none ofthe three keys,
low_confidenceistrueon exactly theabstainedanswers,0.0is a thresholdthat was set (so states are reported and all read
passed), and the answer is kept in every state —the gate marks, it does not drop.
test(serve): hold the documented answer keys to the keys the agents and the gate build—tests/test_serve.pygainstest_http_api_page_documents_the_gate_report_on_an_answerand threehelpers:
_answer_literal_keys()reads theanswers[qid] = {...}dict literals out of an agent'sown source,
_gate_written_keys()reads thea[...] = ...assignments out oflaya/confidence.py,_md_answer_table()parses the| answer type | keys |table. Nothing istranscribed by hand, so the test is not a third copy of the contract.
Why
The page already sells the gate:
min_confidenceis in the request-control table, so a caller can payfor it. It did not document what the caller then receives.
apply_confidence_gate(#361) writes threekeys onto the very answers the table describes, and the table named none of them — so there was no
documented name for the state, no documented name for the threshold echo, and no documented way to tell
"the gate ran and this answer cleared it" from "this run had no gate at all". That last distinction is
the gate's whole contract (
laya/confidence.py:92-96: absence, not a fourth value), and it was exactlythe thing the page left a reader to guess.
This is the answer-level counterpart of #816/#819, which hold the decision response key set and the
usage shapes on the same page to the code that builds them. The answer table was the one table on
### Responseleft out — and it had already absorbed a new always-on key (answer_confidence, #126)with no test noticing, which is the drift a checked table cannot have.
What the two writers actually produce, printed by the helpers themselves rather than transcribed:
Three deliberate choices:
asserts
documented == built | gateper question type, so a key the code stops writing fails too. Andthe sample response is checked against the request the page itself prints — that request sends no
threshold, so its answers must carry the always-on keys and no gate report. A sample that showed an
abstentionon an ungated call would teach the wrong contract to everyone who copies it.all.alldescribes what every answer carries;the gate's three keys are conditional on the request. The test keeps them apart, which is what makes
arm 4 below fail instead of pass.
torch_site == onnx_siteis asserted, not assumed. One row set on the page describes bothruntimes, so the page can only be correct if the two agents agree key-for-key. That is the same parity
claim docs(http-api): report every key a decision response actually carries #816 makes at the response level, checked here at the answer level.
How it was verified
The gate's teeth, by breaking each claim it makes — 12 arms, each a single-line edit, tree committed
first (
git status --porcelainempty) because the harness restores withgit checkout, run with-Bso a same-length mutant cannot be masked by a live
.pyc:0 survivors,
FAIL: 0, tree clean afterwards (git status --porcelainempty). The first two arms provethe old page was under-documented and that a made-up key would not have slipped in; arms 9-10 prove the
parity claim is real rather than decorative.
Checklist
the test that holds it; no SDK, no schema, no behaviour.
main(git merge-base --is-ancestor upstream/main HEADpasses on4aa6761,which is still
upstream/mainat push time).196 passed, ruff andcompileallclean, mutation 12/12 red).what keeps it from drifting again.
Note for review:
tests/test_serve.pyis also touched by #716 and #667. There is no overlap — theirhunks land at lines 375 and 845-867 of the current file, and everything here appends at EOF (lines
2558-2686). The
docs (strict build)job is the inherited red everymain-based PR shows until #794 lands; the docs change here adds a table row and prose to an existingpage, so it adds no nav or link surface.
Fixes #