Skip to content

docs(structured): gate every documented usage shape against the code - #819

Merged
NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
aashish254:fix-usage-shape-claims
Oct 1, 2026
Merged

NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
aashish254:fix-usage-shape-claims

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

Two pages tell a reader what an answered usage block contains. Both were wrong, and nothing could
see it. This fixes the shapes and adds the gate, in the suite that already owns this page's claims:

  • docs/structured.md — result.usage shown as the six keys an answer builds, plus a paragraph
    saying what each is and that options is the conditional seventh.
  • examples/03_reading_the_result.py — the same shape in the banner beginners are sent to, and the
    prose extended past the two keys it explained.
  • tests/test_structured_docs.py — 26 new checks (+113 total, from 87) that derive the key sets
    from laya/agent.py and laya/onnx_agent.py with ast and hold each documented shape to the
    path it illustrates.

Why

docs/structured.md:153 showed {"input_tokens": 42, "output_tokens": 0} and
examples/03_reading_the_result.py:16 showed the same two keys as the shape of every predict()
call. That is the shape of a call with no questions: the agents put six keys into every answered
result -- #174's truncation report (state_tokens, state_tokens_dropped, truncated,
truncated_questions) -- and a seventh, options, when the head budget collapsed a question's
options (#538). Measured on this tree, cached english checkpoint, CPU:

$ decide(agent, state, schema=Ticket, return_details=True).usage
{"input_tokens": 137, "output_tokens": 0, "state_tokens": 41, "state_tokens_dropped": 0,
 "truncated": false, "truncated_questions": []}          # laya/structured.py:277 forwards the dict whole
$ agent.system_one(state, questions).usage
{"input_tokens": 69, "output_tokens": 0, "state_tokens": 41, ...}   # the same six
shape documented what its path builds
docs/structured.md 2 6 always + options when a question's options collapse
examples/03 banner 2 the same 6 + the same conditional
README.md empty-questions sentence 2 2 -- correct, and kept as the gate's control
docs/http-api.md 6 + options correct, gated by #816

The reader who cannot see truncated cannot see that the answer came from a state the model only
partly read, which is the difference between trusting 0.95 and knowing it was 0.95 of a truncated
input. The example is where the docs send people for this shape, so the two-key version is the one
most likely to be copied into glue code.

Because the gate is per-path rather than blanket, README.md's two-key sentence stays: it is about
agent.predict(state, {}), whose result really is that shape. The gate also found a second
empty-answers literal -- predict_long builds one, and its usage carries windows -- so the
derivation is scoped to the function that holds the answered usage = {...} literal. Asking the
docs for windows would have been a false claim: decide cannot return that shape.

How it was verified

python tests/test_structured_docs.py     # 87 passed -> 113 passed, 0 failed
python tests/test_empty_questions.py     # OK
python tests/test_truncation.py          # 37 passed, 0 failed
python tests/test_doc_tables.py          # exit 0
python tests/test_structured.py          # all passed
python tests/test_structured_api.py      # all passed
python -m compileall -q laya/ tests/     # clean; examples/03 compiles too
python -m compileall -q examples/03_reading_the_result.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120     # All checks passed!
ruff check tests/test_structured_docs.py --select=E9,F63,F7,F82,F401,F811,F841 --line-length=120

zensical build --strict was not run: zensical is not installed here. The change adds no link,
no nav entry and no heading, so nothing the strict build resolves is touched -- but that is a
gap in this report, not a pass.

Then the new checks were mutation-checked. Eleven arms, control green, none survived:

arm verdict the suite's own count
control: committed tree green 113 passed, 0 failed
1 the page's shape back to the two-key claim red 112 / 1
2 example 03's banner back to the two-key claim red 112 / 1
3 the page promises options as an always key red 112 / 1
4 the page stops explaining the truncation keys red 111 / 2
5 README's empty-questions sentence widened to the answer's shape red 112 / 1
6 the page names options without bounding it as conditional red 112 / 1
7 the agent stops reporting truncated red 107 / 4
8 onnx renames the worst-case key red 112 / 1
9 the answered path adds a second conditional key red 111 / 2
10 decide forwards a subset instead of the block red 112 / 1
11 the no-questions path grows a key README never mentions red 111 / 2

Arm 11 needed two tries, and the first one is worth recording: the initial helper identified the
no-questions literal by its key set ({"input_tokens", "output_tokens"}), so an arm that added a
third key made the helper raise instead of failing a named check -- a gate that aborts rather than
reports, and a control defined by the thing it is meant to check. It now finds that literal by
structure (the result dict whose answers is {}) inside the answered path's own function, and
arm 11 turns red on README's empty-questions shape stays the empty one.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Fixes # -- no open issue claims this, and no open PR touches docs/structured.md,
examples/03_reading_the_result.py or tests/test_structured_docs.py (checked against all 28 open
PRs' file lists).

Two claims were deliberately left alone, because they cannot be checked from this repository:
README.md:607 calls the payload's usage block {input_tokens, output_tokens} inside a sentence
about TypeSafe's hosted Jev wire format -- comparing to a hosted API is out of scope here and the
sentence would need Jev's own schema to fix honestly. docs/hooks/api.md:59 documents the hook
context's
usage, which laya/hooks.py:452-456 aggregate_usage really does return as two summed
keys; that row is correct and stays.

Related: #816 (merged as 350df4b) pinned this contract for docs/http-api.md's Response section;
this covers the two other places it is written down, in the suite that already gates this page.
#817 is open on the laya-ts side, filling in the same usage.options field there.

@aashish254
aashish254 force-pushed the fix-usage-shape-claims branch from 1e9b962 to 4c26433 Compare October 1, 2026 18:16
@aashish254

Copy link
Copy Markdown
Contributor Author

Short answer: the red check was not from this PR's code -- rebased onto current main and it should clear.

The only failing check was TypeScript (node20), on laya-ts/tests/revision-pinning.test.ts: createNodeProvider refuses mismatched ONNX digests expects a /SHA-256 mismatch/ throw but got Cannot find package 'onnxruntime-node'. That test lives in the laya-ts lane and this PR never touches it -- the three files changed are docs/structured.md, examples/03_reading_the_result.py, tests/test_structured_docs.py.

Root cause was a stale base: this branch was cut 15 commits behind main, before the onnxruntime-node optional dependency landed in the lockfile, so the import failed before the digest check ran. Rebased onto current main (e5e2883); git diff main..HEAD -- laya-ts/ is now empty, so the TypeScript job runs byte-identical to green main.

Local gates on the rebased tree:

  • tests/test_structured_docs.py 113 passed / 0 failed
  • ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 clean
  • python3 -m compileall -q laya/ tests/ examples/ clean

@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.24.

Deriving the key sets from laya/agent.py and laya/onnx_agent.py with ast is the third page in this release to get that treatment, after #811's /health contract and #816's response keys, and it is the right pattern: a documented shape that is typed into a test is two copies of the same claim, and both drift together.

I checked the gate is load-bearing rather than decorative. Renaming one key in the documented sample:

FAIL docs/structured.md's usage shape is what an answer builds:
  got  [... 'state_tokens_dropped', ...]
  want [... 'state_tokens', 'state_tokens_dropped', ...]

So it fails on exactly the drift it exists to catch, and it goes back to 113 passing when reverted.

Fixing examples/03_reading_the_result.py in the same PR matters more than the docs page, for the reason its own docstring gives: it is the example people are sent to when they want to know what an answer contains, and it was showing two keys out of six. That example also got answer_confidence added in #769 this release, so it has now been corrected twice in one cycle, which suggests it is worth the gate.

Verified: tests/test_structured_docs.py 113 checks, test_structured, test_structured_api and test_doc_tables pass, full ci.yml test lane plus the pytest lane green, and a decision diff against 0.3.23 on the real english and multilingual checkpoints showing zero decision flips.

Thank you.

@NandhaKishorM
NandhaKishorM merged commit 4aa6761 into NandhaKishorM:main Oct 1, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants