Skip to content

docs(examples): the truncation pages must teach the report and the clamp they deny - #841

Merged
NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
aashish254:example-truncation-report
Oct 2, 2026
Merged

NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
aashish254:example-truncation-report

Conversation

@aashish254

@aashish254 aashish254 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What

Two truncation-teaching examples deny things the code does, and tests/test_truncation.py grows the
gate that makes that unfalsifiable later:

  • examples/15_truncation_basics.py printed the batch token total and then told the reader
    "nothing warns you". It now prints the four report keys off the same call, plus a second call whose
    state fits, so a reader sees truncated=False with an empty truncated_questions too.
  • examples/37_long_document.py said build_sequence has a truncate_left=True switch "but
    predict/system_one does not expose it", and that the tail is "only recoverable by setting the
    budget". Both agents derive the clamp from the state's type (truncate_left = isinstance(state, list)), so the tail is reachable without touching the budget. It now measures that: the same
    1452-token state handed over as a string and as the identical list, printing which end reached the
    model.
  • The gate reads the usage key set out of laya/agent.py by AST (not repeated in the test) and
    holds both examples to it, plus the clamp rule out of both agents' source.
  • examples/03 loses two hand-counted ordinals ("the four keys after them") that no gate can see.

4 files, +198/−17: tests/test_truncation.py (+112/−1), examples/37_long_document.py (+63/−8),
examples/15_truncation_basics.py (+20/−5), examples/03_reading_the_result.py (+3/−3).

Why

examples/README.md sends a reader to 15 for "what happens when the state does not fit max_len,
and how to see it in usage", and to 37 for the window arithmetic. Each page asserted the opposite
of what ships:

Truncation is silent in both directions: nothing warns you, and the answer looks as certain as any
other. (15)

build_sequence has a truncate_left=True switch to keep the tail instead, but
predict/system_one does not expose it. Anything longer than the window is invisible to the
model, and nothing warns you. (37)

The first was false when it was written: predict_batch has published truncated, state_tokens,
state_tokens_dropped and truncated_questions since #174. The second is false in the other
direction -- laya/agent.py:1023 and laya/onnx_agent.py:606 pick the clamp side from the state's
type, because a chronological list serializes newest-last, and state_room's own docstring already
documents it ("the start is kept, or the end for a conversation turn list"). A reader who believes 37
thinks a long document's tail is unrecoverable except by re-configuring the window; the actual answer
is to hand the state over as a list.

Neither claim could fail anything, because nothing compared an example's prose with the code that
builds the dict and the clamp. That is the gap the gate closes, in both directions: a key added to
the dict has to be taught by both examples, and a key renamed or dropped fails the example still
pointing at it.

Measured on laya-multilingual, one 1452-token state, one question, default window:

   the same 1452 tokens as a string : `truncate_left` = isinstance(state, list) -> False,
        so `state_tokens_dropped`=465 (my slice: 465)
                           the closing note is in it: False   noul=0.998  conf=0.998
   the same 1452 tokens as a list   : `truncate_left` = isinstance(state, list) -> True,
        so `state_tokens_dropped`=465 (my slice: 465)
                           the closing note is in it: True    noul=1.000  conf=1.000

Same characters, same state_tokens, same state_tokens_dropped -- the type alone chose which end
the model read. Both answers say yes, which is the second half of the lesson and why the example says
so: the report, not the score, is what tells you the evidence was cut.

How it was verified

python tests/test_truncation.py
python tests/test_structured_docs.py
python examples/15_truncation_basics.py
python examples/37_long_document.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya tests examples

tests/test_truncation.py -> 57 passed, 0 failed (37 on main). The new 20: 6 pinning example 15, 10
pinning example 37, 2 on the clamp derivation in each agent, 2 on the usage dict itself being one
literal and bigger than the two token totals. tests/test_structured_docs.py -> 113 checks passed, 0 failed, unchanged. ruff with CI's own selection -> All checks passed! (laya/ is untouched by
this PR). Both examples ran offline against the cached checkpoints.

Every suite that reads examples/ re-ran green: test_compose_env 237, test_server_example 50,
test_example_server_limits 94, test_example_server_errors 14,
test_example_server_per_call_controls 42, test_evals/test_hooks_api/test_mcp/test_packaging
clean.

Mutation witness (15 arms, 15 behaved)

Each arm edits a real file, runs the gate, and is scored by which named checks failed. All originals
are held in memory and rewritten before every arm; nothing is restored with git.

arm result
baseline, the committed tree 57/57 green
laya/agent.py clamps from neither side (truncate_left = False) KILL -- clamp/laya/agent.py
laya/agent.py clamps by the wrong type (isinstance(state, dict)) KILL -- clamp/laya/agent.py
laya/onnx_agent.py clamps from neither side KILL -- clamp/laya/onnx_agent.py
37 re-publishes the old does not expose it banner sentence KILL -- 37/a sentence about \truncate_left`, 37/a claim that the cut is invisible`
37 says "nothing warns you" without naming truncated KILL -- 37/a claim that the cut is invisible
37 stops printing the report on the call that truncates KILL -- 37/prints every key of the report
37 reads usage["truncated_q"] KILL -- 37/reads no key, 37/prints every key
37 drops the backticks from every state_tokens_dropped KILL -- 37/names \state_tokens_dropped``
37 stops naming truncate_left anywhere KILL -- 37/teaches the clamp by its real name
the agents stop publishing state_tokens KILL -- 15/reads no key, 37/reads no key (+1)
the agents rename truncated KILL -- 7 checks across both examples
control: #687's inline reformatting (truncate_left=isinstance(state, list) in the call, no local) green, as it should be
control: a true sentence saying the scores hide the cut, naming truncated green
the clamp sentence paraphrased to "the state's type" KILL -- see the second note below

Three limits I want on the record rather than discovered in review:

  • Deleting only 15's four label print lines while keeping its "a state that fits the window" line
    leaves the gate green, by design: the keys are still read from a live call and still named in
    backticks. That arm therefore deletes both blocks. A rule making the labelled block itself
    mandatory is stronger than I thought was true, and I did not write it.
  • The truncate_left sentence rule is deliberately per sentence. Paraphrasing "takes it from
    isinstance(state, list)" to "takes it from the state's type" fails even though the page names the
    expression two lines earlier -- the pairing of the flag with the type, in one sentence, is the
    thing a reader takes away. Say the word if you would rather it were per string.
  • CI does not run examples/, so the printed values are verified by local runs against the cached
    checkpoints plus the static gate. The gate adds only stdlib imports (ast, re) to a file that
    already imports laya.common, and runs offline in the job that already calls
    python tests/test_truncation.py (ci.yml:125).

No overlap with open work: against every open PR's file list, nothing but this PR touches
examples/15_truncation_basics.py, examples/03_reading_the_result.py or
tests/test_truncation.py, and nothing touches examples/37_long_document.py (#687 touches
tests/test_truncation_direction.py, a different file, and rewrites the clamp line inline -- the
derivation is read as a pattern for exactly that, and the control arm above proves it). The gate is
appended at EOF, after the existing usage/ source checks, and leaves them untouched.

Checklist

  • Focused on one change -- what the truncation examples tell a reader about usage and about the
    clamp, and the gate that holds them to the code. The examples/03 edit is three words ("the
    four keys after them" -> "the keys after them", "a seventh key" -> "one more key") removed in
    the same pass because they are hand-counted transcriptions of the dict this PR is about and no
    gate can see them.
  • Rebased on the latest main (base 4aa6761, which is upstream/main at push time; git rev-list --count HEAD..upstream/main -> 0).
  • Tests pass locally (57 passed, 113 checks passed, every suite that reads examples/ green,
    ruff and compileall clean, mutation 15/15).
  • Docs or examples updated when the public API changed -- no public API changed; this PR is the
    examples change, and the test is what keeps the pages from drifting back.

The docs (strict build) job is the inherited red every main-based PR shows until #794 lands; no
file in this PR is a docs page, so it adds no nav or link surface.

This covers the documentation half of #174: the report exists, and now the pages that teach it cannot
claim otherwise without failing a test.

…does not exist

The example was written to show that a state longer than the window is cut down to its
head, and its closing paragraph said "Truncation is silent in both directions: nothing
warns you, and the answer looks as certain as any other". `predict_batch` has published
`truncated`, `state_tokens`, `state_tokens_dropped` and `truncated_questions` since NandhaKishorM#174,
so the page taught readers to estimate from the length of what they sent instead of
reading the four keys sitting next to the token count it already printed.

It now prints them off the same call, and prints a second call whose state fits so the
report reads `truncated`=False with an empty `truncated_questions`. The sentence about
the two answers staying indistinguishable is kept -- that is what the example is
about -- but it names `truncated` as the thing that says so.
The page that draws the whole `usage` block then described it as "the four keys after
them" and `options` as "a seventh key". tests/test_structured_docs.py holds the drawn
block to the dict the agents build, so a fifth report key turns those two counts into a
false statement while the gate stays green -- the count is not in the block it checks.
Naming the keys does the teaching either way; the ordinals only rot.
Example 15 was free to describe the report however it liked: nothing compared its
printed keys with the `usage = {...}` literal in laya/agent.py, which is how the wrong
"nothing warns you" sentence survived NandhaKishorM#174's fix in the page a reader comes to for that
exact answer.

The key set is now read out of that literal by AST -- not repeated here, so a key added
to the dict has to be taught and a renamed one fails the example still pointing at it --
and the example is held to it in both directions: every report key read off a live call,
no key read that the agents do not publish, each one named in backticks so it is
greppable. A sentence may still say the cut is invisible in the answers, but only if it
names the key that reports it.

Example 03's drawn shape and `options` are already gated by tests/test_structured_docs.py,
so this only adds what that gate cannot see.
…ly picks

The banner said `build_sequence` has a `truncate_left=True` switch that
`predict`/`system_one` does not expose, and that anything past the window is
invisible with nothing warning. Both are wrong in opposite directions: `predict`
derives `truncate_left = isinstance(state, list)` on both agents, so the tail is
reachable without touching the budget, and the `usage` report has named the cut
since NandhaKishorM#174. `state_room` documents the type rule already.

Measured on laya-multilingual: one 1452-token state handed over as a string and
as the identical list drops 465 tokens either way, and the only difference is
which end survives -- the closing sentence reaches the model in the second call
and not in the first, while `noul` says yes to both.
@aashish254 aashish254 changed the title docs(examples): example 15 must print the report it calls silent docs(examples): the truncation pages must teach the report and the clamp they deny Oct 1, 2026
@aashish254

Copy link
Copy Markdown
Contributor Author

Short answer: this PR got a second example in it, so I retitled it.

It started as example 15 telling the reader "nothing warns you" when the report has warned since
#174. While writing the gate I found example 37 making the mirror mistake: it says predict does
not expose the keep-the-tail switch, when both agents pick the clamp side from the state's type
(truncate_left = isinstance(state, list), agent.py:1023 and onnx_agent.py:606) and state_room's
docstring already documents that. So the same 20 checks now hold both pages to the code, and 37
measures it: one 1452-token state handed over as a string and as the identical list drops 465 tokens
either way, and only which end the model read changes.

I folded it in rather than opening a separate PR because both live in tests/test_truncation.py at
EOF, so two PRs here would collide on the same lines. The mutation table in the body is now 15 arms
(12 kills, 3 green controls), and the gate is 57 passed, 0 failed against 37 on main.

@NandhaKishorM
NandhaKishorM merged commit 553012d into NandhaKishorM:main Oct 2, 2026
23 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.24.

Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, compileall clean, both TypeScript packages green (laya-ts 597 tests, the SDK 26), and a decision diff against 0.3.23 on the real english and multilingual checkpoints (74 answers over choice, score and noul; short and long states; budgeted calls and predict_long) showing zero decision flips and a max probability delta of 0.000000.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants