Repository navigation
docs(examples): example 21 must read its cost claim off the measurement it takes - #975
Merged
NandhaKishorM merged 2 commits intoOct 7, 2026
Merged
Conversation
The page measures `total_ms` and `median_ms` from `perf_counter` / `timed()` on
the same 10-case loop, prints them, and then closed with:
Ten routing decisions cost well under a millisecond in total, and the router
still holds zero checkpoints.
Every clause in that sentence is a quantity the page already had and did not
read back: "Ten" was hardcoded while `len(CASES)` sat right there; "well under
a millisecond" asserted a threshold that goes stale on slower hardware; "holds
zero checkpoints" asserted residency rather than reading `r.loaded`. This is
the same defect class NandhaKishorM#850 fixed on example 28's preset page and NandhaKishorM#972 fixed on
example 40's Part C, and the fix is the same shape: two pure helpers turn the
measurement into a verdict and the paragraph interpolates the verdict and the
raw number.
Measured on the current MPS run: `0.415 ms total / 10 cases`, `router.loaded
= []` after every call, so the page prints "10 routing decisions cost under
1.0 ms in total (0.415 ms measured this run), and the router holds zero
checkpoints". Lengthen `CASES` or slow the machine and the sentence
rephrases rather than lies.
Add `ExampleRouteCostPageTests` next to the existing `ExampleResidencyPageTests`
for example 24 -- the same file already owns router-page gates. Five new tests:
- helpers are defined and pure (no `print` in a helper body)
- `cost_verdict` flips on the measured `total_ms` at both sides of the bar,
and honours a caller-set threshold
- `resident_state` reads `r.loaded`: empty yields the zero-checkpoint phrase,
non-empty yields the actual count and names
- three bans on main's wording (a hardcoded "Ten routing decisions", an
unmeasured "well under a millisecond", an asserted "still holds zero
checkpoints") fire on `OLD_PAGE_21` -- the closing paragraph quoted verbatim
from main -- and not on the current source
- a positive rule proves the current source interpolates `cost_verdict(...)`,
`resident_state(...)` and `len(CASES)`, and that the closing sentence still
carries a raw `%.3f ms`
Suite: 14 tests OK (baseline 9 tests OK on upstream/main a4a8921). Mutation
harness `laya-bench/mutate_example21_gate.py`: six mutants covering the closing
paragraph, `cost_verdict` ignoring `total_ms`, `resident_state` ignoring
`r.loaded`, a rename, a print-instead-of-return, and a hardcoded count in the
closing -- 6/6 caught, tree restored: True.
Owner
|
Merged in 0.3.29. An example that states a cost claim it does not derive from its own measurement teaches the reader to trust a number the page cannot support. Reading the claim off the measurement the example already takes is the only version that stays true when the hardware changes, which on this repository it does: the same operation is a different number on a T4, on CPU and on Apple silicon. This is the same discipline I applied to #977 in this pass, where a "sub-35ms" headline turned out to be a T4 figure stated unconditionally and measured 125.6 ms p50 on my machine. Pages that do their own arithmetic do not have that problem. |
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
examples/21_routing_without_running.pyis the "route without running" page — the intro toRouter.route(), the pure-Python script/language detection call that picks a checkpoint without building one. The page measures its own cost:…and prints both numbers. Then it closed with:
Three quantities in that one sentence were already on the page and were not read back:
len(CASES)was the loop bound one line above. LengthenCASESand the number in the prose goes stale.total_msactually is. Same page, slower CI machine (or atime.perf_counterbracket with any scheduling jitter), the sentence lies.r.loadedon the same page is right there to read.The page now has two pure helpers next to the measurement:
and the closing sentence interpolates
len(CASES),cost_verdict(total_ms),total_msitself, andresident_state(r). Measured on the MPS run:Same conclusion, same page, but the number and the verdict both now come from the measurement, so the sentence rephrases rather than silently lies on a slower machine or a longer
CASESlist.Why
Laya's routing cost is one of the load-bearing pitches the docs make:
route()runs on every inbound request before you decide whether a model is needed. That pitch is only as trustworthy as the number attached to it. This is the exact same defect class the repo has been closing on the example pages, and the same fix shape:answer_confidenceis "calibrated"; now it reads back the temperature it actually applies per bucket.confidenceon a score is the entropy itself, when the code returns1 - H/log(k); now it recomputes in place.This PR is the same pattern applied to timing and residency. Three independent quantities a page prints and then re-asserts as literal text. If the maintainer's stance is that a doc page's claims must be verifiable from the page's own measurements — which the last four merged example PRs establish — this one belongs in that group.
How it was verified
The five new tests:
test_page21_helpers_are_defined_and_pure— AST-extractcost_verdictandresident_state, assert neither body contains aprintcall (so they return rather than narrate).test_cost_verdict_flips_on_the_measured_total— under-threshold (0.437) yields"under 1.0 ms", over-threshold (3.7) yields the "or over" phrase, and a caller-setthreshold_ms=5.0is respected.test_resident_state_reads_the_router_state—r.loaded = []yields the zero-checkpoint phrase;["english"]and["english", "multilingual"]yield the actual counts and names, and neither contains "zero".test_page21_drops_the_three_hardcoded_closing_claims— three ban regexes on main's wording (Ten routing decisions,well under a millisecond,still holds zero checkpoints) each fire onOLD_PAGE_21(main's closing paragraph quoted verbatim) and not on the current source.test_page21_closing_interpolates_helpers_count_and_ms— the positive half: the current source containscost_verdict(total_ms),resident_state(r),len(CASES), and a%d routing decisions cost %s in total (%.3f msliteral template. None of those substrings appear inOLD_PAGE_21, so the rules have teeth.Mutation harness
laya-bench/mutate_example21_gate.py— six edits to the example, each byte-restored, run withPYTHONDONTWRITEBYTECODE=1:Baseline check runs first and the harness exits
2if the pristine tree is not green, so an aborted suite cannot masquerade as a green run.Relationship to the other open example gates
None of #972 / #973 / #974 touch
examples/21_routing_without_running.pyortests/test_router_memory.py— verified viagh pr list --state open --json files. Deconflicted.Checklist
main(a4a8921 release: 0.3.28)OK)ruff check --select=E9,F63,F7,F82,F401,F811 --line-length=120cleanpython -m compileall -q laya/ tests/clean