Skip to content

docs(examples): example 21 must read its cost claim off the measurement it takes - #975

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:fix-example21-timing-claim
Oct 7, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:fix-example21-timing-claim

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

examples/21_routing_without_running.py is the "route without running" page — the intro to Router.route(), the pure-Python script/language detection call that picks a checkpoint without building one. The page measures its own cost:

_, median_ms = timed(lambda: r.route({"body": CASES[1][1]}), repeat=200)
t0 = time.perf_counter()
for _, text in CASES:
    r.route({"body": text})
total_ms = (time.perf_counter() - t0) * 1000

…and prints both numbers. Then it closed with:

Ten routing decisions cost well under a millisecond in total, and the router still
holds zero checkpoints.

Three quantities in that one sentence were already on the page and were not read back:

  • "Ten" — a hardcoded count while len(CASES) was the loop bound one line above. Lengthen CASES and the number in the prose goes stale.
  • "well under a millisecond" — a threshold claim that does not know what total_ms actually is. Same page, slower CI machine (or a time.perf_counter bracket with any scheduling jitter), the sentence lies.
  • "holds zero checkpoints" — a residency claim asserted in prose, when r.loaded on the same page is right there to read.

The page now has two pure helpers next to the measurement:

def cost_verdict(total_ms, threshold_ms=1.0):
    if total_ms < threshold_ms:
        return "under %.1f ms" % threshold_ms
    return "%.1f ms or over on this run" % threshold_ms


def resident_state(r):
    if not r.loaded:
        return "the router holds zero checkpoints"
    return "the router holds %d: %r" % (len(r.loaded), list(r.loaded))

and the closing sentence interpolates len(CASES), cost_verdict(total_ms), total_ms itself, and resident_state(r). Measured on the MPS run:

   10 routing decisions cost under 1.0 ms in total (0.415 ms measured this run), and the router holds zero checkpoints.

Same conclusion, same page, but the number and the verdict both now come from the measurement, so the sentence rephrases rather than silently lies on a slower machine or a longer CASES list.

Why

Laya's routing cost is one of the load-bearing pitches the docs make: route() runs on every inbound request before you decide whether a model is needed. That pitch is only as trustworthy as the number attached to it. This is the exact same defect class the repo has been closing on the example pages, and the same fix shape:

This PR is the same pattern applied to timing and residency. Three independent quantities a page prints and then re-asserts as literal text. If the maintainer's stance is that a doc page's claims must be verifiable from the page's own measurements — which the last four merged example PRs establish — this one belongs in that group.

How it was verified

# CI path (script mode; the file is a unittest suite run directly):
python tests/test_router_memory.py
# → Ran 14 tests in 0.694s / OK. Baseline on upstream/main a4a8921 in the same venv: 9 tests OK.
#   The five new tests are all in the added `ExampleRouteCostPageTests` class; the existing
#   `RouterMemoryTests` and `ExampleResidencyPageTests` classes for example 24 are unchanged.

python examples/21_routing_without_running.py
# → exit=0 against the real checkpoints offline; prints the closing paragraph quoted above.

# gates from AGENTS.md:
ruff check laya/ examples/21_routing_without_running.py tests/test_router_memory.py \
  --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
# → All checks passed / OK.

The five new tests:

  1. test_page21_helpers_are_defined_and_pure — AST-extract cost_verdict and resident_state, assert neither body contains a print call (so they return rather than narrate).
  2. test_cost_verdict_flips_on_the_measured_total — under-threshold (0.437) yields "under 1.0 ms", over-threshold (3.7) yields the "or over" phrase, and a caller-set threshold_ms=5.0 is respected.
  3. test_resident_state_reads_the_router_state — r.loaded = [] yields the zero-checkpoint phrase; ["english"] and ["english", "multilingual"] yield the actual counts and names, and neither contains "zero".
  4. test_page21_drops_the_three_hardcoded_closing_claims — three ban regexes on main's wording (Ten routing decisions, well under a millisecond, still holds zero checkpoints) each fire on OLD_PAGE_21 (main's closing paragraph quoted verbatim) and not on the current source.
  5. test_page21_closing_interpolates_helpers_count_and_ms — the positive half: the current source contains cost_verdict(total_ms), resident_state(r), len(CASES), and a %d routing decisions cost %s in total (%.3f ms literal template. None of those substrings appear in OLD_PAGE_21, so the rules have teeth.

Mutation harness

laya-bench/mutate_example21_gate.py — six edits to the example, each byte-restored, run with PYTHONDONTWRITEBYTECODE=1:

  caught  M1 closing paragraph reverts to main's three hardcoded claims  (test_page21_drops_the_three_hardcoded_closing_claims)
  caught  M2 cost_verdict ignores total_ms, always reports under  (test_cost_verdict_flips_on_the_measured_total)
  caught  M3 resident_state ignores r.loaded, always reports zero  (test_resident_state_reads_the_router_state)
  caught  M4 rename cost_verdict to cost_verdict_v2  (test_page21_helpers_are_defined_and_pure)
  caught  M5 helper prints instead of returning  (test_page21_helpers_are_defined_and_pure)
  caught  M6 closing drops the interpolated len(CASES), hardcodes the count  (test_page21_closing_interpolates_helpers_count_and_ms)
6/6 caught, tree restored: True.

Baseline check runs first and the harness exits 2 if the pristine tree is not green, so an aborted suite cannot masquerade as a green run.

Relationship to the other open example gates

None of #972 / #973 / #974 touch examples/21_routing_without_running.py or tests/test_router_memory.py — verified via gh pr list --state open --json files. Deconflicted.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (a4a8921 release: 0.3.28)
  • Tests pass locally (script mode: 14 tests, OK)
  • Docs or examples updated when the public API changed (no public API change; example prose and a gate)
  • ruff check --select=E9,F63,F7,F82,F401,F811 --line-length=120 clean
  • python -m compileall -q laya/ tests/ clean
  • No secrets in the diff (grep for api key / secret / password / bearer / private key / BEGIN OPENSSH / AKIA / ghp_ — no matches)

The page measures `total_ms` and `median_ms` from `perf_counter` / `timed()` on
the same 10-case loop, prints them, and then closed with:

    Ten routing decisions cost well under a millisecond in total, and the router
    still holds zero checkpoints.

Every clause in that sentence is a quantity the page already had and did not
read back: "Ten" was hardcoded while `len(CASES)` sat right there; "well under
a millisecond" asserted a threshold that goes stale on slower hardware; "holds
zero checkpoints" asserted residency rather than reading `r.loaded`. This is
the same defect class NandhaKishorM#850 fixed on example 28's preset page and NandhaKishorM#972 fixed on
example 40's Part C, and the fix is the same shape: two pure helpers turn the
measurement into a verdict and the paragraph interpolates the verdict and the
raw number.

Measured on the current MPS run: `0.415 ms total / 10 cases`, `router.loaded
= []` after every call, so the page prints "10 routing decisions cost under
1.0 ms in total (0.415 ms measured this run), and the router holds zero
checkpoints". Lengthen `CASES` or slow the machine and the sentence
rephrases rather than lies.
Add `ExampleRouteCostPageTests` next to the existing `ExampleResidencyPageTests`
for example 24 -- the same file already owns router-page gates. Five new tests:

  - helpers are defined and pure (no `print` in a helper body)
  - `cost_verdict` flips on the measured `total_ms` at both sides of the bar,
    and honours a caller-set threshold
  - `resident_state` reads `r.loaded`: empty yields the zero-checkpoint phrase,
    non-empty yields the actual count and names
  - three bans on main's wording (a hardcoded "Ten routing decisions", an
    unmeasured "well under a millisecond", an asserted "still holds zero
    checkpoints") fire on `OLD_PAGE_21` -- the closing paragraph quoted verbatim
    from main -- and not on the current source
  - a positive rule proves the current source interpolates `cost_verdict(...)`,
    `resident_state(...)` and `len(CASES)`, and that the closing sentence still
    carries a raw `%.3f ms`

Suite: 14 tests OK (baseline 9 tests OK on upstream/main a4a8921). Mutation
harness `laya-bench/mutate_example21_gate.py`: six mutants covering the closing
paragraph, `cost_verdict` ignoring `total_ms`, `resident_state` ignoring
`r.loaded`, a rename, a print-instead-of-return, and a hardcoded count in the
closing -- 6/6 caught, tree restored: True.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.3.29.

An example that states a cost claim it does not derive from its own measurement teaches the reader to trust a number the page cannot support. Reading the claim off the measurement the example already takes is the only version that stays true when the hardware changes, which on this repository it does: the same operation is a different number on a T4, on CPU and on Apple silicon.

This is the same discipline I applied to #977 in this pass, where a "sub-35ms" headline turned out to be a T4 figure stated unconditionally and measured 125.6 ms p50 on my machine. Pages that do their own arithmetic do not have that problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants