Repository navigation
fix(examples): example 28 must compute the conclusions it prints - #850
Merged
NandhaKishorM merged 1 commit intoOct 2, 2026
Merged
Conversation
The moderation page asserted three things its own numbers contradicted. It printed "benign -> every flag 0.000" from `toxic` alone while its table showed the benign post reading 0.066 on `spam`; it wrote the `severity` comparison signs into the format string, so "orders the set correctly" stayed true whatever the scores came back as; and it named `harassment` and `threat` as the questions that separate the two hostile posts, while the gaps measured on the same run were harassment 0.518 and spam 0.193, with threat third at 0.054. The summary now derives its flag list and rubric ceiling from `laya.moderation_questions()` and prints every probability it speaks about, so the names, gaps and ranking follow the answers instead of the other way round.
Owner
|
Merged for 0.3.24. Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, Thank you. |
This was referenced Oct 6, 2026
NandhaKishorM
pushed a commit
that referenced
this pull request
Oct 7, 2026
The page measures `total_ms` and `median_ms` from `perf_counter` / `timed()` on
the same 10-case loop, prints them, and then closed with:
Ten routing decisions cost well under a millisecond in total, and the router
still holds zero checkpoints.
Every clause in that sentence is a quantity the page already had and did not
read back: "Ten" was hardcoded while `len(CASES)` sat right there; "well under
a millisecond" asserted a threshold that goes stale on slower hardware; "holds
zero checkpoints" asserted residency rather than reading `r.loaded`. This is
the same defect class #850 fixed on example 28's preset page and #972 fixed on
example 40's Part C, and the fix is the same shape: two pure helpers turn the
measurement into a verdict and the paragraph interpolates the verdict and the
raw number.
Measured on the current MPS run: `0.415 ms total / 10 cases`, `router.loaded
= []` after every call, so the page prints "10 routing decisions cost under
1.0 ms in total (0.415 ms measured this run), and the router holds zero
checkpoints". Lengthen `CASES` or slow the machine and the sentence
rephrases rather than lies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
examples/28_presets_moderation.pyprints three conclusions about the moderation preset's answers,and all three were typed into the
printstatements instead of computed from the answers. This makesthe page derive them, and adds a weight-free gate in
tests/test_confidence.pythat drives the page'sown conclusion functions on fabricated answers.
print(" benign -> every flag %.3f, severity %.2f / 3"\n % (ben["toxic"]["noul"], ben["severity"]["score"])), i.e. one flag's value under a word that means four. Theside by sidetable above it, from the same run, prints that post as
benign 0.000 0.000 0.000 0.066 0.24 / 3.severityorders the set correctly (%.2f > %.2f > %.2f > %.2f)" — the comparison signs areliterals in the format string, so the sentence is true for any four numbers the model returns. It
now prints a chain built by sorting, with each post named, and states separately whether that
ranking is the order the posts are listed in.
harassmentandthreat" — names two of four flags,hardcoded. The page's own run measures
harassmentas the widest gap andspamas the second, sothe sentence omits the second-widest and includes the third. It now prints every per-flag gap with
its sign and reads the names off the sorted gaps.
The page also derives its flag list from
laya.moderation_questions()(q["type"] == "noul") and itsrubric ceiling from
len(criteria) - 1, so the Swedish-presets work in #729 and the workflow presets in#153 cannot leave this page describing a shape the preset no longer has.
laya/presets.pyis not inthis diff.
Why
An example is the API contract most readers actually run. This one's conclusions are the part a reader
quotes in a moderation triage rule, and they were the part least tied to the evidence: the universality
line contradicts the table above it, the ordering sentence cannot be falsified by its own numbers, and
the separator sentence is a claim about four questions that names two of them. Which question crosses
0.5 is a property of the checkpoint and the device, so a hardcoded conclusion is stale the moment
anything is re-scored — there is no CI signal when that happens, because nothing compares the prose to
the answers.
How it was verified
The suite, before and after (torch 2.14.0 / py3.14 venv,
HF_HUB_OFFLINE=1, no weights loaded):Eight tests. Four drive the page's helpers, which are extracted from
examples/28_presets_moderation.pyby AST and
exec'd here (only the function definitions, so the module'sload("english")never runs);the answers they are fed are fabricated fixtures, labelled as such in the section comment. They
assert:
flag_lineshows the non-zero flag rather than dropping it;past_halfuses the same strict> 0.5cut_common.describelabels with (so 0.5 reads false);gapsorders by magnitude and keepsthe sign;
rank_by_severityflips its verdict when the ranking does, and the chain it prints alwaysdescends. One pins the preset shape this suite's own docstring has claimed all along -- four
noulplus one
scorewith four criteria. No suite asserted it:tests/test_local_e2e.pydrives the preset,but needs checkpoints and only checks which way
toxicpoints, never the key set, the types or therubric size. Two hold the page's prose to ban and positive rules, and one proves each helper is
called at module level and reads no globals outside its arguments, so the gate cannot be satisfied
by dead code.
Every ban is witnessed against main's wording, which is embedded as
OLD_PAGE, so a regex that cannottrip the text it bans fails the suite instead of passing quietly:
Mutation harness — 22 arms over
examples/28_presets_moderation.py,tests/test_confidence.pyandlaya/presets.py, each run withpython -B, every file restored and byte-compared after every arm.P*breaks one line of the page's logic;
B*puts one of main's three sentences back;C*puts thesentence back and deletes the rule that catches it (the suite must go green, which is what proves
that rule was doing the work);
D1renames a preset key inlaya/presets.pylocally;Hreplaces thewhole page with
git show 4aa6761:examples/28_presets_moderation.py:The example runs end to end on MPS against the English checkpoint. Its conclusion block now prints
(this is the run's own output, with the numbers this checkpoint gave):
The probabilities are unchanged from main's run — only what the page says about them moved. Where the
old prose said the benign post has "every flag 0.000", the new output says
spam 0.066, and where itnamed
harassmentandthreatas the separators, the measured gaps putspamsecond andthreatthird.
spam -0.193is printed with its sign because the generic insult reads more spam than thetargeted one, which a name-only sentence would have hidden.
Checklist
suite;
laya/is untouchedmain(4aa6761)122 passed, 0 failed; 22/22 mutation arms behaved as expected)example page is the thing being fixed
Fixes #
No issue: found by reading example 28's conclusion block against the table the same page prints above
it. Same class as #841, #842, #843, #844 and #845 -- an example page whose prose or printed conclusion
does not match what the code or the run actually does -- and no file overlap: those touch examples
03/15/37, 35, 39, 24 and 33, with
tests/test_truncation.py,test_runtime_fixes.py,test_load_errors.py,test_router_memory.pyandtest_state_budget.py.