Skip to content

fix(examples): example 28 must compute the conclusions it prints - #850

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:example28-moderation-claims
Oct 2, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:example28-moderation-claims

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

examples/28_presets_moderation.py prints three conclusions about the moderation preset's answers,
and all three were typed into the print statements instead of computed from the answers. This makes
the page derive them, and adds a weight-free gate in tests/test_confidence.py that drives the page's
own conclusion functions on fabricated answers.

  1. "every flag 0.000" — the line read
    print(" benign -> every flag %.3f, severity %.2f / 3"\n % (ben["toxic"]["noul"], ben["severity"]["score"])), i.e. one flag's value under a word that means four. The side by side
    table above it, from the same run, prints that post as
    benign 0.000 0.000 0.000 0.066 0.24 / 3.
  2. "severity orders the set correctly (%.2f > %.2f > %.2f > %.2f)" — the comparison signs are
    literals in the format string, so the sentence is true for any four numbers the model returns. It
    now prints a chain built by sorting, with each post named, and states separately whether that
    ranking is the order the posts are listed in.
  3. "the questions that separate them are harassment and threat" — names two of four flags,
    hardcoded. The page's own run measures harassment as the widest gap and spam as the second, so
    the sentence omits the second-widest and includes the third. It now prints every per-flag gap with
    its sign and reads the names off the sorted gaps.

The page also derives its flag list from laya.moderation_questions() (q["type"] == "noul") and its
rubric ceiling from len(criteria) - 1, so the Swedish-presets work in #729 and the workflow presets in
#153 cannot leave this page describing a shape the preset no longer has. laya/presets.py is not in
this diff.

Why

An example is the API contract most readers actually run. This one's conclusions are the part a reader
quotes in a moderation triage rule, and they were the part least tied to the evidence: the universality
line contradicts the table above it, the ordering sentence cannot be falsified by its own numbers, and
the separator sentence is a claim about four questions that names two of them. Which question crosses
0.5 is a property of the checkpoint and the device, so a hardcoded conclusion is stale the moment
anything is re-scored — there is no CI signal when that happens, because nothing compares the prose to
the answers.

How it was verified

python tests/test_confidence.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
ruff check examples/28_presets_moderation.py tests/test_confidence.py --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
python examples/28_presets_moderation.py

The suite, before and after (torch 2.14.0 / py3.14 venv, HF_HUB_OFFLINE=1, no weights loaded):

baseline, main's tests/test_confidence.py:  114 passed, 0 failed
this branch:                                122 passed, 0 failed   (+8)
main's example 28 under this gate:          115 passed, 7 failed
  FAIL page-28/test_helpers_are_live_and_pure: example 28 lost a helper: []
  FAIL page-28/test_flag_line_prints_every_flag raised KeyError: 'noul_flags'
  FAIL page-28/test_past_half_uses_describes_cut raised KeyError: 'noul_flags'
  FAIL page-28/test_gaps_names_the_widest_separators raised KeyError: 'noul_flags'
  FAIL page-28/test_rank_cannot_assert_its_own_order raised KeyError: 'rank_by_severity'
  FAIL page-28/test_page_drops_the_hardcoded_conclusions: every flag %.3f is still in example 28
  FAIL page-28/test_page_qualifies_the_banner_and_derives_its_numbers: the banner must say that `nothing` is not zero

Eight tests. Four drive the page's helpers, which are extracted from examples/28_presets_moderation.py
by AST and exec'd here (only the function definitions, so the module's load("english") never runs);
the answers they are fed are fabricated fixtures, labelled as such in the section comment. They
assert: flag_line shows the non-zero flag rather than dropping it; past_half uses the same strict
> 0.5 cut _common.describe labels with (so 0.5 reads false); gaps orders by magnitude and keeps
the sign; rank_by_severity flips its verdict when the ranking does, and the chain it prints always
descends. One pins the preset shape this suite's own docstring has claimed all along -- four noul
plus one score with four criteria. No suite asserted it: tests/test_local_e2e.py drives the preset,
but needs checkpoints and only checks which way toxic points, never the key set, the types or the
rubric size. Two hold the page's prose to ban and positive rules, and one proves each helper is
called at module level and reads no globals outside its arguments, so the gate cannot be satisfied
by dead code.

Every ban is witnessed against main's wording, which is embedded as OLD_PAGE, so a regex that cannot
trip the text it bans fails the suite instead of passing quietly:

UNIVERSAL_ONE_FLOAT = re.compile(r"every (?:flag|score|level)[^.]{0,12}%\.\d+f", re.I)
ASSERTED_CHAIN      = re.compile(r"%\.2f\s*>\s*%\.2f")
NAMED_SEPARATORS    = re.compile(r"separat\w+[^.]{0,60}`(?:toxic|harassment|threat|spam)`"
                                 r"[^.]{0,30}`(?:toxic|harassment|threat|spam)`", re.I)
HARDCODED_FLAG_READ = re.compile(r"\[[\"\'](?:toxic|harassment|threat|spam)[\"\']\]\[[\"\']noul[\"\']\]")
NOTHING_QUALIFIED   = re.compile(r"nothing past 0\.5[^,]{0,10}, not zero", re.I)   # positive rule
DERIVED_FLAGS       = re.compile(r"q\[.type.\]\s*==\s*[\"\']noul[\"\']")            # positive rule
DERIVED_CEILING     = re.compile(r"len\(q\[.criteria.\]\)\s*-\s*1")                     # positive rule

Mutation harness — 22 arms over examples/28_presets_moderation.py, tests/test_confidence.py and
laya/presets.py, each run with python -B, every file restored and byte-compared after every arm. P*
breaks one line of the page's logic; B* puts one of main's three sentences back; C* puts the
sentence back and deletes the rule that catches it (the suite must go green, which is what proves
that rule was doing the work); D1 renames a preset key in laya/presets.py locally; H replaces the
whole page with git show 4aa6761:examples/28_presets_moderation.py:

control (committed tree): 122 passed, 0 failed []
KILLED  P1 flag_line prints one flag                 121 pass 1 fail  ['test_flag_line_prints_every_flag']
KILLED  P2 past_half takes 0.5 as true               121 pass 1 fail  ['test_past_half_uses_describes_cut']
KILLED  P3 gaps sorted narrowest first               121 pass 1 fail  ['test_gaps_names_the_widest_separators']
KILLED  P4 gaps never sorted                         121 pass 1 fail  ['test_gaps_names_the_widest_separators']
KILLED  P5 rank always claims the listed order       121 pass 1 fail  ['test_rank_cannot_assert_its_own_order']
KILLED  P6 rank chains ascending                     121 pass 1 fail  ['test_rank_cannot_assert_its_own_order']
KILLED  P7 ceiling typed as 3                        121 pass 1 fail  ['test_page_qualifies_...']
KILLED  P8 flags typed as a list                     121 pass 1 fail  ['test_page_qualifies_...']
KILLED  P9 flag_line prints instead of returning     120 pass 2 fail  ['test_flag_line_prints_every_flag', 'test_helpers_are_live_and_pure']
KILLED  P10 rank_by_severity never called            121 pass 1 fail  ['test_helpers_are_live_and_pure']
KILLED  B1 'every flag %.3f' universality            121 pass 1 fail  ['test_page_drops_the_hardcoded_conclusions']
KILLED  B2 comparison signs in the format string     121 pass 1 fail  ['test_page_drops_the_hardcoded_conclusions']
KILLED  B3 separators named by hand                  121 pass 1 fail  ['test_page_drops_the_hardcoded_conclusions']
KILLED  B4 banner claims 'nothing at all'            121 pass 1 fail  ['test_page_qualifies_...']
KILLED  B5 severity divisor typed as 3               121 pass 1 fail  ['test_page_qualifies_...']
KILLED  C1 escape: universality+flag-read bans out   122 pass 0 fail  []
KILLED  C2 escape: asserted-chain ban out            122 pass 0 fail  []
KILLED  C3 escape: named-separator ban out           122 pass 0 fail  []
KILLED  C4 escape: banner ban out                    122 pass 0 fail  []
KILLED  C5 escape: divisor ban out                   122 pass 0 fail  []
KILLED  D1 preset renames threat                     118 pass 4 fail  [..., 'test_preset_is_four_flags_and_one_rubric']
KILLED  H main's page under this gate                115 pass 7 fail  [the seven named above]

22 arms, 0 not as expected
restored control: 122 passed, 0 failed []

The example runs end to end on MPS against the English checkpoint. Its conclusion block now prints
(this is the run's own output, with the numbers this checkpoint gave):

   -- what the pattern means --
   targeted insult  -> `toxic` 0.587  `harassment` 0.662  `threat` 0.142  `spam` 0.049
                       past 0.5: `toxic`, `harassment`
   generic insult   -> `toxic` 0.584  `harassment` 0.144  `threat` 0.088  `spam` 0.242
                       past 0.5: `toxic`
   spam             -> `toxic` 0.000  `harassment` 0.003  `threat` 0.061  `spam` 1.000
                       past 0.5: `spam`
   benign           -> `toxic` 0.000  `harassment` 0.000  `threat` 0.000  `spam` 0.066
                       past 0.5: nothing

   the two hostile posts, per-flag gap (widest first, `+` = the targeted one reads higher):
   `harassment` +0.517, `spam` -0.193, `threat` +0.054, `toxic` +0.003
   the question that separates them most is `harassment` (0.517) and the least is `toxic` (0.003),
   so the least of them cannot tell a targeted insult from a generic one on its own.

   `severity` ranks the set: targeted insult 1.34 > generic insult 1.05 > spam 0.99 > benign 0.24
   that is the order the posts are listed in. Whatever the order, `severity` is a coarse
   0-3 rubric, so use it to sort a queue rather than as a hard threshold.

The probabilities are unchanged from main's run — only what the page says about them moved. Where the
old prose said the benign post has "every flag 0.000", the new output says spam 0.066, and where it
named harassment and threat as the separators, the measured gaps put spam second and threat
third. spam -0.193 is printed with its sign because the generic insult reads more spam than the
targeted one, which a name-only sentence would have hidden.

Checklist

  • Focused on one change (split unrelated work into another PR) -- one example page and one
    suite; laya/ is untouched
  • Rebased on the latest main (4aa6761)
  • Tests pass locally (122 passed, 0 failed; 22/22 mutation arms behaved as expected)
  • Docs or examples updated when the public API changed -- the public API did not change; the
    example page is the thing being fixed

Fixes #

No issue: found by reading example 28's conclusion block against the table the same page prints above
it. Same class as #841, #842, #843, #844 and #845 -- an example page whose prose or printed conclusion
does not match what the code or the run actually does -- and no file overlap: those touch examples
03/15/37, 35, 39, 24 and 33, with tests/test_truncation.py, test_runtime_fixes.py,
test_load_errors.py, test_router_memory.py and test_state_budget.py.

The moderation page asserted three things its own numbers contradicted. It printed
"benign -> every flag 0.000" from `toxic` alone while its table showed the benign post
reading 0.066 on `spam`; it wrote the `severity` comparison signs into the format string,
so "orders the set correctly" stayed true whatever the scores came back as; and it named
`harassment` and `threat` as the questions that separate the two hostile posts, while the
gaps measured on the same run were harassment 0.518 and spam 0.193, with threat third at
0.054.

The summary now derives its flag list and rubric ceiling from
`laya.moderation_questions()` and prints every probability it speaks about, so the names,
gaps and ranking follow the answers instead of the other way round.
@NandhaKishorM
NandhaKishorM merged commit 9de1fd0 into NandhaKishorM:main Oct 2, 2026
23 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.24.

Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, compileall clean, both TypeScript packages green (laya-ts 597 tests, the SDK 26), and a decision diff against 0.3.23 on the real english and multilingual checkpoints (74 answers over choice, score and noul; short and long states; budgeted calls and predict_long) showing zero decision flips and a max probability delta of 0.000000.

Thank you.

NandhaKishorM pushed a commit that referenced this pull request Oct 7, 2026
The page measures `total_ms` and `median_ms` from `perf_counter` / `timed()` on
the same 10-case loop, prints them, and then closed with:

    Ten routing decisions cost well under a millisecond in total, and the router
    still holds zero checkpoints.

Every clause in that sentence is a quantity the page already had and did not
read back: "Ten" was hardcoded while `len(CASES)` sat right there; "well under
a millisecond" asserted a threshold that goes stale on slower hardware; "holds
zero checkpoints" asserted residency rather than reading `r.loaded`. This is
the same defect class #850 fixed on example 28's preset page and #972 fixed on
example 40's Part C, and the fix is the same shape: two pure helpers turn the
measurement into a verdict and the paragraph interpolates the verdict and the
raw number.

Measured on the current MPS run: `0.415 ms total / 10 cases`, `router.loaded
= []` after every call, so the page prints "10 routing decisions cost under
1.0 ms in total (0.415 ms measured this run), and the router holds zero
checkpoints". Lengthen `CASES` or slow the machine and the sentence
rephrases rather than lies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants