Repository navigation
feat(research): add a position-prior regression check for score questions (#131) - #259
Merged
NandhaKishorM merged 1 commit intoSep 23, 2026
Conversation
…ions (NandhaKishorM#131) laya-multilingual's score head has a learned prior against the first-listed level (NandhaKishorM#131), and the fix is a position-balanced retrain. This adds research/eval/presentation_checks.py, which says whether a retrained checkpoint removed that prior. It needs no labelled data: ten short English states are fixed in the file. Two checks, both on raw marker logits read through laya_eval.score_cases: * score_slot0_identical -- the identical-option control from @AlKor13 in NandhaKishorM#131. K levels with the same text (moderate / a request, K = 3, 4, 5), so slots differ only by position and the level-N prefix. Metric: slot-0 logit minus the slot mean, averaged over states and configurations. Gate >= -0.20. * score_first_slot_permuted -- the three real levels in all 6 orders per state. Each level sits in each slot exactly twice, so an order-invariant checkpoint picks the first slot in exactly 1/3 of the 60 decisions. Gate >= 0.15. The script first compares its logit path with Agent.system_one (max |dp| 4.98e-5, the 4-decimal rounding); above 1e-3 it exits 2 and reports nothing as a verdict. Measured (CPU, fp32, convaiinnovations/laya@1c5edc1, laya 0.3.7), metric and leave-one-out range over the 10 states: english slot0 +0.664 (+0.520 .. +0.741) first-slot 0.217 (0.204 .. 0.241) PASS multilingual slot0 -0.492 (-0.563 .. -0.425) first-slot 0.017 (0.000 .. 0.019) FAIL Thresholds: two conditions were fixed before the 10-state run. (1) The multilingual checkpoint fails and the english checkpoint passes in every leave-one-out subset. (2) Each checkpoint's worst leave-one-out value clears the threshold by at least 0.10 logit (slot 0) or 0.05 (first-slot rate). Margins: slot0 0.225 (multilingual) / 0.720 (english); first-slot 0.131 / 0.054. The english first-slot rate is the tightest. The gate is one-sided on purpose. With identical options the english checkpoint prefers early slots, more so as K grows (slot 0 at +0.10 for K=3 up to +1.68 for K=5), so a two-sided "no position effect" gate would fail it too. research/eval/test_presentation_checks.py adds 69 offline checks that need no checkpoint. They cover the fixed inputs and gates, the level-indexed rendering of the identical options, the permutation balance, the metric arithmetic, leave-one-out, the one-sided gates and the exit codes. Like test_laya_eval.py, it is not wired into CI. research/results/presentation_checks_shipped.json holds both runs. Nothing in laya/ changes, and there are no new dependencies. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Owner
|
Thank you, a label-free check that answers one question (did the retrain remove the |
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
…ions (NandhaKishorM#131) (NandhaKishorM#259) laya-multilingual's score head has a learned prior against the first-listed level (NandhaKishorM#131), and the fix is a position-balanced retrain. This adds research/eval/presentation_checks.py, which says whether a retrained checkpoint removed that prior. It needs no labelled data: ten short English states are fixed in the file. Two checks, both on raw marker logits read through laya_eval.score_cases: * score_slot0_identical -- the identical-option control from @AlKor13 in NandhaKishorM#131. K levels with the same text (moderate / a request, K = 3, 4, 5), so slots differ only by position and the level-N prefix. Metric: slot-0 logit minus the slot mean, averaged over states and configurations. Gate >= -0.20. * score_first_slot_permuted -- the three real levels in all 6 orders per state. Each level sits in each slot exactly twice, so an order-invariant checkpoint picks the first slot in exactly 1/3 of the 60 decisions. Gate >= 0.15. The script first compares its logit path with Agent.system_one (max |dp| 4.98e-5, the 4-decimal rounding); above 1e-3 it exits 2 and reports nothing as a verdict. Measured (CPU, fp32, convaiinnovations/laya@1c5edc1, laya 0.3.7), metric and leave-one-out range over the 10 states: english slot0 +0.664 (+0.520 .. +0.741) first-slot 0.217 (0.204 .. 0.241) PASS multilingual slot0 -0.492 (-0.563 .. -0.425) first-slot 0.017 (0.000 .. 0.019) FAIL Thresholds: two conditions were fixed before the 10-state run. (1) The multilingual checkpoint fails and the english checkpoint passes in every leave-one-out subset. (2) Each checkpoint's worst leave-one-out value clears the threshold by at least 0.10 logit (slot 0) or 0.05 (first-slot rate). Margins: slot0 0.225 (multilingual) / 0.720 (english); first-slot 0.131 / 0.054. The english first-slot rate is the tightest. The gate is one-sided on purpose. With identical options the english checkpoint prefers early slots, more so as K grows (slot 0 at +0.10 for K=3 up to +1.68 for K=5), so a two-sided "no position effect" gate would fail it too. research/eval/test_presentation_checks.py adds 69 offline checks that need no checkpoint. They cover the fixed inputs and gates, the level-indexed rendering of the identical options, the permutation balance, the metric arithmetic, leave-one-out, the one-sided gates and the exit codes. Like test_laya_eval.py, it is not wired into CI. research/results/presentation_checks_shipped.json holds both runs. Nothing in laya/ changes, and there are no new dependencies. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Sep 27, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#131 stays open until a position-balanced
laya-multilingualcheckpoint lands. This adds a check that answers one question about that checkpoint: did the retrain remove thescoreslot prior? It needs no labelled data, so it can run the day a retrain is ready:Exit
0= pass,1= a check failed,2= the harness disagrees withAgent.system_one.What it checks
Ten short English support messages are fixed in the file. There is no dataset and there are no labels.
score_slot0_identicalmoderate,a request; K = 3, 4, 5)>= -0.20score_first_slot_permutedNot urgent/Soon/Work is blockedin all 6 orders per state>= 0.15score_slot0_identicalis the identical-option control from @AlKor13 in #131, turned into a gate. With identical texts, the rendered options differ only by position and thelevel N:prefix.score_first_slot_permutedputs each level in each slot exactly twice per state. A checkpoint whose answer does not depend on the order therefore picks the first slot in exactly 1/3 of the decisions, whatever the states say.Both checks read raw marker logits through
laya_eval.score_cases(thebench_local.pyforward pass). The script first compares that path withAgent.system_oneon every state.Measured on the shipped checkpoints
CPU, fp32,
convaiinnovations/laya@1c5edc1, laya 0.3.7. Values in parentheses are the leave-one-out range over the 10 states.score_slot0_identicalscore_first_slot_permutedlaya(english)laya-multilingualParity with
Agent.system_one: max |Δp| 4.98e-5 / 4.92e-5, i.e. the 4-decimal rounding. Per-state and per-configuration numbers are inresearch/results/presentation_checks_shipped.json.Thresholds
Two conditions were fixed before the 10-state run:
score_slot0_identicalscore_first_slot_permutedThe tightest margin is the english first-slot rate.
Why the gate is one-sided
The english checkpoint is not flat under identical options either. It prefers the early slots, more as K grows: slot 0 sits at +0.10 / +0.41 / +0.85 for K = 3 / 4 / 5 with
moderate, and at +0.21 / +0.74 / +1.68 witha request. At K = 3 withmoderateit is close to flat, which matches the #131 control.A two-sided "no position effect" gate would fail the english checkpoint too. So the gate asks the narrower question #131 is about: whether slot 0 is suppressed.
Tests
research/eval/test_presentation_checks.pyadds 69 regression checks with scripted logits and no checkpoint. They cover:level i: <same text>A scripted slot-0 hole fails both checks, and an order-invariant model scores exactly 1/3.
ruffwith CI's rule set is clean, andpython -m compileall -q research/eval/passes. After rebasing onto43cde5aI re-ran both checkpoints: every number above is unchanged, and parity matches to the last digit.Like
test_laya_eval.py, the offline test is not wired into CI. Happy to register the offline test inci.ymlandrelease.ymlonce #184 settles how research tests are wired.Shape
research/eval/presentation_checks.py: the CLI plus pure check functions, registered inCHECKSresearch/eval/test_presentation_checks.py: the offline testsresearch/eval/README.md: a new section covering method, measurements, thresholds and limitsresearch/results/presentation_checks_shipped.json: both runs (4 KB)Nothing in
laya/changes, and there are no new dependencies.Limits
scoreonly, 10 short states.Agentuses autocast andscore_casesdoes not; the parity check reports that difference instead of hiding it.This check is deliberately narrow and specific to #131; if the metamorphic evaluation proposed in #244 takes shape under
research/eval/, it could move into that framework.🤖 Generated with Claude Code