Skip to content

feat(research): add a position-prior regression check for score questions (#131) - #259

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
hiroki-abe-58:feat/score-position-check
Sep 23, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
hiroki-abe-58:feat/score-position-check

Conversation

@hiroki-abe-58

Copy link
Copy Markdown
Contributor

#131 stays open until a position-balanced laya-multilingual checkpoint lands. This adds a check that answers one question about that checkpoint: did the retrain remove the score slot prior? It needs no labelled data, so it can run the day a retrain is ready:

python research/eval/presentation_checks.py --model <retrained checkpoint> [--subfolder multilingual]

Exit 0 = pass, 1 = a check failed, 2 = the harness disagrees with Agent.system_one.

What it checks

Ten short English support messages are fixed in the file. There is no dataset and there are no labels.

check input metric gate
score_slot0_identical K levels with the same text (moderate, a request; K = 3, 4, 5) raw slot-0 marker logit minus the slot mean, averaged over 10 states × 6 configurations >= -0.20
score_first_slot_permuted Not urgent / Soon / Work is blocked in all 6 orders per state share of the 60 decisions whose argmax is the first slot >= 0.15

score_slot0_identical is the identical-option control from @AlKor13 in #131, turned into a gate. With identical texts, the rendered options differ only by position and the level N: prefix.

score_first_slot_permuted puts each level in each slot exactly twice per state. A checkpoint whose answer does not depend on the order therefore picks the first slot in exactly 1/3 of the decisions, whatever the states say.

Both checks read raw marker logits through laya_eval.score_cases (the bench_local.py forward pass). The script first compares that path with Agent.system_one on every state.

Measured on the shipped checkpoints

CPU, fp32, convaiinnovations/laya@1c5edc1, laya 0.3.7. Values in parentheses are the leave-one-out range over the 10 states.

checkpoint score_slot0_identical score_first_slot_permuted verdict
laya (english) +0.664 (+0.520 .. +0.741) 0.217 (0.204 .. 0.241) PASS
laya-multilingual −0.492 (−0.563 .. −0.425) 0.017 (0.000 .. 0.019) FAIL

Parity with Agent.system_one: max |Δp| 4.98e-5 / 4.92e-5, i.e. the 4-decimal rounding. Per-state and per-configuration numbers are in research/results/presentation_checks_shipped.json.

Thresholds

Two conditions were fixed before the 10-state run:

  1. multilingual fails and english passes in every leave-one-out subset;
  2. each checkpoint's worst leave-one-out value clears the gate by at least 0.10 logit (slot 0) or 0.05 (first-slot rate).
check gate multilingual worst → margin english worst → margin
score_slot0_identical −0.20 −0.425 → 0.225 +0.520 → 0.720
score_first_slot_permuted 0.15 0.019 → 0.131 0.204 → 0.054

The tightest margin is the english first-slot rate.

Why the gate is one-sided

The english checkpoint is not flat under identical options either. It prefers the early slots, more as K grows: slot 0 sits at +0.10 / +0.41 / +0.85 for K = 3 / 4 / 5 with moderate, and at +0.21 / +0.74 / +1.68 with a request. At K = 3 with moderate it is close to flat, which matches the #131 control.

A two-sided "no position effect" gate would fail the english checkpoint too. So the gate asks the narrower question #131 is about: whether slot 0 is suppressed.

Tests

research/eval/test_presentation_checks.py adds 69 regression checks with scripted logits and no checkpoint. They cover:

  • the fixed inputs and both gates
  • that identical options render as level i: <same text>
  • that every level sits in every slot exactly twice
  • the metric arithmetic by hand, the leave-one-out bounds and the one-sided gates
  • the exit codes

A scripted slot-0 hole fails both checks, and an order-invariant model scores exactly 1/3.

python research/eval/test_presentation_checks.py   ->  69 passed, 0 failed
python research/eval/test_laya_eval.py             ->  64 passed, 0 failed

ruff with CI's rule set is clean, and python -m compileall -q research/eval/ passes. After rebasing onto 43cde5a I re-ran both checkpoints: every number above is unchanged, and parity matches to the last digit.

Like test_laya_eval.py, the offline test is not wired into CI. Happy to register the offline test in ci.yml and release.yml once #184 settles how research tests are wired.

Shape

  • research/eval/presentation_checks.py: the CLI plus pure check functions, registered in CHECKS
  • research/eval/test_presentation_checks.py: the offline tests
  • research/eval/README.md: a new section covering method, measurements, thresholds and limits
  • research/results/presentation_checks_shipped.json: both runs (4 KB)

Nothing in laya/ changes, and there are no new dependencies.

Limits

  • Passing is not accuracy. A checkpoint can clear both gates and still rank urgency badly.
  • English only, score only, 10 short states.
  • Thresholds were set on CPU fp32. On CUDA, Agent uses autocast and score_cases does not; the parity check reports that difference instead of hiding it.

This check is deliberately narrow and specific to #131; if the metamorphic evaluation proposed in #244 takes shape under research/eval/, it could move into that framework.

🤖 Generated with Claude Code

…ions (NandhaKishorM#131)

laya-multilingual's score head has a learned prior against the first-listed
level (NandhaKishorM#131), and the fix is a position-balanced retrain. This adds
research/eval/presentation_checks.py, which says whether a retrained
checkpoint removed that prior. It needs no labelled data: ten short English
states are fixed in the file.

Two checks, both on raw marker logits read through laya_eval.score_cases:

* score_slot0_identical -- the identical-option control from @AlKor13 in
  NandhaKishorM#131. K levels with the same text (moderate / a request, K = 3, 4, 5), so
  slots differ only by position and the level-N prefix. Metric: slot-0 logit
  minus the slot mean, averaged over states and configurations. Gate >= -0.20.
* score_first_slot_permuted -- the three real levels in all 6 orders per
  state. Each level sits in each slot exactly twice, so an order-invariant
  checkpoint picks the first slot in exactly 1/3 of the 60 decisions.
  Gate >= 0.15.

The script first compares its logit path with Agent.system_one (max |dp|
4.98e-5, the 4-decimal rounding); above 1e-3 it exits 2 and reports nothing
as a verdict.

Measured (CPU, fp32, convaiinnovations/laya@1c5edc1, laya 0.3.7), metric and
leave-one-out range over the 10 states:

  english       slot0 +0.664 (+0.520 .. +0.741)  first-slot 0.217 (0.204 .. 0.241)  PASS
  multilingual  slot0 -0.492 (-0.563 .. -0.425)  first-slot 0.017 (0.000 .. 0.019)  FAIL

Thresholds: two conditions were fixed before the 10-state run. (1) The
multilingual checkpoint fails and the english checkpoint passes in every
leave-one-out subset. (2) Each checkpoint's worst leave-one-out value clears
the threshold by at least 0.10 logit (slot 0) or 0.05 (first-slot rate).
Margins: slot0 0.225 (multilingual) / 0.720 (english); first-slot 0.131 /
0.054. The english first-slot rate is the tightest.

The gate is one-sided on purpose. With identical options the english
checkpoint prefers early slots, more so as K grows (slot 0 at +0.10 for K=3
up to +1.68 for K=5), so a two-sided "no position effect" gate would fail it
too.

research/eval/test_presentation_checks.py adds 69 offline checks that need no
checkpoint. They cover the fixed inputs and gates, the level-indexed
rendering of the identical options, the permutation balance, the metric
arithmetic, leave-one-out, the one-sided gates and the exit codes. Like
test_laya_eval.py, it is not wired into CI.
research/results/presentation_checks_shipped.json holds both runs.
Nothing in laya/ changes, and there are no new dependencies.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, a label-free check that answers one question (did the retrain remove the score slot prior?) is exactly what #131 needs, and exit codes that separate a failed check from a harness disagreement make it easy to trust. Merging.

@NandhaKishorM
NandhaKishorM merged commit 42dbf0f into NandhaKishorM:main Sep 23, 2026
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…ions (NandhaKishorM#131) (NandhaKishorM#259)

laya-multilingual's score head has a learned prior against the first-listed
level (NandhaKishorM#131), and the fix is a position-balanced retrain. This adds
research/eval/presentation_checks.py, which says whether a retrained
checkpoint removed that prior. It needs no labelled data: ten short English
states are fixed in the file.

Two checks, both on raw marker logits read through laya_eval.score_cases:

* score_slot0_identical -- the identical-option control from @AlKor13 in
  NandhaKishorM#131. K levels with the same text (moderate / a request, K = 3, 4, 5), so
  slots differ only by position and the level-N prefix. Metric: slot-0 logit
  minus the slot mean, averaged over states and configurations. Gate >= -0.20.
* score_first_slot_permuted -- the three real levels in all 6 orders per
  state. Each level sits in each slot exactly twice, so an order-invariant
  checkpoint picks the first slot in exactly 1/3 of the 60 decisions.
  Gate >= 0.15.

The script first compares its logit path with Agent.system_one (max |dp|
4.98e-5, the 4-decimal rounding); above 1e-3 it exits 2 and reports nothing
as a verdict.

Measured (CPU, fp32, convaiinnovations/laya@1c5edc1, laya 0.3.7), metric and
leave-one-out range over the 10 states:

  english       slot0 +0.664 (+0.520 .. +0.741)  first-slot 0.217 (0.204 .. 0.241)  PASS
  multilingual  slot0 -0.492 (-0.563 .. -0.425)  first-slot 0.017 (0.000 .. 0.019)  FAIL

Thresholds: two conditions were fixed before the 10-state run. (1) The
multilingual checkpoint fails and the english checkpoint passes in every
leave-one-out subset. (2) Each checkpoint's worst leave-one-out value clears
the threshold by at least 0.10 logit (slot 0) or 0.05 (first-slot rate).
Margins: slot0 0.225 (multilingual) / 0.720 (english); first-slot 0.131 /
0.054. The english first-slot rate is the tightest.

The gate is one-sided on purpose. With identical options the english
checkpoint prefers early slots, more so as K grows (slot 0 at +0.10 for K=3
up to +1.68 for K=5), so a two-sided "no position effect" gate would fail it
too.

research/eval/test_presentation_checks.py adds 69 offline checks that need no
checkpoint. They cover the fixed inputs and gates, the level-indexed
rendering of the identical options, the permutation balance, the metric
arithmetic, leave-one-out, the one-sided gates and the exit codes. Like
test_laya_eval.py, it is not wired into CI.
research/results/presentation_checks_shipped.json holds both runs.
Nothing in laya/ changes, and there are no new dependencies.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@hiroki-abe-58
hiroki-abe-58 deleted the feat/score-position-check branch October 1, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants