Skip to content

feat(eval): metamorphic label-renaming invariance for choice - #517

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
devloper961-maker:feat/metamorphic-label-renaming
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
devloper961-maker:feat/metamorphic-label-renaming

Conversation

@devloper961-maker

Copy link
Copy Markdown
Contributor

What

Extends research/eval/metamorphic.py with a deterministic label-renaming transformation for choice, as proposed in #512:

  • New rename_labels(case, alphabet=...): the option at each canonical position keeps its slot and description, while its model-facing key becomes an opaque label (A, B, C, ... Z, then key_26, key_27, ... via opaque_labels). Option order is unchanged, so the transform isolates lexical-label sensitivity from the position sensitivity already measured by option_order.
  • make_variants now emits baseline + one option permutation + one label rename per case; _report gains a label_rename group and quality section; evaluate_variants accepts the new kind; the CLI config records label_renames_per_case: 1.
  • The generated canonical_to_transformed / transformed_to_canonical mapping is stored per variant, so every comparison is auditable.

No runtime or model behavior changes.

Why

#156 showed model-facing label tokens can dominate a decision even when descriptions and state are unchanged, and #163 added the labels override for noul. Until now those behaviours were documented only through individual reproductions. This turns the question from #512 —

If only the model-facing labels change, while option semantics and order stay fixed, does the canonical output distribution stay stable?

— into a reproducible metric. Useful for checkpoint comparisons, fine-tune regression testing, and deciding whether model-facing labels are safe for a downstream workflow.

Kept narrow per the issue's guidance: choice only, opaque labels, order preserved, existing canonical remapping / reporting / offline-test infrastructure reused, no pass/fail threshold. noul support via #163's labels override is left for a follow-up, as is combining renaming with permutation (deliberately avoided so disagreements stay attributable).

How it was verified

python -m unittest research.eval.test_metamorphic        # 20 passed (17 before)
python -m unittest research.eval.test_laya_eval          # 64 passed
python -m unittest research.eval.test_presentation_checks
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/

New tests cover: order/description preservation under renaming, the deterministic alphabet fallback past 26 options, a stub scorer that exposes label sensitivity (winner moves, drift > 0.3) while remaining position-invariant, and a pure position scorer as the isolation control (rename group must show zero drift, since slots are untouched).

Follow-ups deliberately left out

  • noul label renaming through fix(agent): allow custom noul labels #163's labels override (semantic order stays [false, true], so P(true) can be compared directly).
  • Combined rename + permutation transforms.
  • Paraphrased (semantic) label variants.

…ishorM#512)

Extend research/eval/metamorphic.py with a deterministic
rename_labels transformation: the option at each canonical position
keeps its slot and description while its model-facing key becomes an
opaque label (A, B, ... Z, then key_26, key_27, ...). Order and
semantics are unchanged, so drift in the new label_rename group
isolates lexical-label sensitivity (the failure mode of NandhaKishorM#156) from
the position sensitivity already measured by option_order (NandhaKishorM#269).

make_variants now emits baseline, one option permutation and one
label rename per case; _report and evaluate_variants accept the new
kind; the CLI config records label_renames_per_case. The generated
canonical_to_transformed / transformed_to_canonical mapping is saved
per variant so the comparison is auditable. No runtime or model
behavior changes.

Verified:
- python -m unittest research.eval.test_metamorphic   # 20 passed
  (17 before; adds order/description preservation, alphabet fallback,
  a stub scorer that exposes label sensitivity with position
  invariance as the control, and the position-scorer isolation check)
- python -m unittest research.eval.test_laya_eval     # 64 passed
- python -m unittest research.eval.test_presentation_checks
- ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
- python -m compileall -q laya/ tests/
@aashish254

Copy link
Copy Markdown
Contributor

While measuring #542 I ran a rename over a 58-label question and turned up a confound worth flagging: at 58 options the default head_max_len=192 leaves each option 4 tokens, so iot_hue_lightoff: iot hue lightoff reaches the head as iot_, and the rename changes which fragment survives rather than only its name (24/58 → 6/58 correct, 46/58 answers flip). Turns out this is the same boundary #538 is asking about, and I've put the numbers, the decoded spans and a repro in #543.

At ≤ ~20 options nothing of substance is truncated, so rename_labels is a clean lexical test there. Emitting the variant with head_max_len >= n * 48 + 16 would keep it clean at any size. Happy to send that as a commit here if it's useful, no rush either way.

@NandhaKishorM
NandhaKishorM merged commit 6d8f2fd into NandhaKishorM:main Sep 27, 2026
20 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @devloper961-maker, merged. This closes out #512. It ships in 0.3.21.

NandhaKishorM pushed a commit that referenced this pull request Sep 29, 2026
Add an evaluation-side budget diagnostic under research/eval/. Each variant
is rendered through the same tokenizer/build_sequence budget path the model
input uses, and every baseline/transformed pair reports whether both sides
still carry equivalent option information (clean/confounded/unknown, with
named reasons), so budget-induced content drift is not read as
lexical-label sensitivity. Unverifiable pairs -- renaming options that carry
no description text, spans not attributable to a character prefix -- are
reported unknown, never assumed clean. Additive fields only; runtime
inference is untouched and a missing budget leaves reports unchanged.

Refs #543 #517 #569 #647
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants