Repository navigation
feat(eval): metamorphic label-renaming invariance for choice - #517
NandhaKishorM merged 1 commit into
Conversation
…ishorM#512) Extend research/eval/metamorphic.py with a deterministic rename_labels transformation: the option at each canonical position keeps its slot and description while its model-facing key becomes an opaque label (A, B, ... Z, then key_26, key_27, ...). Order and semantics are unchanged, so drift in the new label_rename group isolates lexical-label sensitivity (the failure mode of NandhaKishorM#156) from the position sensitivity already measured by option_order (NandhaKishorM#269). make_variants now emits baseline, one option permutation and one label rename per case; _report and evaluate_variants accept the new kind; the CLI config records label_renames_per_case. The generated canonical_to_transformed / transformed_to_canonical mapping is saved per variant so the comparison is auditable. No runtime or model behavior changes. Verified: - python -m unittest research.eval.test_metamorphic # 20 passed (17 before; adds order/description preservation, alphabet fallback, a stub scorer that exposes label sensitivity with position invariance as the control, and the position-scorer isolation check) - python -m unittest research.eval.test_laya_eval # 64 passed - python -m unittest research.eval.test_presentation_checks - ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 - python -m compileall -q laya/ tests/
|
While measuring #542 I ran a rename over a 58-label question and turned up a confound worth flagging: at 58 options the default At ≤ ~20 options nothing of substance is truncated, so |
|
Thanks @devloper961-maker, merged. This closes out #512. It ships in 0.3.21. |
Add an evaluation-side budget diagnostic under research/eval/. Each variant is rendered through the same tokenizer/build_sequence budget path the model input uses, and every baseline/transformed pair reports whether both sides still carry equivalent option information (clean/confounded/unknown, with named reasons), so budget-induced content drift is not read as lexical-label sensitivity. Unverifiable pairs -- renaming options that carry no description text, spans not attributable to a character prefix -- are reported unknown, never assumed clean. Additive fields only; runtime inference is untouched and a missing budget leaves reports unchanged. Refs #543 #517 #569 #647
What
Extends
research/eval/metamorphic.pywith a deterministic label-renaming transformation forchoice, as proposed in #512:rename_labels(case, alphabet=...): the option at each canonical position keeps its slot and description, while its model-facing key becomes an opaque label (A,B,C, ...Z, thenkey_26,key_27, ... viaopaque_labels). Option order is unchanged, so the transform isolates lexical-label sensitivity from the position sensitivity already measured byoption_order.make_variantsnow emits baseline + one option permutation + one label rename per case;_reportgains alabel_renamegroup and quality section;evaluate_variantsaccepts the new kind; the CLIconfigrecordslabel_renames_per_case: 1.canonical_to_transformed/transformed_to_canonicalmapping is stored per variant, so every comparison is auditable.No runtime or model behavior changes.
Why
#156 showed model-facing label tokens can dominate a decision even when descriptions and state are unchanged, and #163 added the
labelsoverride fornoul. Until now those behaviours were documented only through individual reproductions. This turns the question from #512 —— into a reproducible metric. Useful for checkpoint comparisons, fine-tune regression testing, and deciding whether model-facing labels are safe for a downstream workflow.
Kept narrow per the issue's guidance:
choiceonly, opaque labels, order preserved, existing canonical remapping / reporting / offline-test infrastructure reused, no pass/fail threshold.noulsupport via #163'slabelsoverride is left for a follow-up, as is combining renaming with permutation (deliberately avoided so disagreements stay attributable).How it was verified
New tests cover: order/description preservation under renaming, the deterministic alphabet fallback past 26 options, a stub scorer that exposes label sensitivity (winner moves, drift > 0.3) while remaining position-invariant, and a pure position scorer as the isolation control (rename group must show zero drift, since slots are untouched).
Follow-ups deliberately left out
noullabel renaming through fix(agent): allow custom noul labels #163'slabelsoverride (semantic order stays[false, true], soP(true)can be compared directly).