Skip to content

research(eval): add metamorphic option-permutation robustness checks - #269

Merged
NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
Rukafuu:feat/metamorphic-option-permutation
Sep 23, 2026
Merged

NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
Rukafuu:feat/metamorphic-option-permutation

Conversation

@Rukafuu

@Rukafuu Rukafuu commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

This PR addresses #244 by introducing metamorphic choice option-permutation robustness evaluation to the research/eval harness, following the maintainer's requested first scope.

Summary of Changes

  • Scope & Transformations: Focuses exclusively on choice option-order permutation without modifying model or runtime code in laya/.
  • Deterministic 1-Permutation Per Case: Each case produces one seeded non-identity option permutation (falling back to a single-slot rotation if a shuffle yields identity).
  • Canonical Remapping: Evaluated probability vectors are explicitly remapped back to the canonical (original) option ordering before computing argmax predictions, accuracy, ECE, or distribution metrics.
  • Robustness Metrics: Reports pair-weighted summary metrics under option_order:
    • semantic_agreement_rate
    • mean_probability_drift
    • max_probability_drift
    • mean_js_divergence
    • mean_confidence_drift
    • mean_absolute_confidence_drift
    • worst_confidence_increase_on_disagreement
  • Agreement vs. Stability: Disagreements (argmax shifts) and continuous probability drifts are reported separately. No universal pass/fail threshold is imposed.
  • Verification: Fully verified offline via unit tests (python -m unittest research.eval.test_metamorphic -v), existing harness tests (python research/eval/test_laya_eval.py), compilation checks (python -m compileall -q research/eval), and git diff --check. No real-checkpoint measurements are claimed as no real-checkpoint run was executed.
  • Deferred Transforms: Label renaming (neutral labels), paraphrases, structured-state permutations, score and noul perturbations are intentionally deferred.

@Rukafuu

Rukafuu commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Oopsie, let me update a couple thigs first UwU

@Rukafuu

Rukafuu commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Updated. I restored the latest upstream eval docs and relaxed the permutation case validation so None/blank option descriptions are accepted as expected. Tests still pass.

# Conflicts:
#	research/eval/README.md
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for turning #244 into a focused first step: option-permutation only, one seeded non-identity permutation per case, and explicit remapping back to canonical order. It stays inside research/eval/ and changes no runtime code. #259 added a section to the same README first, so I merged main into your branch and kept both sections. Merging.

@NandhaKishorM
NandhaKishorM merged commit b731f2e into NandhaKishorM:main Sep 23, 2026
@Rukafuu

Rukafuu commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Thank you! I really appreciate the guidance on scoping it down for the first pass.
Keeping the first step focused made the implementation much cleaner, and I’m glad it fits well into the existing eval harness.

I’ll keep the other metamorphic transforms for follow-up work rather than expanding this one retroactively. Thanks for the merge!

baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…andhaKishorM#269)

* feat(eval): add metamorphic option-permutation evaluation

* test(eval): cover option permutation invariance offline

* docs(eval): document option-order robustness evaluation

* fix(eval): preserve upstream docs and relax permutation criteria

---------

Co-authored-by: rukafuu <lucas@heon.com.br>
Co-authored-by: Nandakishor <48623612+NandhaKishorM@users.noreply.github.com>
NandhaKishorM pushed a commit that referenced this pull request Sep 27, 2026
Extend research/eval/metamorphic.py with a deterministic
rename_labels transformation: the option at each canonical position
keeps its slot and description while its model-facing key becomes an
opaque label (A, B, ... Z, then key_26, key_27, ...). Order and
semantics are unchanged, so drift in the new label_rename group
isolates lexical-label sensitivity (the failure mode of #156) from
the position sensitivity already measured by option_order (#269).

make_variants now emits baseline, one option permutation and one
label rename per case; _report and evaluate_variants accept the new
kind; the CLI config records label_renames_per_case. The generated
canonical_to_transformed / transformed_to_canonical mapping is saved
per variant so the comparison is auditable. No runtime or model
behavior changes.

Verified:
- python -m unittest research.eval.test_metamorphic   # 20 passed
  (17 before; adds order/description preservation, alphabet fallback,
  a stub scorer that exposes label sensitivity with position
  invariance as the control, and the position-scorer isolation check)
- python -m unittest research.eval.test_laya_eval     # 64 passed
- python -m unittest research.eval.test_presentation_checks
- ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
- python -m compileall -q laya/ tests/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants