Repository navigation
research(eval): add metamorphic option-permutation robustness checks - #269
NandhaKishorM merged 5 commits into
Conversation
|
Oopsie, let me update a couple thigs first UwU |
|
Updated. I restored the latest upstream eval docs and relaxed the permutation case validation so None/blank option descriptions are accepted as expected. Tests still pass. |
# Conflicts: # research/eval/README.md
|
Thank you for turning #244 into a focused first step: option-permutation only, one seeded non-identity permutation per case, and explicit remapping back to canonical order. It stays inside |
|
Thank you! I really appreciate the guidance on scoping it down for the first pass. I’ll keep the other metamorphic transforms for follow-up work rather than expanding this one retroactively. Thanks for the merge! |
…andhaKishorM#269) * feat(eval): add metamorphic option-permutation evaluation * test(eval): cover option permutation invariance offline * docs(eval): document option-order robustness evaluation * fix(eval): preserve upstream docs and relax permutation criteria --------- Co-authored-by: rukafuu <lucas@heon.com.br> Co-authored-by: Nandakishor <48623612+NandhaKishorM@users.noreply.github.com>
Extend research/eval/metamorphic.py with a deterministic rename_labels transformation: the option at each canonical position keeps its slot and description while its model-facing key becomes an opaque label (A, B, ... Z, then key_26, key_27, ...). Order and semantics are unchanged, so drift in the new label_rename group isolates lexical-label sensitivity (the failure mode of #156) from the position sensitivity already measured by option_order (#269). make_variants now emits baseline, one option permutation and one label rename per case; _report and evaluate_variants accept the new kind; the CLI config records label_renames_per_case. The generated canonical_to_transformed / transformed_to_canonical mapping is saved per variant so the comparison is auditable. No runtime or model behavior changes. Verified: - python -m unittest research.eval.test_metamorphic # 20 passed (17 before; adds order/description preservation, alphabet fallback, a stub scorer that exposes label sensitivity with position invariance as the control, and the position-scorer isolation check) - python -m unittest research.eval.test_laya_eval # 64 passed - python -m unittest research.eval.test_presentation_checks - ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 - python -m compileall -q laya/ tests/
This PR addresses #244 by introducing metamorphic choice option-permutation robustness evaluation to the
research/evalharness, following the maintainer's requested first scope.Summary of Changes
laya/.option_order:semantic_agreement_ratemean_probability_driftmax_probability_driftmean_js_divergencemean_confidence_driftmean_absolute_confidence_driftworst_confidence_increase_on_disagreementpython -m unittest research.eval.test_metamorphic -v), existing harness tests (python research/eval/test_laya_eval.py), compilation checks (python -m compileall -q research/eval), andgit diff --check. No real-checkpoint measurements are claimed as no real-checkpoint run was executed.scoreandnoulperturbations are intentionally deferred.