Skip to content

research(eval): stability hook prototype for option reorder/rename (#635) - #926

Open
Charanraj-24 wants to merge 2 commits into
NandhaKishorM:mainfrom
Charanraj-24:reliability-hook
Open

Charanraj-24 wants to merge 2 commits into
NandhaKishorM:mainfrom
Charanraj-24:reliability-hook

Conversation

@Charanraj-24

@Charanraj-24 Charanraj-24 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

What

A prototype of #635 as a predict hook, with no core change, updated for #951 as suggested in #635. research/eval/stability.py adds:

  • StabilityHook: in on_predict_start it adds re-asked copies of each choice question as extra rows in the same forward pass, and in on_predict_end folds them into a reliability field on the original answer. Probes are chosen from the checkpoint's option_layout, read from ctx.agent: rename probes always (same slots, keys A/B/C assigned forwards, backwards and shuffled, up to 20 options per Measured: option-span collapse caps a 58-label question at 43/58, which is also why a rename variant is not information-neutral #543), and reorder probes (reversed and three shuffles through option_order) only on a sequential checkpoint. reliability["probes"] names the kinds that ran, by_probe reports each kind separately, and when no probe could run the signals are null, so a caller can tell an absence from a zero. Laya's answer is unchanged, and usage["input_tokens"] keeps the copies' cost.
  • python -m research.eval.stability: runs MASSIVE through the existing harness, plain and hooked per case, and reports AUROC plus feat(evals): selective-classification metrics (brier, aurc, selective accuracy) #854's AURC, Brier and selective accuracy for answer_confidence vs the stability signals, overall and per probe kind, with bootstrap intervals, and the measured cost.

Results

MASSIVE en, 300 cases, 20 options, laya 0.3.28, convaiinnovations/laya (sequential), CPU, 65 wrong answers:

Rename and reorder (8 rows) Rename only (4 rows)
AUROC, answer_confidence 0.831 0.831
AUROC, soft_stability 0.883 0.870
AUROC difference, bootstrap 95% +0.052 (+0.013 to +0.096) +0.039 (-0.005 to +0.089)
AURC, answer_confidence 0.084 0.084
AURC, soft_stability 0.064 0.066
AURC difference, bootstrap 95% -0.021 (-0.040 to -0.004) -0.018 (-0.038 to -0.001)
Confident errors flagged (of 37 at answer_confidence >= 0.90) 27 19

All 300 hooked answers match plain predict in both runs.

The rename-only column is the probe set a parallel checkpoint would get, measured on the sequential checkpoint. A checkpoint trained on the parallel layout has different weights, so its rename sensitivity could differ; no parallel checkpoint is published to measure it on.

Compared with the earlier version of this PR (AUROC 0.907), the rename probes no longer reorder the options as well, so rename and reorder effects are measured separately, the same separation as #517. That costs some signal on a sequential checkpoint but makes the rename-only number mean what it says.

Cost

CPU, 20 options Plain Rename and reorder Rename only
Rows per question 1 8 4
Input tokens per case 190 1,392 633
ms per case, predict 666 4,966 2,535
ms per state, predict_batch (64 states, batch 16) 665 5,380 2,520

So on CPU it's about 7-8x with both probe kinds and about 4x with renames only, and batching doesn't reduce it. GPU isn't measured.

On head_max_len: the original options collapse in 140 of 300 cases. The reorder copies collapse in exactly the same cases; the rename copies never do, because A/B/C keys are shorter.

How it was verified

python -m unittest research.eval.test_stability   # Ran 15 tests, OK
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m research.eval.stability --langs en --per-lang 300 --n-opts 20 --out stability_k20.json
python -m research.eval.stability --langs en --per-lang 300 --n-opts 20 --probes rename --out stability_k20_rename.json

The tests simulate the hook order in predict_batch and cover: probes chosen from the layout (rename and reorder on sequential, rename only on parallel, explicit override), null signals when no probe can run, rename probes keeping every option in its slot, a position-following model flagged by reorder probes only, a label-following model flagged by rename probes only (answers mapped back to the caller's keys), identical copies across states, untouched non-choice questions, the failure and skip paths, and usage cleanup.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Refs #635, #951, #361, #543, #854

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant