Repository navigation
research(eval): stability hook prototype for option reorder/rename (#635) - #926
Open
Charanraj-24 wants to merge 2 commits into
Open
Charanraj-24 wants to merge 2 commits into
Charanraj-24 wants to merge 2 commits into
Conversation
1 task done
Charanraj-24
force-pushed
the
reliability-hook
branch
from
October 5, 2026 20:02
10c9150 to
b1a753d
Compare
This was referenced Oct 17, 2026
This was referenced Oct 19, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A prototype of #635 as a predict hook, with no core change, updated for #951 as suggested in #635.
research/eval/stability.pyadds:StabilityHook: inon_predict_startit adds re-asked copies of eachchoicequestion as extra rows in the same forward pass, and inon_predict_endfolds them into areliabilityfield on the original answer. Probes are chosen from the checkpoint'soption_layout, read fromctx.agent: rename probes always (same slots, keys A/B/C assigned forwards, backwards and shuffled, up to 20 options per Measured: option-span collapse caps a 58-label question at 43/58, which is also why a rename variant is not information-neutral #543), and reorder probes (reversed and three shuffles throughoption_order) only on a sequential checkpoint.reliability["probes"]names the kinds that ran,by_probereports each kind separately, and when no probe could run the signals arenull, so a caller can tell an absence from a zero. Laya's answer is unchanged, andusage["input_tokens"]keeps the copies' cost.python -m research.eval.stability: runs MASSIVE through the existing harness, plain and hooked per case, and reports AUROC plus feat(evals): selective-classification metrics (brier, aurc, selective accuracy) #854's AURC, Brier and selective accuracy foranswer_confidencevs the stability signals, overall and per probe kind, with bootstrap intervals, and the measured cost.Results
MASSIVE
en, 300 cases, 20 options, laya 0.3.28,convaiinnovations/laya(sequential), CPU, 65 wrong answers:answer_confidencesoft_stabilityanswer_confidencesoft_stabilityanswer_confidence >= 0.90)All 300 hooked answers match plain
predictin both runs.The rename-only column is the probe set a parallel checkpoint would get, measured on the sequential checkpoint. A checkpoint trained on the parallel layout has different weights, so its rename sensitivity could differ; no parallel checkpoint is published to measure it on.
Compared with the earlier version of this PR (AUROC 0.907), the rename probes no longer reorder the options as well, so rename and reorder effects are measured separately, the same separation as #517. That costs some signal on a sequential checkpoint but makes the rename-only number mean what it says.
Cost
predictpredict_batch(64 states, batch 16)So on CPU it's about 7-8x with both probe kinds and about 4x with renames only, and batching doesn't reduce it. GPU isn't measured.
On
head_max_len: the original options collapse in 140 of 300 cases. The reorder copies collapse in exactly the same cases; the rename copies never do, because A/B/C keys are shorter.How it was verified
The tests simulate the hook order in
predict_batchand cover: probes chosen from the layout (rename and reorder on sequential, rename only on parallel, explicit override), null signals when no probe can run, rename probes keeping every option in its slot, a position-following model flagged by reorder probes only, a label-following model flagged by rename probes only (answers mapped back to the caller's keys), identical copies across states, untouched non-choice questions, the failure and skip paths, and usage cleanup.Checklist
mainRefs #635, #951, #361, #543, #854