Repository navigation
feat(common): report options that collapse into the same token span (#538) - #569
Conversation
`build_sequence` caps every option at 48 tokens and re-caps all of them to
`max(4, (head_max_len - 16) // n)` once they overflow the head budget. Two
options that share a prefix can come out of that cut as the *same* token span:
the marker count still equals the option count, so the guard in
`Agent._encode_state` passes and the request is answered normally -- by a
question that can no longer name those options apart.
Measured on `mteb/amazon_massive_intent` (en), the first 58 labels as one
choice question, English checkpoint:
head_max_len=192 (default) 4 tokens/option 42 of 58 spans differ
head_max_len=384 6 tokens/option 53 of 58
so at the default budget no answer better than 42/58 is reachable, and nothing
in the response says so: a caller cannot tell a ceiling that is the budget's
from one that is the model's (NandhaKishorM#538, NandhaKishorM#543).
`build_sequence(..., return_stats=True)` now returns a third value with
`options`, `options_distinct` and `tokens_per_option`, and both agents publish
`usage["options"]` for the questions where the two counts differ:
usage {"input_tokens": 253, "output_tokens": 0,
"options": {"intent": {"total": 58, "distinct": 42,
"tokens_per_option": 4}}}
Additive and conditional: a question whose options all survive is absent, so a
request that collapses nothing is byte-identical to before, and the default
two-value return of `build_sequence` is unchanged.
Two details that decide whether the number means anything, both from NandhaKishorM#543's
thread: the count is taken from the capped option ids before assembly, because
re-slicing the finished sequence cannot close the last option's span (it runs
into the serialized state, which differs per request, so the last option always
looks distinguishable), and `total` is the number of options the question
defines rather than the number of markers that reached the sequence, or the
entry would read "42 of 58" about a request where 16 options were never in the
input at all.
`return_stats` is the same seam NandhaKishorM#181 adds for state truncation, named to match:
if that lands first this becomes three more keys in the dict it already returns.
Checks: `python tests/test_option_collapse.py` -- 23 passed (new, no weights,
wired into both CI lanes); `tests/test_hooks_api.py` 134 passed with the usage
contract pinned; `test_criteria` 139, `test_training` 61, `test_runtime_fixes`,
`test_truncation_direction` (spies forward the new keyword), `test_router`,
`test_hooks`, `test_load_errors`, `test_empty_questions`, `test_onnx_lang_parity`,
`test_serve` all pass; ruff and compileall clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks for the PR. Surfacing option-collapse in A few points:
Otherwise a clean, well-documented change. |
|
Thanks — all four points are fair, and two of them wanted numbers, so here they are. 1. Cost of computing the stats unconditionally. Measured on
Against a forward pass of ~30 ms on this box that is 0.004–0.01% of a It is deliberately unconditional rather than lazy, and that is the part worth arguing about: the numbers can only be taken from the capped option ids before they are assembled into the sequence. After assembly the last option's span cannot be closed — it runs on into the serialized state, which differs per request, so a lazily re-derived count reports the last option as distinguishable whatever it collided with. @aashish254 hit exactly that in #538's thread, and it is the difference between 20 and 21 at 25 options. Computing it later would mean either re-tokenizing the options or reporting a wrong number. 2. No re-derivation in 3 and 4. Agreed on both, and to confirm the second: One path your point 4 made me check, since nobody asked about it: and |
Add an evaluation-side budget diagnostic under research/eval/. Each variant is rendered through the same tokenizer/build_sequence budget path the model input uses, and every baseline/transformed pair reports whether both sides still carry equivalent option information (clean/confounded/unknown, with named reasons), so budget-induced content drift is not read as lexical-label sensitivity. Unverifiable pairs -- renaming options that carry no description text, spans not attributable to a character prefix -- are reported unknown, never assumed clean. Additive fields only; runtime inference is untouched and a missing budget leaves reports unchanged. Refs #543 #517 #569 #647
`build_sequence` clamps the state to whatever room the head leaves, and the clamp was silent: `system_one` returned a token count and nothing about what it had dropped. Callers could only estimate, and the estimate reported in #174 -- a fixed character threshold -- cannot be right, because the budget is in tokens and the room left for the state moves with `max_len`, `head_max_len`, the instruction and the rendered options. Measured with a real tokenizer, truncation begins at 2,135 characters on a 512-token window and at 4,455 on a 1,024-token window, so one threshold is wrong on the other checkpoint in both directions: silent while evidence is being dropped, or announcing a loss that did not happen. It also varies per question against one state on one checkpoint -- between 2,294 and 2,480 characters a wide `choice` truncates while a narrow `noul` does not. The missing signal is not cosmetic. End-to-end on the shipped `english` checkpoint, one `noul` question over a state whose answering sentence sits at the end returns 0.874 when the state fits and 0.147 once right-truncation drops that sentence -- confidently inverted, with `usage` reporting only `input_tokens: 512` and no indication that anything was lost. `build_sequence(..., return_truncation_stats=True)` now returns one more value reporting `state_tokens`, `state_tokens_used`, `state_tokens_dropped` and `truncated`, after the option stats when `return_stats=True` is also set. The counts come from the state slice itself: `room` already leaves space for the closing `[SEP]`, so the final `[:max_len]` clamp never cuts into the state. The stats are opt-in, so the existing two-value callers in `research/scripts/bench_local.py` and the fine-tuning notebook are unaffected. [Som Tripathi: rewritten on main 9d95567, where #569 had taken return_stats; the usage wiring is in the next commit. The measurements above are from #181's original base.] Fixes #174
#181 built the truncation counts inside system_one. Since then main tokenizes the state once per call (#343), system_one delegates to predict_batch, and #569 took build_sequence's return_stats flag for the option-collapse stats. The previous commit therefore reports the counts through a second flag, return_truncation_stats. Here _encode_state asks for both and keeps the new counts as item["state_stats"] next to item["options"], and predict_batch writes them into each state's usage. The empty-questions usage is unchanged. test_hooks_api.py checks the new parameter's default. The tests follow the new item shape: test_truncation's stub tokenizer accepts main's truncation/max_length option cap and its source check reads _encode_state and predict_batch, and the _encode_state stubs in test_hooks, test_batch and test_predict_long carry state_stats. DecisionResult.usage is annotated Dict[str, Any], since truncated_questions is a list. The previous commit alone fails test_truncation.py; this one completes it. On the English checkpoint, answers and input_tokens match main at 11 state lengths (55 to 6,175 chars). truncated flips between 2,656 and 2,809 chars, where P(refund) for a state ending in the refund request drops from 0.9504 to 0.0000.
Closes #538, and reports the number #543 measured.
What
build_sequencecaps every option at 48 tokens, then re-caps all of them tomax(4, (head_max_len - 16) // n)once they overflow the head budget (laya/common.py:127-131). Two options that share a prefix can come out of that cut as the same token span. The marker count still equals the option count, so the guard inAgent._encode_statepasses, and the request is answered normally — by a question that can no longer name those options apart.mteb/amazon_massive_intent(en), the first 58 labels as onechoicequestion, English checkpoint:head_max_lenNine options present the identical window
iot_, fiveqa_, threeaudio_volume, twotakeaway_. @aashish254 measured 41.4% actual accuracy against that 72% ceiling in #543 — and a caller cannot tell the budget's ceiling from the model's, because nothing in the response mentions it.What this adds
build_sequence(..., return_stats=True)returns a third value —options,options_distinct,tokens_per_option— and both agents publishusage["options"]for the questions where the first two differ:Additive and conditional, as @Bruce-Yii and @aashish254 both preferred in #538: a question whose options all survive is not listed, a request that collapses nothing is byte-identical to before, and
build_sequence's default two-value return is unchanged.ONNXAgentreports the same thing from the same helper.Two details that decide whether the number means anything
Both came out of #538's thread, and both are wrong in the obvious implementation:
totalis what the question defines, not how many markers reached the sequence. Markers pastmax_lenare dropped (common.py:146), so a count taken from the marker list would report "42 of 58" about a request where 16 options were never in the input at all.Relationship to #181
return_statsis deliberately the same seam and the same keyword #181 adds for state truncation, and the keys are prefixed so the two families never collide (options*here,state*/truncated*there). Whichever lands first, the other becomes a few more keys in the dict it already returns; happy to rebase in either order.Checks
tests/test_option_collapse.pyis wired into both CI lanes. The three spies intest_truncation_direction.pythat stand in forbuild_sequencenow forward**kwargs, so they keep the real function's return shape; nothing else in the suite changed.Verified end-to-end on the real checkpoint (RTX 4070 SUPER): the four
usageblocks quoted above are actual output, including the mixed case where a 2-option and a 58-option question are asked together and only the second is listed.