Skip to content

feat(common): report options that collapse into the same token span (#538) - #569

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
AlKor13:feat/report-option-collapse
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
AlKor13:feat/report-option-collapse

Conversation

@AlKor13

@AlKor13 AlKor13 commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

Closes #538, and reports the number #543 measured.

What

build_sequence caps every option at 48 tokens, then re-caps all of them to max(4, (head_max_len - 16) // n) once they overflow the head budget (laya/common.py:127-131). Two options that share a prefix can come out of that cut as the same token span. The marker count still equals the option count, so the guard in Agent._encode_state passes, and the request is answered normally — by a question that can no longer name those options apart.

mteb/amazon_massive_intent (en), the first 58 labels as one choice question, English checkpoint:

head_max_len tokens/option distinct spans best reachable
192 (default) 4 42 of 58 72.4%
384 6 53 of 58 91.4%

Nine options present the identical window iot_, five qa_, three audio_volume, two takeaway_. @aashish254 measured 41.4% actual accuracy against that 72% ceiling in #543 — and a caller cannot tell the budget's ceiling from the model's, because nothing in the response mentions it.

What this adds

build_sequence(..., return_stats=True) returns a third value — options, options_distinct, tokens_per_option — and both agents publish usage["options"] for the questions where the first two differ:

agent.system_one("wake me up at five am this week", {"intent": <58 labels>})["usage"]
# {"input_tokens": 253, "output_tokens": 0,
#  "options": {"intent": {"total": 58, "distinct": 42, "tokens_per_option": 4}}}

agent.system_one(..., head_max_len=384)["usage"]["options"]["intent"]
# {"total": 58, "distinct": 53, "tokens_per_option": 6}

agent.system_one("...", {"dept": <2 labels>})["usage"]
# {"input_tokens": 28, "output_tokens": 0}      <- absent, nothing collapsed

Additive and conditional, as @Bruce-Yii and @aashish254 both preferred in #538: a question whose options all survive is not listed, a request that collapses nothing is byte-identical to before, and build_sequence's default two-value return is unchanged. ONNXAgent reports the same thing from the same helper.

Two details that decide whether the number means anything

Both came out of #538's thread, and both are wrong in the obvious implementation:

  • Counted on the capped option ids, before assembly. Re-slicing the finished sequence cannot close the last option's span — it runs on into the serialized state, which differs per request, so the last option always looks distinguishable however it collided. That alone is the difference between 20 and 21 at 25 options.
  • total is what the question defines, not how many markers reached the sequence. Markers past max_len are dropped (common.py:146), so a count taken from the marker list would report "42 of 58" about a request where 16 options were never in the input at all.

Relationship to #181

return_stats is deliberately the same seam and the same keyword #181 adds for state truncation, and the keys are prefixed so the two families never collide (options* here, state*/truncated* there). Whichever lands first, the other becomes a few more keys in the dict it already returns; happy to rebase in either order.

Checks

python tests/test_option_collapse.py     # 23 passed -- new, no weights, no network
python tests/test_hooks_api.py           # 134 passed -- usage contract pinned
python tests/test_criteria.py            # 139 passed
python tests/test_training.py            # 61 passed
python tests/test_runtime_fixes.py tests/test_router.py tests/test_hooks.py
python tests/test_load_errors.py tests/test_empty_questions.py tests/test_onnx_lang_parity.py
python -m pytest tests/test_truncation_direction.py tests/test_audit_regressions.py tests/test_serve.py
python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/

tests/test_option_collapse.py is wired into both CI lanes. The three spies in test_truncation_direction.py that stand in for build_sequence now forward **kwargs, so they keep the real function's return shape; nothing else in the suite changed.

Verified end-to-end on the real checkpoint (RTX 4070 SUPER): the four usage blocks quoted above are actual output, including the mixed case where a 2-option and a 58-option question are asked together and only the second is listed.

`build_sequence` caps every option at 48 tokens and re-caps all of them to
`max(4, (head_max_len - 16) // n)` once they overflow the head budget. Two
options that share a prefix can come out of that cut as the *same* token span:
the marker count still equals the option count, so the guard in
`Agent._encode_state` passes and the request is answered normally -- by a
question that can no longer name those options apart.

Measured on `mteb/amazon_massive_intent` (en), the first 58 labels as one
choice question, English checkpoint:

    head_max_len=192 (default)  4 tokens/option  42 of 58 spans differ
    head_max_len=384            6 tokens/option  53 of 58

so at the default budget no answer better than 42/58 is reachable, and nothing
in the response says so: a caller cannot tell a ceiling that is the budget's
from one that is the model's (NandhaKishorM#538, NandhaKishorM#543).

`build_sequence(..., return_stats=True)` now returns a third value with
`options`, `options_distinct` and `tokens_per_option`, and both agents publish
`usage["options"]` for the questions where the two counts differ:

    usage {"input_tokens": 253, "output_tokens": 0,
           "options": {"intent": {"total": 58, "distinct": 42,
                                  "tokens_per_option": 4}}}

Additive and conditional: a question whose options all survive is absent, so a
request that collapses nothing is byte-identical to before, and the default
two-value return of `build_sequence` is unchanged.

Two details that decide whether the number means anything, both from NandhaKishorM#543's
thread: the count is taken from the capped option ids before assembly, because
re-slicing the finished sequence cannot close the last option's span (it runs
into the serialized state, which differs per request, so the last option always
looks distinguishable), and `total` is the number of options the question
defines rather than the number of markers that reached the sequence, or the
entry would read "42 of 58" about a request where 16 options were never in the
input at all.

`return_stats` is the same seam NandhaKishorM#181 adds for state truncation, named to match:
if that lands first this becomes three more keys in the dict it already returns.

Checks: `python tests/test_option_collapse.py` -- 23 passed (new, no weights,
wired into both CI lanes); `tests/test_hooks_api.py` 134 passed with the usage
contract pinned; `test_criteria` 139, `test_training` 61, `test_runtime_fixes`,
`test_truncation_direction` (spies forward the new keyword), `test_router`,
`test_hooks`, `test_load_errors`, `test_empty_questions`, `test_onnx_lang_parity`,
`test_serve` all pass; ruff and compileall clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rahul05ranjan

Copy link
Copy Markdown
Contributor

Thanks for the PR. Surfacing option-collapse in usage is a useful observability addition — a caller choosing among 42 distinguishable spans of 58 has a ceiling they can't otherwise see.

A few points:

  1. return_stats default: build_sequence(..., return_stats=False) adds a new kwarg with a safe default, so existing callers are unaffected. Good. But the stats returned is now always computed in _encode_state (via return_stats=True) even when no collapse happens — worth confirming the stats computation isn't a meaningful per-question cost on the hot path.

  2. collapsed_options(ids, items): the items now carry an "options": stats entry. The collapsed_options function reads back from items — is stats the right shape for it, or is there a redundant re-derivation? Worth a quick look at whether stats could be computed once and reused rather than recomputed in collapsed_options.

  3. Conditional key: only adding usage["options"] when collapse actually happened is the right call — an always-present key would be noise. The docstring explains this well.

  4. predict_batch vs system_one: the change is in predict_batch, and system_one delegates to it, so both paths get it. Good. But the usage dict is now built unconditionally (usage = {...} then conditionally extended) — the output_tokens: 0 is still there, so the shape is backward-compatible.

Otherwise a clean, well-documented change.

@AlKor13

AlKor13 commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — all four points are fair, and two of them wanted numbers, so here they are.

1. Cost of computing the stats unconditionally. Measured on build_sequence itself, real ModernBERT tokenizer, median of 400 calls each:

question return_stats=False return_stats=True delta
3 options, short state 121.7 µs 123.0 µs +1.3 µs (+1.07%)
58 options (the collapsing case) 1572.5 µs 1575.4 µs +2.9 µs (+0.18%)

Against a forward pass of ~30 ms on this box that is 0.004–0.01% of a predict call. The work is one pass over the already-built opt_ids to make a set of tuples, so it scales with the option count and is dwarfed by the tokenization that precedes it.

It is deliberately unconditional rather than lazy, and that is the part worth arguing about: the numbers can only be taken from the capped option ids before they are assembled into the sequence. After assembly the last option's span cannot be closed — it runs on into the serialized state, which differs per request, so a lazily re-derived count reports the last option as distinguishable whatever it collided with. @aashish254 hit exactly that in #538's thread, and it is the difference between 20 and 21 at 25 options. Computing it later would mean either re-tokenizing the options or reporting a wrong number.

2. No re-derivation in collapsed_options. The stats are computed once, inside build_sequence, and travel on the item (items.append({..., "options": stats})). collapsed_options only filters the questions where options_distinct < options and renames the three keys into the public shape (total/distinct/tokens_per_option), so the per-question cost outside the tokenizer is one dict comparison and, for a collapsing question only, one small dict. The rename is there so the internal names can change without breaking the response shape; happy to drop it and publish the internal keys directly if you would rather have one name for each thing.

3 and 4. Agreed on both, and to confirm the second: system_one delegates to predict_batch, so both paths report it, ONNXAgent reports the same thing through the same helper, and input_tokens/output_tokens are still unconditional — tests/test_hooks_api.py pins that.

One path your point 4 made me check, since nobody asked about it: predict_long aggregates usage across windows by summing numeric fields and taking the last value for anything else, so the options dict survives that unchanged (the same questions are asked in every window, so every window agrees). Verified on the real checkpoint:

predict_long     {"input_tokens": 8192, "output_tokens": 0,
                  "options": {"intent": {"total": 58, "distinct": 42, "tokens_per_option": 4}},
                  "windows": 16}
predict_batch[0] {"input_tokens": 250, "output_tokens": 0,
                  "options": {"intent": {"total": 58, "distinct": 42, "tokens_per_option": 4}}}

and aggregate_usage in laya/hooks.py reads only the two token fields, so a hook's ctx.usage is unaffected.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @AlKor13, merged. #489 had moved ONNX encoding into _encode_state / _infer_batch, so I carried usage["options"] over to the batch path there. It ships in 0.3.21.

NandhaKishorM pushed a commit that referenced this pull request Sep 29, 2026
Add an evaluation-side budget diagnostic under research/eval/. Each variant
is rendered through the same tokenizer/build_sequence budget path the model
input uses, and every baseline/transformed pair reports whether both sides
still carry equivalent option information (clean/confounded/unknown, with
named reasons), so budget-induced content drift is not read as
lexical-label sensitivity. Unverifiable pairs -- renaming options that carry
no description text, spans not attributable to a character prefix -- are
reported unknown, never assumed clean. Additive fields only; runtime
inference is untouched and a missing budget leaves reports unchanged.

Refs #543 #517 #569 #647
NandhaKishorM pushed a commit that referenced this pull request Sep 29, 2026
`build_sequence` clamps the state to whatever room the head leaves, and the
clamp was silent: `system_one` returned a token count and nothing about what
it had dropped. Callers could only estimate, and the estimate reported in #174
-- a fixed character threshold -- cannot be right, because the budget is in
tokens and the room left for the state moves with `max_len`, `head_max_len`,
the instruction and the rendered options.

Measured with a real tokenizer, truncation begins at 2,135 characters on a
512-token window and at 4,455 on a 1,024-token window, so one threshold is
wrong on the other checkpoint in both directions: silent while evidence is
being dropped, or announcing a loss that did not happen. It also varies per
question against one state on one checkpoint -- between 2,294 and 2,480
characters a wide `choice` truncates while a narrow `noul` does not.

The missing signal is not cosmetic. End-to-end on the shipped `english`
checkpoint, one `noul` question over a state whose answering sentence sits at
the end returns 0.874 when the state fits and 0.147 once right-truncation
drops that sentence -- confidently inverted, with `usage` reporting only
`input_tokens: 512` and no indication that anything was lost.

`build_sequence(..., return_truncation_stats=True)` now returns one more value
reporting `state_tokens`, `state_tokens_used`, `state_tokens_dropped` and
`truncated`, after the option stats when `return_stats=True` is also set. The
counts come from the state slice itself: `room` already leaves space for the
closing `[SEP]`, so the final `[:max_len]` clamp never cuts into the state.

The stats are opt-in, so the existing two-value callers in
`research/scripts/bench_local.py` and the fine-tuning notebook are unaffected.

[Som Tripathi: rewritten on main 9d95567, where #569 had taken return_stats;
the usage wiring is in the next commit. The measurements above are from
#181's original base.]

Fixes #174
NandhaKishorM pushed a commit that referenced this pull request Sep 29, 2026
#181 built the truncation counts inside system_one. Since then main
tokenizes the state once per call (#343), system_one delegates to
predict_batch, and #569 took build_sequence's return_stats flag for
the option-collapse stats. The previous commit therefore reports the
counts through a second flag, return_truncation_stats. Here
_encode_state asks for both and keeps the new counts as
item["state_stats"] next to item["options"], and predict_batch writes
them into each state's usage. The empty-questions usage is unchanged.

test_hooks_api.py checks the new parameter's default. The tests follow
the new item shape: test_truncation's stub tokenizer accepts main's
truncation/max_length option cap and its source check reads
_encode_state and predict_batch, and the _encode_state stubs in
test_hooks, test_batch and test_predict_long carry state_stats.
DecisionResult.usage is annotated Dict[str, Any], since
truncated_questions is a list.

The previous commit alone fails test_truncation.py; this one completes it.

On the English checkpoint, answers and input_tokens match main at 11
state lengths (55 to 6,175 chars). truncated flips between 2,656 and
2,809 chars, where P(refund) for a state ending in the refund request
drops from 0.9504 to 0.0000.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Should option-token identity collapse after head truncation be reported?

3 participants