Skip to content

feat(eval): report budget-confounded metamorphic comparisons - #2

Merged
Rukafuu merged 1 commit into
mainfrom
metamorphic-budget-diagnostics
Sep 28, 2026
Merged

Rukafuu merged 1 commit into
mainfrom
metamorphic-budget-diagnostics

Conversation

@Rukafuu

@Rukafuu Rukafuu commented Sep 28, 2026

Copy link
Copy Markdown
Owner

Evaluation-side follow-up to the upstream design issue NandhaKishorM#647 (related upstream threads: NandhaKishorM#244, NandhaKishorM#269, NandhaKishorM#512, NandhaKishorM#517, NandhaKishorM#543). Branch based on upstream main (9d95567); the diff is the four files below.

Summary

A metamorphic comparison (option permutation, label renaming) assumes both variants present the same option information. Laya's option rendering and token-budget path can break that assumption silently, so observed probability drift can be budget-induced content drift rather than lexical-label sensitivity. This patch renders every variant through the same tokenizer/build_sequence budget path the model input uses, and reports per comparison whether both sides still carry equivalent option information.

  • Per variant (budget): options, distinct_spans, tokens_per_option, instruction_tokens, span_classes, retained_text, retained_descriptions, description_present, truncated, tail_truncated.
  • Per pair (budget_comparison): both sides aligned back to canonical option order, budget_confounded (true / false / null) and named reasons -- clean, option_count, distinct_spans, instruction_tokens, tail_truncation, collision_partition, retained_description, unverified_retained_text, missing_semantic_description, measurement_error.
  • Per run (report["budget"]): clean / confounded / unknown counts and confounded_rate over verifiable pairs only (null when none are verifiable, so an all-unknown run cannot read as clean).

Two conditions are reported as unknown, never clean:

  • Renaming options that carry no description text: the label is the only semantic content the option had.
  • Spans not attributable to a character prefix by exactly one exact token witness: prefix token counts are not monotone, so the witness search is global over every character prefix; none/several witnesses, or an option above the documented length bound, leave the slot null.

The diagnostic is deterministic, defines no threshold, and never erases or overrides the observed model metrics.

Scope

  • Additive report fields only; runtime inference is untouched.
  • evaluate(..., budget=None) leaves reports byte-identical to earlier versions.
  • evaluate() probes only when passed an explicit budget=; the CLI and evaluate_variants() derive one from a real agent (agent.cfg).

Verification

  • python -m unittest research.eval.test_metamorphic -- Ran 38 tests, OK
  • python research/eval/test_laya_eval.py -- 64 passed, 0 failed, exit 0
  • python -m compileall -q laya/ tests/ research/eval/ -- clean
  • python -m ruff check --select=E9,F63,F7,F82,F401,F811 --line-length=120 laya/ research/eval/ -- clean
  • git diff --check -- clean

Add an evaluation-side budget diagnostic under research/eval/. Each variant
is rendered through the same tokenizer/build_sequence budget path the model
input uses, and every baseline/transformed pair reports whether both sides
still carry equivalent option information (clean/confounded/unknown, with
named reasons), so budget-induced content drift is not read as
lexical-label sensitivity. Unverifiable pairs -- renaming options that carry
no description text, spans not attributable to a character prefix -- are
reported unknown, never assumed clean. Additive fields only; runtime
inference is untouched and a missing budget leaves reports unchanged.

Refs NandhaKishorM#543 NandhaKishorM#517 NandhaKishorM#569 NandhaKishorM#647
@Rukafuu
Rukafuu merged commit 0e27125 into main Sep 28, 2026
19 of 20 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c47ece74de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

sep = self.tok.sep_token_id
# With an empty state the last two ids are the option-block and state separators;
# anything else at max_len means max_len truncated the option block itself.
tail_truncated = len(ids) == self.max_len and ids[-2:] != [sep, sep]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Distinguish a dropped final separator from option truncation

When the empty-state probe is exactly one token over max_len, build_sequence can drop only the final state separator while preserving every option token and the option-block separator. In that case the last two IDs are not [sep, sep], so this marks the option tail as truncated; the last span then includes the remaining separator and compare_budgets reports every otherwise equivalent pair as budget-confounded. Detect whether option content itself was cut rather than inferring that from the missing second separator.

Useful? React with 👍 / 👎.

Comment on lines +322 to +323
marker = label + ": "
start = len(marker) if rendered.startswith(marker) else len(rendered)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Sanitize labels before locating the description

For an option key containing the tokenizer's mask-token literal, build_sequence and line 433 replace that literal with a space in rendered, but this marker is built from the original unsanitized label. The prefix check therefore fails even with a roomy budget, _retained_description returns an empty string, and a label-renaming comparison falsely reports that the semantic description changed. Construct the marker from the same sanitized label representation used for rendered.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant