Repository navigation
feat(eval): report budget-confounded metamorphic comparisons - #2
Conversation
Add an evaluation-side budget diagnostic under research/eval/. Each variant is rendered through the same tokenizer/build_sequence budget path the model input uses, and every baseline/transformed pair reports whether both sides still carry equivalent option information (clean/confounded/unknown, with named reasons), so budget-induced content drift is not read as lexical-label sensitivity. Unverifiable pairs -- renaming options that carry no description text, spans not attributable to a character prefix -- are reported unknown, never assumed clean. Additive fields only; runtime inference is untouched and a missing budget leaves reports unchanged. Refs NandhaKishorM#543 NandhaKishorM#517 NandhaKishorM#569 NandhaKishorM#647
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c47ece74de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sep = self.tok.sep_token_id | ||
| # With an empty state the last two ids are the option-block and state separators; | ||
| # anything else at max_len means max_len truncated the option block itself. | ||
| tail_truncated = len(ids) == self.max_len and ids[-2:] != [sep, sep] |
There was a problem hiding this comment.
Distinguish a dropped final separator from option truncation
When the empty-state probe is exactly one token over max_len, build_sequence can drop only the final state separator while preserving every option token and the option-block separator. In that case the last two IDs are not [sep, sep], so this marks the option tail as truncated; the last span then includes the remaining separator and compare_budgets reports every otherwise equivalent pair as budget-confounded. Detect whether option content itself was cut rather than inferring that from the missing second separator.
Useful? React with 👍 / 👎.
| marker = label + ": " | ||
| start = len(marker) if rendered.startswith(marker) else len(rendered) |
There was a problem hiding this comment.
Sanitize labels before locating the description
For an option key containing the tokenizer's mask-token literal, build_sequence and line 433 replace that literal with a space in rendered, but this marker is built from the original unsanitized label. The prefix check therefore fails even with a roomy budget, _retained_description returns an empty string, and a label-renaming comparison falsely reports that the semantic description changed. Construct the marker from the same sanitized label representation used for rendered.
Useful? React with 👍 / 👎.
Evaluation-side follow-up to the upstream design issue NandhaKishorM#647 (related upstream threads: NandhaKishorM#244, NandhaKishorM#269, NandhaKishorM#512, NandhaKishorM#517, NandhaKishorM#543). Branch based on upstream
main(9d95567); the diff is the four files below.Summary
A metamorphic comparison (option permutation, label renaming) assumes both variants present the same option information. Laya's option rendering and token-budget path can break that assumption silently, so observed probability drift can be budget-induced content drift rather than lexical-label sensitivity. This patch renders every variant through the same tokenizer/
build_sequencebudget path the model input uses, and reports per comparison whether both sides still carry equivalent option information.budget):options,distinct_spans,tokens_per_option,instruction_tokens,span_classes,retained_text,retained_descriptions,description_present,truncated,tail_truncated.budget_comparison): both sides aligned back to canonical option order,budget_confounded(true/false/null) and namedreasons--clean,option_count,distinct_spans,instruction_tokens,tail_truncation,collision_partition,retained_description,unverified_retained_text,missing_semantic_description,measurement_error.report["budget"]):clean/confounded/unknowncounts andconfounded_rateover verifiable pairs only (nullwhen none are verifiable, so an all-unknownrun cannot read as clean).Two conditions are reported as
unknown, never clean:null.The diagnostic is deterministic, defines no threshold, and never erases or overrides the observed model metrics.
Scope
evaluate(..., budget=None)leaves reports byte-identical to earlier versions.evaluate()probes only when passed an explicitbudget=; the CLI andevaluate_variants()derive one from a real agent (agent.cfg).Verification
python -m unittest research.eval.test_metamorphic--Ran 38 tests, OKpython research/eval/test_laya_eval.py--64 passed, 0 failed, exit 0python -m compileall -q laya/ tests/ research/eval/-- cleanpython -m ruff check --select=E9,F63,F7,F82,F401,F811 --line-length=120 laya/ research/eval/-- cleangit diff --check-- clean