Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions research/eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,25 @@ Metrics are grouped under `option_order`, `label_rename` and `overall`:

`overall` is pair-weighted, not a fraction of cases where *all* variants agree. `quality` separately reports accuracy and the existing harness's 15-bin ECE for baseline and each transformation on labelled cases only. Empty groups contain `n: 0`; unlabelled quality groups contain `n_labelled: 0` without inventing an accuracy or ECE. Robustness agreement is not a correctness measure: consistently wrong predictions can be perfectly invariant.

### Option budget and rendering information

At high option counts Laya's head token budget can cut the options themselves, and two options that share a prefix can come out of the cut as the same token span, which removes the question's ability to tell them apart. This matters for `label_rename`: a shorter label leaves more room for its description, so the baseline and the renamed input can survive the budget with **different content**. For `iot_hue_lightoff: iot hue lightoff` a roomy budget keeps the semantic description on both sides, but a tight budget can keep `iot_` on the baseline and `A: t` on the renamed variant -- no longer information-equivalent.

This section is an evaluation-side follow-up motivated by [#543](https://github.com/NandhaKishorM/laya/issues/543), [#517](https://github.com/NandhaKishorM/laya/issues/517) and [#569](https://github.com/NandhaKishorM/laya/issues/569). #543 raised option collapse at high option counts and was closed as addressed at runtime in v0.3.21 via #569/#542: the model side now reports, for an individual inference, which options lost a token span of their own. That report does not say whether the baseline and transformed variants of a *metamorphic comparison* still carry equivalent option information, which is the question this diagnostic answers -- per comparison, so the resulting drift is not read as pure lexical-label sensitivity.

With a real model, the CLI and `evaluate_variants()` derive a budget probe from the agent (`agent.cfg`) and attach a budget diagnostic to every variant, measured by rendering the options through the same tokenizer/`build_sequence` budget path the model input uses. `evaluate()` has no agent, so it probes only when it is passed an explicit `budget`:

* `budget` per variant: `options`, `distinct_spans` (how many options still have a token span of their own; fewer than `options` means some options can no longer be told apart), `tokens_per_option` (the uniform re-cap applied when the head budget runs out, or `null` when none was), `instruction_tokens`, `span_classes` (which presented options share a span), `retained_text` (the characters the surviving span covers, `null` when unattributable), `retained_descriptions` (the description characters inside that prefix, ignoring the label), `description_present` (whether each option carried description text at all -- `None`, empty and whitespace-only descriptions are absent; numbers, booleans and structured values render as text and count as present), `truncated` and `tail_truncated`.
* `budget_comparison` per pair: both sides' `options`, `distinct_spans`, `tokens_per_option` and `instruction_tokens`, their `retained_descriptions` aligned back to canonical order, both sides' `description_present` on `label_rename` pairs, `budget_confounded` (`true`, `false`, or `null` when the rendering could not be measured or attributed), and `reasons` naming what decided it: `clean`, `option_count`, `distinct_spans`, `instruction_tokens`, `tail_truncation`, `collision_partition` (the two sides group the options into indistinguishable spans differently, even when the number of collapsed options matches), `retained_description` or `unverified_retained_text` (with the affected canonical `slots`), `missing_semantic_description` (a `label_rename` pair whose options never had description text, with the affected canonical `slots`), or `measurement_error`.
* `report["budget"]`: `clean`/`confounded`/`unknown` counts and `confounded_rate` for `option_order`, `label_rename` and `overall`. The rate is over verifiable pairs only and is `null` when there are none, so an all-`unknown` run cannot read as clean.

Two conditions make a comparison's validity unknown and are reported as such, never as clean:

* **Renaming options that have no descriptions.** Choice criteria may legitimately carry `None`, empty or whitespace-only descriptions. Then the label is the only semantic content the option had, and rewriting it removes exactly what the transform promises to preserve, so `budget_confounded` is `null` with `missing_semantic_description` even when both sides report empty `retained_descriptions` and the budget never truncates anything. `option_order` pairs are exempt: a permutation moves every key/value pair together, so the content is preserved whatever the descriptions are, and the diagnostic never invents an `unknown` for a permutation.
* **Unattributable spans.** `retained_text`/`retained_descriptions` are attributed from character prefixes, and the only sound attribution is an exact witness: a prefix whose encoding equals the surviving span token for token. A growing prefix's token count is not guaranteed to be non-decreasing -- a BPE merge can encode a longer prefix to fewer tokens, and lossy normalization can drop characters -- so witnesses need not be adjacent and the search is global, over every character prefix. Exactly one witness is required: none, several, or an option longer than the probe's documented length bound (`metamorphic._ATTRIBUTION_LIMIT` characters) leaves the slot `null` and the pair unknown -- a conservative result instead of a possibly false clean one.

The diagnostic is deterministic and defines no threshold. It never erases or overrides the observed model metrics: a confounded pair still reports its `semantic_agreement` and drift, and `budget_confounded: true` with `semantic_agreement: false` (or with perfect agreement) are both representable. It does **not** prove semantic equivalence -- it reports whether the rendered option content still matches well enough for the drift to be read as lexical-label sensitivity. Options that the budget collapsed on both sides for the same reason stay clean, because option order and label renaming move or rewrite labels by design; only *content* differences count. `tokens_per_option` is reported but never confounds by itself: when the re-cap does not bite, the model sees the same sequence either way. Comparisons the probe cannot measure are reported as unknown rather than assumed clean. Without a model there is no tokenizer to probe, so `evaluate(..., budget=None)` leaves the report byte-identical to earlier versions.

For another corpus, the Python API accepts `(state, questions)` cases in the same shape as the harness, and a callback returning probability vectors in presented option order:

```python
Expand All @@ -388,7 +407,7 @@ results = evaluate_variants(agent, case, variants)
report = compare_predictions(baseline=results.baseline, variants=results.variants)
```

For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`. The scorer takes a batch of harness `(state, questions)` inputs and returns one probability vector per input. The public transformations return independent copies and explicit mappings in both directions, including identity label mappings for order-only transformations.
For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`. The scorer takes a batch of harness `(state, questions)` inputs and returns one probability vector per input. The public transformations return independent copies and explicit mappings in both directions, including identity label mappings for order-only transformations. Both entry points accept `budget=BudgetProbe(tok, max_len=..., head_max_len=...)` to attach the budget diagnostic to a stub tokenizer. `evaluate_variants()` derives one from `agent.cfg` automatically when a real agent is used, and skips it when a `score` is injected; `evaluate()` has no agent and probes only when a `budget` is passed explicitly.

**Semantic agreement and distribution stability are different properties.**
A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold.
A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold. **Experimental validity is a third property.** A `budget_confounded` comparison is still recorded with its metrics, but its drift no longer isolates lexical-label sensitivity.
13 changes: 10 additions & 3 deletions research/eval/laya_eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,15 @@ def available_languages() -> List[str]:


# ---------------------------------------------------------------------- scoring
def internal_question(qdef: Dict[str, Any]) -> Dict[str, Any]:
"""The `build_sequence` question dict for one harness question definition.

Shared with the metamorphic budget probe so a diagnostic measures exactly the
question that gets scored.
"""
return {"t": qdef["type"], "ins": qdef["instructions"], "crit": qdef.get("criteria")}


def score_cases(agent, cases) -> List[Any]:
"""Raw marker logits per case, from one collated forward pass.

Expand All @@ -132,9 +141,7 @@ def score_cases(agent, cases) -> List[Any]:
items = []
for state, questions in cases:
for _qid, qdef in questions.items():
q = {"t": qdef["type"],
"ins": qdef["instructions"],
"crit": qdef.get("criteria")}
q = internal_question(qdef)
ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)
if len(markers) != len(render_options(q)):
raise ValueError("marker/option count mismatch; question exceeds head_max_len")
Expand Down
Loading
Loading