Skip to content

fix(evals): report what a request waited, and name the throughput share - #592

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:fix-evals-latency-quantity
Sep 27, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:fix-evals-latency-quantity

Conversation

@aashish254

@aashish254 aashish254 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes #585.

What

laya/evals.py published elapsed / len(chunk) as every row's latency, while docs/evals.md called
that metric "per request | wall time". A chunk is one call, so each of its rows waited the whole
call; dividing it by the chunk size is a throughput share. One list therefore mixed two quantities —
and an ungrouped row kept its full cost next to grouped rows that kept 1/n of theirs.

Three consequences, all from #585:

  • A latency gate could be passed by a run where nothing finished in time. --max reads
    report.overall directly, so with one forward at 100 ms and a batch of n at 0.6·n·100 ms,
    --max latency_p50_ms=80 failed the unbatched run and passed --batch-size 8 on
    identical decisions — the passing run's 6-row chunk returned each request at +360 ms.
  • compare could not see it: it skips any *_ms metric unless a tolerance names it
    (tests/test_evals.py:75 asserts that skip), so a baseline pair taken across the flag showed
    nothing — or, with a 1 ms latency tolerance, a −40 ms "improvement" that was pure definition.
  • The artifact did not say which measurement it was. config carried only dataset/model/
    device, so a batched --json was indistinguishable from an unbatched one.

Stack

Depends on #584 (aashish254:feat-evals-runner) — this edits the chunk loop that PR rewrites,
and #584's _batch_form is the fact the new config.timing reports. It targets main because
GitHub rejects a fork branch as a base; I'll rebase onto main the moment #584 lands. Nothing here
needs #584 to be correct, only the same lines: the defect is unchanged between main and #584's
head (identical probe output apart from the checkout path).

Change

evaluate() keeps measuring the same thing once per chunk and publishes both honest numbers:

metric is was
latency_p50_ms / latency_p95_ms the wall time of the call that returned the row, unsplit 1/n of that call
cost_per_decision_p50_ms / _p95_ms (new) that call divided by its own chunk size (the old latency_* number)
  • An unbatched run is unchanged. With chunks of one the two quantities are the same number, so
    every report taken without --batch-size keeps its values. The new keys end in _ms, so
    compare's existing rule covers them and no committed baseline gains a compared metric.
  • config.timing records what the harness did: batch_size requested, the batch_form it
    resolved to (states / requests / null), chunks, rows_grouped, rows_alone, max_chunk,
    and one line naming each metric. This is also the fix(evals): batch the runner shape evaluate() already documents #584 connection: asking for --batch-size 16
    against a runner it cannot batch now reads back as batch_form: null, rows_grouped: 0.
  • docs/evals.md: corrected the latency_* row, added the cost_per_decision_* row, and added the
    Batching and timing section — the page had no mention of --batch-size at all.
  • No signature or field changes, so tests/test_hooks_api.py needs no contract line (it has no
    evals coverage to update; it stays at 131 passed).

Measured

Fake clock, exact, no weights, 11 rows where 10 share a question schema and one cannot join — same
decisions in every run (probe_evals_latency_mixture.py, logs in
laya-bench/evals_latency_{before,parent,after}.log):

unbatched p50 --batch-size 8 p50 gate at 80 ms
before (main, identical on #584's head) 100.0 ms 60.0 ms unbatched FAILS, batched PASSES
after 100.0 ms 360.0 ms (what each row waited) both FAIL

--max cost_per_decision_p50_ms=80 still separates them (batched passes at 60, unbatched fails at
100), so the throughput win is gateable under the name that measures it.

Real weights, english on cpu, research/evals/fixture.jsonl (12 rows), via laya-evals run --model english --device cpu [--batch-size 16] --json (laya-bench/real_evals_*.log):

run old latency_p50_ms new latency_p50_ms new cost_per_decision_p50_ms
unbatched 59.68 59.46 59.46 (equal by construction)
batched 34.16 139.02 36.88

So the batched report claimed a per-request latency 4.1x below the wait its rows actually had,
and its p95 (384 ms) looked better than the unbatched run's (1956 ms) at identical decisions. The
quality block is byte-identical across all four reports (choice_accuracy 0.75, ece 0.159563,
mean_confidence 0.73035, noul_accuracy 1.0, score_mae 1.341825) — only the timing moved. The
batched run's own config.timing on real weights: chunks 4, rows_grouped 11, rows_alone 1, max_chunk 4, batch_form "states".

Tests

tests/test_evals.py 23 → 30 passing, weight-free. The new ones drive a fake clock through
evaluate, and corroborate the grouping from the runner's own recorded chunks/singles rather
than re-deriving it in the test:

  • test_batched_latency_is_what_a_request_waited — 180 ms is the chunk of three's own call;
    asserts the three-row/one-row split and that batching raises latency.
  • test_the_throughput_share_keeps_its_own_metric — unbatched the two agree; batched the share is
    60 ms where the wait is 180 ms.
  • test_a_latency_gate_cannot_pass_a_run_where_nothing_finished_in_time — through the shipped
    _check_thresholds: a limit below every real wait fails both runs, a limit above every real wait
    passes both, and the cost metric still separates them.
  • test_timing_facts_record_what_the_harness_did / test_a_requested_batch_size_is_not_a_batched_run
    — the flag is not the fact; the --json artifacts differ.
  • test_compare_leaves_the_cost_metrics_alone — new metrics are skipped by default, honoured when
    named.
  • test_docs_and_the_harness_name_the_same_timing_metrics — the metric names in
    docs/evals.md's table must equal the *_ms keys evaluate emits, with the right quantity
    labelled on each row.

Gates: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 clean,
compileall -q laya/ tests/ clean, pytest tests/test_evals.py 30 passed,
tests/test_evals_api.py 12 passed, tests/test_cli.py all passed, tests/test_hooks_api.py 131
passed, python -m zensical build --strict --clean → "No issues found".

Mutation battery (laya-bench/mutate_evals_latency.py, 9 single-edit variants, each reinstating
one defect): every one is caught by a named test, and both files restore byte-identical —
including m5, "the stamp records the flag rather than the grouping", caught only by the
cannot-batch runner, and m8, "the docs stop saying which quantity latency is", caught by the
docs/code consistency test.

Overlap

`evaluate` gated batching on `hasattr(runner, "predict_batch")` and then called it as
`predict_batch(states, questions, model=..., batch_size=...)`. `Router` is the one real runner
whose `predict(state, questions, model=...)` matches the contract the module states, and its
`predict_batch` takes one dict per request, so the gate opened the single branch the harness
could not call: `on_error="fail"` raised `TypeError: ... unexpected keyword argument 'model'`,
and `on_error="skip"` turned that into a report with zero cases and an empty `overall`.

Read the signature of the entry point being called instead, drive `Router`'s documented
request-dict form with the chunk's own state, questions and checkpoint, and score a runner
whose batch method fits neither shape one `predict` at a time. Both docstrings now say which
shapes a runner must offer.

Measured on `english`, CPU, 60 rows: `evaluate(Router, batch_size=16)` goes from `TypeError` to
1321 ms / 4 forward passes against 3746 ms / 60 unbatched, with an identical decision digest and
the same numbers `RouterRunner` produces.
A chunk is one call, so every row it carried returned at the same moment; the
harness published 1/N of that call under the name of the per-request wait, so a
batched run mixed a cost figure and a full cost in one list and a latency gate
could be passed by a run in which no request finished in time (NandhaKishorM#585).

`latency_p50_ms` / `latency_p95_ms` are now the whole call, which is what
`docs/evals.md` always said they were, and the old number keeps its own metric:
`cost_per_decision_p50_ms` / `_p95_ms`. Without `--batch-size` the two coincide,
so an unbatched report is unchanged. The report also records what the grouping
really was in `config.timing` -- the resolved runner shape, chunks issued, rows
grouped and alone, largest chunk -- because the flag a caller asked with does
not say whether anything was batched.
@aashish254

aashish254 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Closes #585. This takes option 2 for latency_* -- the whole call, which is what docs/evals.md promised -- and keeps the old number as cost_per_decision_*, so option 1's disclosure lands as config.timing in the artifact rather than as a docs-only warning. Stacked on #584, which the issue flagged as the reason to wait; the defect is byte-identical between main and that branch, so the numbers in the body are comparable across both.

@Bruce-Yii

Copy link
Copy Markdown
Contributor

The config.timing counters are incremented after the except branch's continue (laya/evals.py:392, counters at :399-405), so a chunk that raises under on_error="skip" contributes to none of them.

Measured on this head with 8 rows, --batch-size 8, and a runner that raises on any multi-row predict_batch:

value
call sizes the harness actually issued [7, 1]
config.timing chunks: 1, rows_grouped: 0, rows_alone: 1, max_chunk: 1
config.errored 7 entries

The report positively asserts nothing was shared, after a 7-row shared forward was issued. The same rows_grouped: 0, max_chunk: 1 is what a runner with no predict_batch at all produces (chunks: 8, rows_alone: 8), so the artifact cannot distinguish "nothing was batched" from "the batched call raised". Failing a row by omitting its answer rather than raising gives max_chunk: 7 for the identical grouping.

This is the part docs/evals.md:74-77 introduces the block for — "how many rows shared a call, and the largest chunk ... because the flag alone does not say whether anything was batched" — and the two counters it names are the two that are wrong, in the one mode where the block has to be right. Hoisting chunks / rows_grouped / rows_alone / max_chunk above the continue, and leaving waits / shares where they are so the metric semantics stay exactly as inherited, would count the calls issued rather than the calls completed.

config.timing counted only calls that returned, so a chunk that raised
under on_error=skip left rows_grouped and max_chunk at the values a
runner with no batching produces. Hoist the shape counters above the
error branch and leave the two metric lists below it, so a failed call
cannot contribute a latency it never measured.
@aashish254

aashish254 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Short answer: fixed in 1171088 — your reading of the mechanism and of the artifact-level consequence are both right.

The shape counters (chunks / rows_grouped / rows_alone / max_chunk) now sit above the call: they answer "what did the harness issue", which is settled the moment the chunk is grouped. waits / shares stay below it, exactly as you proposed, so a call that returned nothing still publishes no latency.

Re-measured on 8 rows, batch_size=8, with a runner that raises on any multi-row predict_batch: the issued call sizes are [8] either way, config.errored has 8 entries either way, and no *_ms metrics publish either way. What differs is config.timing: chunks: 0, rows_grouped: 0, rows_alone: 0, max_chunk: 0 at dfba47a versus chunks: 1, rows_grouped: 8, rows_alone: 0, max_chunk: 8 at 1171088. Your [7, 1] case was the milder shape of the same bug: with every row in one chunk, reporting all four counters as zero positively asserts that no call went out at all.

Tests: python -m pytest tests/test_evals.py -q -> 31 passed. The new test_a_batched_call_that_raises_is_still_recorded_as_batched keeps an independent witness (runner.chunks == [3]), asserts (chunks, rows_grouped, rows_alone) == (2, 3, 1), max_chunk == 3, three errored entries, and that latency_p50_ms is still the one completed call (100 ms), plus [case["correct"] for case in report.cases] == [True]. It closes your ambiguity directly: against a StubRunner with no predict_batch at all, chunks, rows_grouped and max_chunk now differ; before the fix all three matched. Failing-first: (1, 0, 1) at dfba47a where the expectation is (2, 3, 1). Mutation: adding waits.extend([0.0] * len(chunk)) to the except branch — the wrong way to make the counters honest — fails only the latency assertion (Obtained: 0.0 / Expected: 100.0).

docs/evals.md now states the split. Adjacent suites: tests/test_cli.py 38 passed, tests/test_batch.py 49, tests/test_evals_api.py 12; ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 has nothing to report in the changed file.

@NandhaKishorM
NandhaKishorM merged commit 9f0d06c into NandhaKishorM:main Sep 27, 2026
22 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254. Merged (with #584). It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

laya-evals: latency_p50_ms measures a different thing under --batch-size, and the report doesn't say which

3 participants