Repository navigation
fix(evals): report what a request waited, and name the throughput share - #592
Conversation
`evaluate` gated batching on `hasattr(runner, "predict_batch")` and then called it as `predict_batch(states, questions, model=..., batch_size=...)`. `Router` is the one real runner whose `predict(state, questions, model=...)` matches the contract the module states, and its `predict_batch` takes one dict per request, so the gate opened the single branch the harness could not call: `on_error="fail"` raised `TypeError: ... unexpected keyword argument 'model'`, and `on_error="skip"` turned that into a report with zero cases and an empty `overall`. Read the signature of the entry point being called instead, drive `Router`'s documented request-dict form with the chunk's own state, questions and checkpoint, and score a runner whose batch method fits neither shape one `predict` at a time. Both docstrings now say which shapes a runner must offer. Measured on `english`, CPU, 60 rows: `evaluate(Router, batch_size=16)` goes from `TypeError` to 1321 ms / 4 forward passes against 3746 ms / 60 unbatched, with an identical decision digest and the same numbers `RouterRunner` produces.
A chunk is one call, so every row it carried returned at the same moment; the harness published 1/N of that call under the name of the per-request wait, so a batched run mixed a cost figure and a full cost in one list and a latency gate could be passed by a run in which no request finished in time (NandhaKishorM#585). `latency_p50_ms` / `latency_p95_ms` are now the whole call, which is what `docs/evals.md` always said they were, and the old number keeps its own metric: `cost_per_decision_p50_ms` / `_p95_ms`. Without `--batch-size` the two coincide, so an unbatched report is unchanged. The report also records what the grouping really was in `config.timing` -- the resolved runner shape, chunks issued, rows grouped and alone, largest chunk -- because the flag a caller asked with does not say whether anything was batched.
|
Closes #585. This takes option 2 for |
|
The Measured on this head with 8 rows,
The report positively asserts nothing was shared, after a 7-row shared forward was issued. The same This is the part |
config.timing counted only calls that returned, so a chunk that raised under on_error=skip left rows_grouped and max_chunk at the values a runner with no batching produces. Hoist the shape counters above the error branch and leave the two metric lists below it, so a failed call cannot contribute a latency it never measured.
|
Short answer: fixed in 1171088 — your reading of the mechanism and of the artifact-level consequence are both right. The shape counters ( Re-measured on 8 rows, Tests:
|
|
Thanks @aashish254. Merged (with #584). It ships in 0.3.21. |
Closes #585.
What
laya/evals.pypublishedelapsed / len(chunk)as every row's latency, whiledocs/evals.mdcalledthat metric "per request | wall time". A chunk is one call, so each of its rows waited the whole
call; dividing it by the chunk size is a throughput share. One list therefore mixed two quantities —
and an ungrouped row kept its full cost next to grouped rows that kept 1/n of theirs.
Three consequences, all from #585:
--maxreadsreport.overalldirectly, so with one forward at 100 ms and a batch of n at0.6·n·100ms,--max latency_p50_ms=80failed the unbatched run and passed--batch-size 8onidentical decisions — the passing run's 6-row chunk returned each request at +360 ms.
comparecould not see it: it skips any*_msmetric unless a tolerance names it(
tests/test_evals.py:75asserts that skip), so a baseline pair taken across the flag showednothing — or, with a 1 ms latency tolerance, a −40 ms "improvement" that was pure definition.
configcarried onlydataset/model/device, so a batched--jsonwas indistinguishable from an unbatched one.Stack
Depends on #584 (
aashish254:feat-evals-runner) — this edits the chunk loop that PR rewrites,and #584's
_batch_formis the fact the newconfig.timingreports. It targetsmainbecauseGitHub rejects a fork branch as a base; I'll rebase onto
mainthe moment #584 lands. Nothing hereneeds #584 to be correct, only the same lines: the defect is unchanged between
mainand #584'shead (identical probe output apart from the checkout path).
Change
evaluate()keeps measuring the same thing once per chunk and publishes both honest numbers:latency_p50_ms/latency_p95_mscost_per_decision_p50_ms/_p95_ms(new)latency_*number)every report taken without
--batch-sizekeeps its values. The new keys end in_ms, socompare's existing rule covers them and no committed baseline gains a compared metric.config.timingrecords what the harness did:batch_sizerequested, thebatch_formitresolved to (
states/requests/null),chunks,rows_grouped,rows_alone,max_chunk,and one line naming each metric. This is also the fix(evals): batch the runner shape evaluate() already documents #584 connection: asking for
--batch-size 16against a runner it cannot batch now reads back as
batch_form: null, rows_grouped: 0.docs/evals.md: corrected thelatency_*row, added thecost_per_decision_*row, and added theBatching and timing section — the page had no mention of
--batch-sizeat all.tests/test_hooks_api.pyneeds no contract line (it has noevalscoverage to update; it stays at 131 passed).Measured
Fake clock, exact, no weights, 11 rows where 10 share a question schema and one cannot join — same
decisions in every run (
probe_evals_latency_mixture.py, logs inlaya-bench/evals_latency_{before,parent,after}.log):--batch-size 8p50main, identical on #584's head)--max cost_per_decision_p50_ms=80still separates them (batched passes at 60, unbatched fails at100), so the throughput win is gateable under the name that measures it.
Real weights,
englishon cpu,research/evals/fixture.jsonl(12 rows), vialaya-evals run --model english --device cpu [--batch-size 16] --json(laya-bench/real_evals_*.log):latency_p50_mslatency_p50_mscost_per_decision_p50_msSo the batched report claimed a per-request latency 4.1x below the wait its rows actually had,
and its p95 (384 ms) looked better than the unbatched run's (1956 ms) at identical decisions. The
quality block is byte-identical across all four reports (
choice_accuracy 0.75,ece 0.159563,mean_confidence 0.73035,noul_accuracy 1.0,score_mae 1.341825) — only the timing moved. Thebatched run's own
config.timingon real weights:chunks 4, rows_grouped 11, rows_alone 1, max_chunk 4, batch_form "states".Tests
tests/test_evals.py23 → 30 passing, weight-free. The new ones drive a fake clock throughevaluate, and corroborate the grouping from the runner's own recordedchunks/singlesratherthan re-deriving it in the test:
test_batched_latency_is_what_a_request_waited— 180 ms is the chunk of three's own call;asserts the three-row/one-row split and that batching raises latency.
test_the_throughput_share_keeps_its_own_metric— unbatched the two agree; batched the share is60 ms where the wait is 180 ms.
test_a_latency_gate_cannot_pass_a_run_where_nothing_finished_in_time— through the shipped_check_thresholds: a limit below every real wait fails both runs, a limit above every real waitpasses both, and the cost metric still separates them.
test_timing_facts_record_what_the_harness_did/test_a_requested_batch_size_is_not_a_batched_run— the flag is not the fact; the
--jsonartifacts differ.test_compare_leaves_the_cost_metrics_alone— new metrics are skipped by default, honoured whennamed.
test_docs_and_the_harness_name_the_same_timing_metrics— the metric names indocs/evals.md's table must equal the*_mskeysevaluateemits, with the right quantitylabelled on each row.
Gates:
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120clean,compileall -q laya/ tests/clean,pytest tests/test_evals.py30 passed,tests/test_evals_api.py12 passed,tests/test_cli.pyall passed,tests/test_hooks_api.py131passed,
python -m zensical build --strict --clean→ "No issues found".Mutation battery (
laya-bench/mutate_evals_latency.py, 9 single-edit variants, each reinstatingone defect): every one is caught by a named test, and both files restore byte-identical —
including m5, "the stamp records the flag rather than the grouping", caught only by the
cannot-batch runner, and m8, "the docs stop saying which quantity latency is", caught by the
docs/code consistency test.
Overlap
laya/evals.py,laya/evals_cli.py,research/evals/check_regression.pyandtests/test_evals.py. It keeps the rule my change relieson — a
*_msmetric is skipped unless a tolerance names it — but makes a missing baseline metricfail, so under both PRs a tolerance that names
cost_per_decision_p50_msagainst a build thatdoes not publish it fails loudly instead of passing. That is the behaviour I'd want, and it composes
without either side editing the other's lines. The real collision is
tests/test_evals.py: fix(evals): fail the baseline comparison when a metric is missing #510adds cases just before where mine sit, so whichever lands second resolves a test-file conflict.
Adjacent hunks, no logic overlap — I won't touch
compareorassert_regression.docs/evals.mdandlaya/evals_cli.pyin other sections(the model slice,
--revision,--onnx); text collision on that page only.laya/public API change, nolaya-ts(nnlgsakib's lane), and I verified before coding thatno open PR cites this issue:
gh api search/issues q='repo:NandhaKishorM/laya is:pr 585'→ 0.