Skip to content

fix(evals): every laya-evals argument says what it is, in --help and on the page - #813

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:feat-evals-cli-help
Oct 1, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:feat-evals-cli-help

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

laya-evals had five arguments that printed nothing about themselves, and docs/evals.md never
named one of them. This gives all five a help string, names --on-error in the sentence on the
page that already describes what skipping does, and adds four checks over the parser so the next
flag cannot ship the same way.

Why

_build_parser declares 22 arguments; 5 of them carried no help= at all. --help is the only
interface before a caller runs something, and what it printed was:

  dataset

  --on-error {fail,skip}

  report

  --baseline

--on-error is the worst of them: it is a two-valued switch about whether a runner error aborts a
run or becomes a config.errored row, and the help said neither -- not even which of fail and
skip is the default, which argparse does not print on its own. compare --baseline is the one
argument that subcommand requires (required=True) and printed bare too.

The docs page was wrong in the matching direction. docs/evals.md:116 explained the consequence of
skipping -- "with on_error=skip, a chunk whose call raised still counts in rows_grouped and
max_chunk, next to its entries in config.errored" -- using the library parameter's name, while
never saying that the way to ask for it is laya-evals run --on-error skip, or that the default
aborts instead. So the page had the behaviour and the parser had the flag, and neither named the
other.

The help strings state what the code does, read off the code rather than written around it:

  • validate's dataset: blank lines and # lines are skipped and a file that yields no examples
    is an error (laya/evals.py:116-124);
  • --on-error: fail re-raises, skip appends {"index", "error"} rows and keeps scoring what
    returned (laya/evals.py:636-655, which writes them to config.errored at 700-701);
  • compare's report and --baseline: the baseline must agree on schema, dataset_sha256 and
    questions_sha256 (laya/evals.py:42).

The gate reads the parser the CLI builds, not a copy of its source, so it holds for a flag added
tomorrow the way it holds for --on-error today:

  1. every argument carries a non-empty help string;
  2. a flag whose parser sets a real default names that default in the same clause as the word
    default
    -- a help string that merely lists both choices would otherwise still read as if
    either could be the fallback, which is what happens if the default is flipped;
  3. every option the parser defines is named in docs/evals.md;
  4. the other direction: every flag written on a laya-evals <subcommand> line in that page is one
    that subcommand defines, so an example cannot advertise a flag the parser would reject. Backslash
    continuations are folded first, so a flag on the second line of a multi-line example is still
    attributed to the command that started it.

The evals CI job (evals (metric math, dataset, API guard)) already runs
python tests/test_evals_api.py, so the gate runs there without touching the workflow.

How it was verified

laya-evals run --help on this branch, the flag that had nothing before:

  --on-error {fail,skip}
                        'fail' (the default) stops the run when a runner call raises; 'skip'
                        lists every row it could not score under the report's config.errored
                        and reports the metrics for the rows that returned

The same gate run against main -- the two rules that catch the shipped state, with the sites they
name:

FAIL every laya-evals argument carries a help string (got ['compare --baseline', 'compare report',
     'run --on-error', 'run dataset', 'validate dataset'], expected [])
FAIL every laya-evals option is named in docs/evals.md (got ['run --on-error'], expected [])
28 passed, 2 failed

On the branch: 30 passed, 0 failed. main's own tests/test_evals_api.py runs 25 checks, this
adds 5.

Mutation table, 15 arms: each drift injection is killed by the check that owns it and names the
injected site; each ablation deletes one check and injects only the drift that check caught, and
stays green -- which is what proves the four checks are four owners rather than four copies of one.

arm injected fired
drift run --on-error loses its help rule 1 names run --on-error
drift compare --baseline loses its help rule 1 names compare --baseline
drift compare report loses its help rule 1 names compare report
drift run dataset loses its help rule 1 names run dataset
drift validate dataset loses its help rule 1 names validate dataset
drift the default flips to skip, the help still blames fail rule 2 names run --on-error (default 'skip')
drift the help lists both choices but never says which is the default rule 2 names run --on-error
drift the page stops naming the flag rule 3 names run --on-error
drift an example advertises --temperature rule 4 names run --temperature
drift the parser renames the flag, the page keeps the old name rule 3 names run --on-failure, rule 4 names run --on-error
ablate rule 1 deleted + a help string stripped green -- rule 1 is the only owner
ablate rule 2 deleted + the default flipped green -- rule 2 is the only owner
ablate rule 3 deleted + the flag un-named green -- rule 3 is the only owner
ablate rule 4 deleted + a fake flag advertised green -- rule 4 is the only owner
control branch as committed green, nothing fires

10 of 10 drift arms killed; 4/4 ablate arms proved sole ownership; 0 misbehaving (control + 15 arms total).

The suites this touches, before and after:

python tests/test_evals_api.py    # main 25 passed, 0 failed -> branch 30 passed, 0 failed
python -m pytest tests/test_evals.py -q          # 83 passed
python tests/test_evals_onnx.py   # 27 passed, 0 failed
python tests/test_cli.py          # 135 passed, 0 failed
python tests/test_doc_tables.py   # 114 passed, 0 failed
python tests/test_env_docs.py     # 9 passed, 0 failed
laya-evals validate research/evals/fixture.jsonl # 12 examples, 4 question id(s): the CI step, unchanged
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
ruff check tests/test_evals_api.py --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/

No public API changed: laya.evals' signatures are untouched, and the CLI accepts exactly the same
arguments it did -- the parser now describes them. tests/test_hooks_api.py is the hook API
contract, and this touches neither hooks nor that contract.

Related work I can see: #755 and #759 add laya-evals options, and #777 adds --min-confidence.
All three land in _build_parser, so whichever merges last will have to reline that block; the
gate here is the reason they should each carry a help= and a line in docs/evals.md while they
do it, which is the behaviour they already intend. #787 touches the harness, not the parser. The
three files here are laya/evals_cli.py (the _build_parser body only), docs/evals.md (one
sentence inside ## Batching and timing), and tests/test_evals_api.py (a new section ahead of
the summary print).

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Fixes # (no open issue: the gap was found by reading _build_parser against --help output and
docs/evals.md; no issue proposes documenting these arguments)

_build_parser declared 22 arguments; five printed nothing about themselves. `run --on-error`
showed as bare `--on-error {fail,skip}`, so nothing on the command said which of the two is the
fallback or that 'skip' puts a row under the report's config.errored instead of failing the run.
`compare --baseline` is required by that subcommand and printed with no description at all, and
the `dataset` positional of both `validate` and `run` and `compare`'s `report` had nothing under
them either.

The help strings state what the code does, read off it: `validate` ignores blank and '#' lines and
errors on an empty file (laya/evals.py:116-124), `evaluate` re-raises under 'fail' and appends
{"index", "error"} rows to config.errored under 'skip' (laya/evals.py:636-655, 700-701), and a
baseline must match the report's schema and its dataset and question fingerprints
(laya/evals.py:42).

Witness, `laya-evals run --help` before:

    --on-error {fail,skip}

after:

    --on-error {fail,skip}
                          'fail' (the default) stops the run when a runner call raises; 'skip'
                          lists every row it could not score under the report's config.errored
                          and reports the metrics for the rows that returned
…kipping

The page explained what skipping does to the batch counters -- "with `on_error=skip`, a chunk
whose call raised still counts in `rows_grouped` and `max_chunk`" -- using the library parameter's
name, and never said that the way to ask for it is `laya-evals run --on-error skip`, or that the
default aborts the run. The reader had the behaviour and not the flag.
… --help and on the page

Four checks over the parser the CLI really builds, so a flag added tomorrow is held to the same
rules as `--on-error` is today:

- every argument carries a non-empty help string, because argparse prints that text and nothing
  else;
- a flag whose parser sets a real default says so in its own help, in the same clause as the
  quoted value -- argparse prints no default, and a help string that lists both choices reads as
  though either could be the fallback;
- every option the parser defines is named in docs/evals.md, the page a caller reads;
- and the other direction, so an example cannot advertise a flag its subcommand would reject.

Against `main` the first two rules fire on the sites the fix covers and the third on
`run --on-error`: 28 passed, 2 failed. On the branch: 30 passed, 0 failed (main's own file runs
25). A 15-arm mutation table -- 10 drift injections, each killed by the check that owns it, 4
ablations that delete a check and inject only its own drift, and the control -- behaved 15 of 15.
@NandhaKishorM
NandhaKishorM merged commit 298e443 into NandhaKishorM:main Oct 1, 2026
24 of 25 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.23.

Five arguments with no help= is the kind of thing that never gets fixed because it never breaks anything, and --on-error printing nothing but its choices was the worst of them: a caller reading --help could not tell that skip lists the unscored rows under config.errored rather than quietly dropping them. The four parser checks are the part that keeps this fixed, since the next flag now cannot ship bare.

One resolution on the way in: #777 landed --min-confidence with its own help text in this same release, so your --on-error line conflicted with it. I kept #777's argument and took your version of the --on-error line, so both are present with help strings.

Verified on the integrated branch: tests/test_evals_api.py, tests/test_evals.py (111 passed) and tests/test_evals_onnx.py pass, full test lane green.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants