Repository navigation
fix(evals): every laya-evals argument says what it is, in --help and on the page - #813
Conversation
_build_parser declared 22 arguments; five printed nothing about themselves. `run --on-error`
showed as bare `--on-error {fail,skip}`, so nothing on the command said which of the two is the
fallback or that 'skip' puts a row under the report's config.errored instead of failing the run.
`compare --baseline` is required by that subcommand and printed with no description at all, and
the `dataset` positional of both `validate` and `run` and `compare`'s `report` had nothing under
them either.
The help strings state what the code does, read off it: `validate` ignores blank and '#' lines and
errors on an empty file (laya/evals.py:116-124), `evaluate` re-raises under 'fail' and appends
{"index", "error"} rows to config.errored under 'skip' (laya/evals.py:636-655, 700-701), and a
baseline must match the report's schema and its dataset and question fingerprints
(laya/evals.py:42).
Witness, `laya-evals run --help` before:
--on-error {fail,skip}
after:
--on-error {fail,skip}
'fail' (the default) stops the run when a runner call raises; 'skip'
lists every row it could not score under the report's config.errored
and reports the metrics for the rows that returned
…kipping The page explained what skipping does to the batch counters -- "with `on_error=skip`, a chunk whose call raised still counts in `rows_grouped` and `max_chunk`" -- using the library parameter's name, and never said that the way to ask for it is `laya-evals run --on-error skip`, or that the default aborts the run. The reader had the behaviour and not the flag.
… --help and on the page Four checks over the parser the CLI really builds, so a flag added tomorrow is held to the same rules as `--on-error` is today: - every argument carries a non-empty help string, because argparse prints that text and nothing else; - a flag whose parser sets a real default says so in its own help, in the same clause as the quoted value -- argparse prints no default, and a help string that lists both choices reads as though either could be the fallback; - every option the parser defines is named in docs/evals.md, the page a caller reads; - and the other direction, so an example cannot advertise a flag its subcommand would reject. Against `main` the first two rules fire on the sites the fix covers and the third on `run --on-error`: 28 passed, 2 failed. On the branch: 30 passed, 0 failed (main's own file runs 25). A 15-arm mutation table -- 10 drift injections, each killed by the check that owns it, 4 ablations that delete a check and inject only its own drift, and the control -- behaved 15 of 15.
|
Merged for 0.3.23. Five arguments with no One resolution on the way in: #777 landed Verified on the integrated branch: Thank you. |
What
laya-evalshad five arguments that printed nothing about themselves, anddocs/evals.mdnevernamed one of them. This gives all five a help string, names
--on-errorin the sentence on thepage that already describes what skipping does, and adds four checks over the parser so the next
flag cannot ship the same way.
Why
_build_parserdeclares 22 arguments; 5 of them carried nohelp=at all.--helpis the onlyinterface before a caller runs something, and what it printed was:
--on-erroris the worst of them: it is a two-valued switch about whether a runner error aborts arun or becomes a
config.erroredrow, and the help said neither -- not even which offailandskipis the default, which argparse does not print on its own.compare --baselineis the oneargument that subcommand requires (
required=True) and printed bare too.The docs page was wrong in the matching direction.
docs/evals.md:116explained the consequence ofskipping -- "with
on_error=skip, a chunk whose call raised still counts inrows_groupedandmax_chunk, next to its entries inconfig.errored" -- using the library parameter's name, whilenever saying that the way to ask for it is
laya-evals run --on-error skip, or that the defaultaborts instead. So the page had the behaviour and the parser had the flag, and neither named the
other.
The help strings state what the code does, read off the code rather than written around it:
validate'sdataset: blank lines and#lines are skipped and a file that yields no examplesis an error (
laya/evals.py:116-124);--on-error:failre-raises,skipappends{"index", "error"}rows and keeps scoring whatreturned (
laya/evals.py:636-655, which writes them toconfig.erroredat 700-701);compare'sreportand--baseline: the baseline must agree onschema,dataset_sha256andquestions_sha256(laya/evals.py:42).The gate reads the parser the CLI builds, not a copy of its source, so it holds for a flag added
tomorrow the way it holds for
--on-errortoday:default-- a help string that merely lists both choices would otherwise still read as ifeither could be the fallback, which is what happens if the default is flipped;
docs/evals.md;laya-evals <subcommand>line in that page is onethat subcommand defines, so an example cannot advertise a flag the parser would reject. Backslash
continuations are folded first, so a flag on the second line of a multi-line example is still
attributed to the command that started it.
The
evalsCI job (evals (metric math, dataset, API guard)) already runspython tests/test_evals_api.py, so the gate runs there without touching the workflow.How it was verified
laya-evals run --helpon this branch, the flag that had nothing before:The same gate run against
main-- the two rules that catch the shipped state, with the sites theyname:
On the branch:
30 passed, 0 failed. main's owntests/test_evals_api.pyruns 25 checks, thisadds 5.
Mutation table, 15 arms: each drift injection is killed by the check that owns it and names the
injected site; each ablation deletes one check and injects only the drift that check caught, and
stays green -- which is what proves the four checks are four owners rather than four copies of one.
run --on-errorloses its helprun --on-errorcompare --baselineloses its helpcompare --baselinecompare reportloses its helpcompare reportrun datasetloses its helprun datasetvalidate datasetloses its helpvalidate datasetskip, the help still blamesfailrun --on-error (default 'skip')run --on-errorrun --on-error--temperaturerun --temperaturerun --on-failure, rule 4 namesrun --on-error10 of 10 drift arms killed; 4/4 ablate arms proved sole ownership; 0 misbehaving (control + 15 arms total).The suites this touches, before and after:
No public API changed:
laya.evals' signatures are untouched, and the CLI accepts exactly the samearguments it did -- the parser now describes them.
tests/test_hooks_api.pyis the hook APIcontract, and this touches neither hooks nor that contract.
Related work I can see: #755 and #759 add
laya-evalsoptions, and #777 adds--min-confidence.All three land in
_build_parser, so whichever merges last will have to reline that block; thegate here is the reason they should each carry a
help=and a line indocs/evals.mdwhile theydo it, which is the behaviour they already intend. #787 touches the harness, not the parser. The
three files here are
laya/evals_cli.py(the_build_parserbody only),docs/evals.md(onesentence inside
## Batching and timing), andtests/test_evals_api.py(a new section ahead ofthe summary print).
Checklist
mainFixes # (no open issue: the gap was found by reading
_build_parseragainst--helpoutput anddocs/evals.md; no issue proposes documenting these arguments)