Repository navigation
feat(cli): score a file or stdin of requests in one batch - #511
Merged
Merged
Conversation
This was referenced Sep 25, 2026
Owner
|
Thanks @aashish254, merged. After #542 landed, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
pip install layaalready puts alayacommand on the path for quick local testing, but it canonly score one text per process:
laya "text" --predictcallsRouter.predictonce, so tryinga set of tickets means one process — one checkpoint load, one forward pass — per ticket. The
library has had
Router.predict_batch/route_batchfor exactly this since #294-era batchinglanded; the CLI never exposed them. This closes the gap between what the package can do and what
typing
layaat a shell gets you.What
laya/cli.py—--batch FILE(-for stdin): one request per non-blank line, scored througha single
Router.predict_batchcall (orroute_batchfor the no---predictrouting mode).Existing semantics carry over unchanged:
--predict/--preset/--model/--lang/--task, the per-preset state keys, and--json— which in batch mode prints JSONL, onerecord per line, matching how
laya-evalsconsumes datasets.--batch-size Nbounds theforward pass; blank lines are skipped; missing/empty files and
text + --batchexit 2 with anamed error instead of a traceback.
tests/test_cli.py— 14 new checks, still weight-free (stubbed Router): one batch call not Nsingle calls, line skipping/stripping,
--batch-sizeand per-request flag forwarding, presetstate keys, JSONL output, routing mode via
route_batch, and the three error paths.Measured
Real
convaiinnovations/layacheckpoint, 20 English support tickets, one warm process, CPU(M3), best of 3:
predict(what the CLI could do before)--batch --batch-size 8--batch(single pass)Decision parity: 0/20 flips (choice label / noul>=0.5 / round(score)) and max choice
probability drift 0.0 versus the per-state loop. At the process level a full batch also amortizes
checkpoint load:
for t in tickets: laya --predictpays it once per ticket,laya --batchonceper file.
Notes
route_batch's per-request dict shape already acceptsmodel/task/lang, so batch routingneeded no Router changes.
laya/cli.pybeyond the documentation-onlydocs: add command line and MCP server guide #430 (adds
docs/cli-mcp.md, no runtime changes).--batch's flags deliberately mirror thesingle-text flags so docs: add command line and MCP server guide #430's guide stays accurate once this lands.
Test plan
python tests/test_cli.py— 59/59 (45 pre-existing + 14 new)ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120,compileallpython tests/test_router.py/tests/test_hooks_api.py— no regressions