Repository navigation
fix(cli): --preset sent the request under a key no question set names - #426
Merged
NandhaKishorM merged 1 commit intoSep 25, 2026
Merged
Conversation
Each shipped question set names the state field it reads, and they are all different — `message`
for triage, `body` for email, `prompt` for guard, `post` for moderation, `request` for router.
The CLI sent every one of them as `{"text": ...}`, a key none of them names, so the model was
asked about a field that was not in the state:
```
$ laya "Refactor this service to use dependency injection and explain the trade-offs." --preset router
difficulty : 1.78
domain : writing (p=0.569)
# the same call with the key the preset's own instructions name
difficulty : 2.03
domain : code (p=0.918)
```
`writing` is the wrong answer for a refactoring request, and the model had no way to know: the
state contained one field, `text`, and the question asked about `request`. Measured on the
English checkpoint across the repository's own example texts, with only the key differing:
| preset | `text` | the preset's own key |
|---|---|---|
| router | `writing` 0.569 | **`code` 0.918** |
| router (README's B-tree example) | `code` 0.396 | **`data_analysis` 0.557** |
| triage | `other` 0.436 | **`information` 0.437** |
| email | `is_spam` **0.731** | `is_spam` **0.505** |
The email row is the one that matters most: the `noul` probability crosses 0.5, so a caller
thresholding it got the opposite decision from a request that differed only in a key name.
`--predict` alone was affected too, because it answers `router_questions()`, whose instructions
name `request`.
Deliberate choices tested:
- **A mapping beside `PRESETS`, not a key derived from the questions.** The key could be scraped out of each question's `instructions`, and the new test does exactly that to check the two agree — but scraping at run time would make a preset's behaviour depend on the wording of its prompt, so the mapping is explicit and the test is what keeps them in step.
- **The routing path is unchanged.** `route` reads the state only for language detection, and `lang.analyse` is key-invariant, so `{"text": ...}` stays there and `tests/test_cli.py:58`, which pins it, is untouched. Verified across eight texts and six key names: the routed checkpoint is identical in every combination.
- **`--predict` uses `request`.** It answers the router preset, so it takes the router key rather than keeping a special case of its own.
- **The tests assert the key AND that the key is one the preset names.** The first catches the bug; the second stops the mapping and the presets from drifting apart, which is the failure this PR is fixing one level up.
```
# with the single hard-coded key restored
32 passed, 6 failed
FAIL preset email: state carries the text under 'body': {'text': 'the request'}
FAIL preset guard: state carries the text under 'prompt': {'text': 'the request'}
FAIL preset moderation: state carries the text under 'post': {'text': 'the request'}
FAIL preset router: state carries the text under 'request': {'text': 'the request'}
FAIL preset triage: state carries the text under 'message': {'text': 'the request'}
FAIL predict: state carries the text under 'request' (router_questions): {'text': 'the request'}
# with it
$ python tests/test_cli.py 38 passed, 0 failed (22 before)
$ python tests/test_mcp.py 86 passed, 0 failed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```
Known limitation: `--preset` and `--predict` now put the text where the question set reads it,
but nothing warns a caller who passes a state with the wrong key through the library or the MCP
tool surface — the same mismatch is silently accepted there, and the MCP tool description does
not name the field either. That is the same defect one layer out and is not addressed here.
4 tasks done
Owner
|
Thank you, this is a real correctness fix: the presets were being asked about a field that was not in the state. The test that reads each preset's own instructions to confirm its key is a nice touch, because it stops the mapping and the presets from drifting apart. Merging. |
This was referenced Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(cli):
--presetsent the request under a key no question set namesEach shipped question set names the state field it reads, and they are all different —
messagefor triage,
bodyfor email,promptfor guard,postfor moderation,requestfor router.The CLI sent every one of them as
{"text": ...}, a key none of them names, so the model wasasked about a field that was not in the state:
writingis the wrong answer for a refactoring request, and the model had no way to know: thestate contained one field,
text, and the question asked aboutrequest. Measured on theEnglish checkpoint across the repository's own example texts, with only the key differing:
textwriting0.569code0.918code0.396data_analysis0.557other0.436information0.437is_spam0.731is_spam0.505The email row is the one that matters most: the
noulprobability crosses 0.5, so a callerthresholding it got the opposite decision from a request that differed only in a key name.
--predictalone was affected too, because it answersrouter_questions(), whose instructionsname
request.Deliberate choices tested:
PRESETS, not a key derived from the questions. The key could be scraped out of each question'sinstructions, and the new test does exactly that to check the two agree — but scraping at run time would make a preset's behaviour depend on the wording of its prompt, so the mapping is explicit and the test is what keeps them in step.routereads the state only for language detection, andlang.analyseis key-invariant, so{"text": ...}stays there andtests/test_cli.py:58, which pins it, is untouched. Verified across eight texts and six key names: the routed checkpoint is identical in every combination.--predictusesrequest. It answers the router preset, so it takes the router key rather than keeping a special case of its own.Known limitation:
--presetand--predictnow put the text where the question set reads it,but nothing warns a caller who passes a state with the wrong key through the library or the MCP
tool surface — the same mismatch is silently accepted there, and the MCP tool description does
not name the field either. That is the same defect one layer out and is not addressed here.