Skip to content

fix(cli): --preset sent the request under a key no question set names - #426

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/cli-preset-state-key
Sep 25, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/cli-preset-state-key

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

fix(cli): --preset sent the request under a key no question set names

Each shipped question set names the state field it reads, and they are all different — message
for triage, body for email, prompt for guard, post for moderation, request for router.
The CLI sent every one of them as {"text": ...}, a key none of them names, so the model was
asked about a field that was not in the state:

$ laya "Refactor this service to use dependency injection and explain the trade-offs." --preset router
difficulty  : 1.78
domain      : writing (p=0.569)

# the same call with the key the preset's own instructions name
difficulty  : 2.03
domain      : code (p=0.918)

writing is the wrong answer for a refactoring request, and the model had no way to know: the
state contained one field, text, and the question asked about request. Measured on the
English checkpoint across the repository's own example texts, with only the key differing:

preset text the preset's own key
router writing 0.569 code 0.918
router (README's B-tree example) code 0.396 data_analysis 0.557
triage other 0.436 information 0.437
email is_spam 0.731 is_spam 0.505

The email row is the one that matters most: the noul probability crosses 0.5, so a caller
thresholding it got the opposite decision from a request that differed only in a key name.

--predict alone was affected too, because it answers router_questions(), whose instructions
name request.

Deliberate choices tested:

  • A mapping beside PRESETS, not a key derived from the questions. The key could be scraped out of each question's instructions, and the new test does exactly that to check the two agree — but scraping at run time would make a preset's behaviour depend on the wording of its prompt, so the mapping is explicit and the test is what keeps them in step.
  • The routing path is unchanged. route reads the state only for language detection, and lang.analyse is key-invariant, so {"text": ...} stays there and tests/test_cli.py:58, which pins it, is untouched. Verified across eight texts and six key names: the routed checkpoint is identical in every combination.
  • --predict uses request. It answers the router preset, so it takes the router key rather than keeping a special case of its own.
  • The tests assert the key AND that the key is one the preset names. The first catches the bug; the second stops the mapping and the presets from drifting apart, which is the failure this PR is fixing one level up.
# with the single hard-coded key restored
32 passed, 6 failed
  FAIL preset email: state carries the text under 'body': {'text': 'the request'}
  FAIL preset guard: state carries the text under 'prompt': {'text': 'the request'}
  FAIL preset moderation: state carries the text under 'post': {'text': 'the request'}
  FAIL preset router: state carries the text under 'request': {'text': 'the request'}
  FAIL preset triage: state carries the text under 'message': {'text': 'the request'}
  FAIL predict: state carries the text under 'request' (router_questions): {'text': 'the request'}

# with it
$ python tests/test_cli.py            38 passed, 0 failed   (22 before)
$ python tests/test_mcp.py            86 passed, 0 failed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/

Known limitation: --preset and --predict now put the text where the question set reads it,
but nothing warns a caller who passes a state with the wrong key through the library or the MCP
tool surface — the same mismatch is silently accepted there, and the MCP tool description does
not name the field either. That is the same defect one layer out and is not addressed here.

Each shipped question set names the state field it reads, and they are all different — `message`
for triage, `body` for email, `prompt` for guard, `post` for moderation, `request` for router.
The CLI sent every one of them as `{"text": ...}`, a key none of them names, so the model was
asked about a field that was not in the state:

```
$ laya "Refactor this service to use dependency injection and explain the trade-offs." --preset router
difficulty  : 1.78
domain      : writing (p=0.569)

# the same call with the key the preset's own instructions name
difficulty  : 2.03
domain      : code (p=0.918)
```

`writing` is the wrong answer for a refactoring request, and the model had no way to know: the
state contained one field, `text`, and the question asked about `request`. Measured on the
English checkpoint across the repository's own example texts, with only the key differing:

| preset | `text` | the preset's own key |
|---|---|---|
| router | `writing` 0.569 | **`code` 0.918** |
| router (README's B-tree example) | `code` 0.396 | **`data_analysis` 0.557** |
| triage | `other` 0.436 | **`information` 0.437** |
| email | `is_spam` **0.731** | `is_spam` **0.505** |

The email row is the one that matters most: the `noul` probability crosses 0.5, so a caller
thresholding it got the opposite decision from a request that differed only in a key name.

`--predict` alone was affected too, because it answers `router_questions()`, whose instructions
name `request`.

Deliberate choices tested:

- **A mapping beside `PRESETS`, not a key derived from the questions.** The key could be scraped out of each question's `instructions`, and the new test does exactly that to check the two agree — but scraping at run time would make a preset's behaviour depend on the wording of its prompt, so the mapping is explicit and the test is what keeps them in step.
- **The routing path is unchanged.** `route` reads the state only for language detection, and `lang.analyse` is key-invariant, so `{"text": ...}` stays there and `tests/test_cli.py:58`, which pins it, is untouched. Verified across eight texts and six key names: the routed checkpoint is identical in every combination.
- **`--predict` uses `request`.** It answers the router preset, so it takes the router key rather than keeping a special case of its own.
- **The tests assert the key AND that the key is one the preset names.** The first catches the bug; the second stops the mapping and the presets from drifting apart, which is the failure this PR is fixing one level up.

```
# with the single hard-coded key restored
32 passed, 6 failed
  FAIL preset email: state carries the text under 'body': {'text': 'the request'}
  FAIL preset guard: state carries the text under 'prompt': {'text': 'the request'}
  FAIL preset moderation: state carries the text under 'post': {'text': 'the request'}
  FAIL preset router: state carries the text under 'request': {'text': 'the request'}
  FAIL preset triage: state carries the text under 'message': {'text': 'the request'}
  FAIL predict: state carries the text under 'request' (router_questions): {'text': 'the request'}

# with it
$ python tests/test_cli.py            38 passed, 0 failed   (22 before)
$ python tests/test_mcp.py            86 passed, 0 failed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: `--preset` and `--predict` now put the text where the question set reads it,
but nothing warns a caller who passes a state with the wrong key through the library or the MCP
tool surface — the same mismatch is silently accepted there, and the MCP tool description does
not name the field either. That is the same defect one layer out and is not addressed here.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, this is a real correctness fix: the presets were being asked about a field that was not in the state. The test that reads each preset's own instructions to confirm its key is a nice touch, because it stops the mapping and the presets from drifting apart. Merging.

@NandhaKishorM
NandhaKishorM merged commit 72978ad into NandhaKishorM:main Sep 25, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants