Skip to content

fix(agent): name the question when its definition cannot be answered - #183

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
devloper961-maker:fix/malformed-question-errors
Sep 23, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
devloper961-maker:fix/malformed-question-errors

Conversation

@devloper961-maker

Copy link
Copy Markdown
Contributor

Fixes #182.

Agent.system_one validated nothing about the questions it was given, so a typo in a definition surfaced as an exception that named neither the question nor the fix — and for a question with no options at all, from inside the model:

tiny from-config encoder, no checkpoint, Agent.system_one(state, {"q": qdef})

qdef                                                  raised on main
{"type": "choice", "instructions": "..."}             AttributeError: 'NoneType' object has no attribute 'items'  [common.py:38]
choice with criteria: None                            same
choice with criteria: {}                              RuntimeError: selected index k out of range                 [common.py:131]
choice with criteria: ("billing", "tech")             AttributeError: 'tuple' object has no attribute 'items'
score without criteria                                TypeError: 'NoneType' object is not iterable
score with criteria: []                               RuntimeError: selected index k out of range                 [common.py:131]
noul with criteria: ["a", "b"]                        AttributeError: 'list' object has no attribute 'get'
noul with criteria: "spam?"                           AttributeError: 'str' object has no attribute 'get'
{"type": "bool", "instructions": "..."}               KeyError: 'bool'
{"type": "noul"}                                      KeyError: 'instructions'

with this PR                                          ValueError: question 'q': …

Why this is more than message quality

The two RuntimeErrors are not rendering failures: a choice/score question with no options reaches DecisionModel.forward with zero markers and p.topk(1, -1) raises selected index k out of range. That is a crash reported from inside laya, on a caller's typo, and it is the same call #96/#103 fixed for the one-option case.

The rest identify nothing: 'NoneType' object has no attribute 'items' says neither which question nor that criteria is missing, and Router.predict goes straight here, so this is what an HTTP handler sees — after the state was already tokenized. The one message that does name a question (question %r options exceed head_max_len=%d) is the shape the new errors follow.

The fix

Agent._check_question(qid, qdef) runs per question before anything is rendered or tokenized, and raises a ValueError naming the id, the type and the shape expected:

question 'q': a choice question takes 'criteria' as a dict of label -> description, or a list of labels
question 'q': a score question takes 'criteria' as a list of level descriptions, index 0 first
question 'q': a noul question takes 'criteria' as a dict with optional 'true'/'false' descriptions, or omits it
question 'q': unknown type 'bool'; use one of ['choice', 'noul', 'score']
question 'q': no 'instructions'; add the text the model should answer

It also rejects the one malformed shape that did not raise on main: a score question given criteria as a dict, where the keys were rendered as levels (level 0: low) and every description was dropped, so the caller got an answer to a question they did not write. That is the single behavior change beyond turning a crash into an error, and it is a one-line revert if you would rather keep accepting it — I flagged it in #182 rather than deciding unilaterally.

Valid shapes are untouched: choice as a dict or a list of labels, score as a list, noul with or without a true/false dict, non-string instructions (JSON-encoded as before), and odd-but-legal criterion values (0, False, dicts), which the rendering checks above already cover.

Verification

Offline, no weights — the tests build a tiny from-config encoder, so system_one really runs (the repo had no way to exercise it without a checkpoint; tests/test_local_e2e.py still covers the real weights).

The new section in tests/test_criteria.py goes from 34 passed on main to 70 passed / 0 failed here, and it is a genuine regression suite rather than a mirror of the implementation: against unfixed agent.py it fails 14 checks, one per case in the table, each naming the old exception class. It also asserts the valid shapes still answer — choice label and probabilities, score range and legend, noul probability, usage — so validation-only coverage would not pass it.

tests/test_router.py, test_shortlist.py, test_decision_model.py, test_packaging.py and test_email.py are unchanged and pass; ruff with CI's rule set is clean and python -m compileall -q laya/ tests/ passes.

I put the regression checks in test_criteria.py instead of a new file so they run in the existing CI lanes without touching .github/workflows/* (my token cannot push workflow changes, and #175 edits those same files).

Not addressed: the empty-questions case ({}) is #144's, and the dictionary-shaped score criteria are now an error rather than silently accepted, which is the call I flagged above.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff noreply@codebuff.com

`system_one` validated nothing about the questions it was given, so a malformed
definition surfaced as an exception from inside the library that named neither
the question nor the problem (NandhaKishorM#182): `AttributeError: 'NoneType' object has no
attribute 'items'` from `render_options`, `KeyError: 'bool'` from the type
table, and -- for a choice or score question with no options, which reaches the
model with zero markers -- `RuntimeError: selected index k out of range` from
`p.topk(1, -1)` inside `DecisionModel.forward`. A caller with twenty questions
in one dict had no way to tell which one was wrong.

`_check_question` runs per question before anything is rendered or tokenized and
raises a ValueError naming the id, the type and the shape expected. It also
rejects the one malformed shape that did not raise at all: a `score` question
given `criteria` as a dict, where the keys were rendered as levels and every
description was silently dropped, so the caller got an answer to a question they
did not write.

Valid shapes are untouched -- choice as a dict or a list of labels, score as a
list, noul with or without a true/false dict, non-string instructions, and
odd-but-legal criterion values, all still covered by the rendering checks above.

Verification (offline, no weights):
* The malformed-shape section added to tests/test_criteria.py drives the real
  inference path over a tiny from-config encoder, so it needs no checkpoint.
  Against main it fails 14 checks, each one the old exception class; with the
  change, 70 passed, 0 failed (34 before). It also asserts the valid shapes
  still answer -- label, probabilities, score range and legend, noul
  probability, usage -- so it is not validation-only coverage.
* tests/test_router.py, test_shortlist.py, test_decision_model.py,
  test_packaging.py and test_email.py unchanged and passing; ruff with CI's rule
  set and python -m compileall clean.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for this. The option-less cases crashing inside DecisionModel.forward were the real bug, and the error messages that name the question are a genuine usability win for anyone with twenty questions in one call.

The criteria suite goes from 34 to 70 checks, and everything passes locally and against the real English checkpoint (answers unchanged on 396 decisions). I agree with rejecting a dict for score criteria. Merging. #209, #144 and #194 touch the same area, so I'll point them here.

@NandhaKishorM
NandhaKishorM merged commit fcb2cd9 into NandhaKishorM:main Sep 23, 2026
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…andhaKishorM#183)

`system_one` validated nothing about the questions it was given, so a malformed
definition surfaced as an exception from inside the library that named neither
the question nor the problem (NandhaKishorM#182): `AttributeError: 'NoneType' object has no
attribute 'items'` from `render_options`, `KeyError: 'bool'` from the type
table, and -- for a choice or score question with no options, which reaches the
model with zero markers -- `RuntimeError: selected index k out of range` from
`p.topk(1, -1)` inside `DecisionModel.forward`. A caller with twenty questions
in one dict had no way to tell which one was wrong.

`_check_question` runs per question before anything is rendered or tokenized and
raises a ValueError naming the id, the type and the shape expected. It also
rejects the one malformed shape that did not raise at all: a `score` question
given `criteria` as a dict, where the keys were rendered as levels and every
description was silently dropped, so the caller got an answer to a question they
did not write.

Valid shapes are untouched -- choice as a dict or a list of labels, score as a
list, noul with or without a true/false dict, non-string instructions, and
odd-but-legal criterion values, all still covered by the rendering checks above.

Verification (offline, no weights):
* The malformed-shape section added to tests/test_criteria.py drives the real
  inference path over a tiny from-config encoder, so it needs no checkpoint.
  Against main it fails 14 checks, each one the old exception class; with the
  change, 70 passed, 0 failed (34 before). It also asserts the valid shapes
  still answer -- label, probabilities, score range and legend, noul
  probability, usage -- so it is not validation-only coverage.
* tests/test_router.py, test_shortlist.py, test_decision_model.py,
  test_packaging.py and test_email.py unchanged and passing; ruff with CI's rule
  set and python -m compileall clean.

🤖 Generated with Codebuff

Co-authored-by: Nikhi <nikhi@example.com>
Co-authored-by: Codebuff <noreply@codebuff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A malformed question crashes three frames down, and an option-less one crashes inside the decision head

2 participants