Repository navigation
fix(agent): name the question when its definition cannot be answered - #183
Merged
NandhaKishorM merged 1 commit intoSep 23, 2026
Conversation
`system_one` validated nothing about the questions it was given, so a malformed definition surfaced as an exception from inside the library that named neither the question nor the problem (NandhaKishorM#182): `AttributeError: 'NoneType' object has no attribute 'items'` from `render_options`, `KeyError: 'bool'` from the type table, and -- for a choice or score question with no options, which reaches the model with zero markers -- `RuntimeError: selected index k out of range` from `p.topk(1, -1)` inside `DecisionModel.forward`. A caller with twenty questions in one dict had no way to tell which one was wrong. `_check_question` runs per question before anything is rendered or tokenized and raises a ValueError naming the id, the type and the shape expected. It also rejects the one malformed shape that did not raise at all: a `score` question given `criteria` as a dict, where the keys were rendered as levels and every description was silently dropped, so the caller got an answer to a question they did not write. Valid shapes are untouched -- choice as a dict or a list of labels, score as a list, noul with or without a true/false dict, non-string instructions, and odd-but-legal criterion values, all still covered by the rendering checks above. Verification (offline, no weights): * The malformed-shape section added to tests/test_criteria.py drives the real inference path over a tiny from-config encoder, so it needs no checkpoint. Against main it fails 14 checks, each one the old exception class; with the change, 70 passed, 0 failed (34 before). It also asserts the valid shapes still answer -- label, probabilities, score range and legend, noul probability, usage -- so it is not validation-only coverage. * tests/test_router.py, test_shortlist.py, test_decision_model.py, test_packaging.py and test_email.py unchanged and passing; ruff with CI's rule set and python -m compileall clean. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
Owner
|
Thank you for this. The option-less cases crashing inside The criteria suite goes from 34 to 70 checks, and everything passes locally and against the real English checkpoint (answers unchanged on 396 decisions). I agree with rejecting a dict for |
This was referenced Sep 23, 2026
Merged
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
…andhaKishorM#183) `system_one` validated nothing about the questions it was given, so a malformed definition surfaced as an exception from inside the library that named neither the question nor the problem (NandhaKishorM#182): `AttributeError: 'NoneType' object has no attribute 'items'` from `render_options`, `KeyError: 'bool'` from the type table, and -- for a choice or score question with no options, which reaches the model with zero markers -- `RuntimeError: selected index k out of range` from `p.topk(1, -1)` inside `DecisionModel.forward`. A caller with twenty questions in one dict had no way to tell which one was wrong. `_check_question` runs per question before anything is rendered or tokenized and raises a ValueError naming the id, the type and the shape expected. It also rejects the one malformed shape that did not raise at all: a `score` question given `criteria` as a dict, where the keys were rendered as levels and every description was silently dropped, so the caller got an answer to a question they did not write. Valid shapes are untouched -- choice as a dict or a list of labels, score as a list, noul with or without a true/false dict, non-string instructions, and odd-but-legal criterion values, all still covered by the rendering checks above. Verification (offline, no weights): * The malformed-shape section added to tests/test_criteria.py drives the real inference path over a tiny from-config encoder, so it needs no checkpoint. Against main it fails 14 checks, each one the old exception class; with the change, 70 passed, 0 failed (34 before). It also asserts the valid shapes still answer -- label, probabilities, score range and legend, noul probability, usage -- so it is not validation-only coverage. * tests/test_router.py, test_shortlist.py, test_decision_model.py, test_packaging.py and test_email.py unchanged and passing; ruff with CI's rule set and python -m compileall clean. 🤖 Generated with Codebuff Co-authored-by: Nikhi <nikhi@example.com> Co-authored-by: Codebuff <noreply@codebuff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #182.
Agent.system_onevalidated nothing about the questions it was given, so a typo in a definition surfaced as an exception that named neither the question nor the fix — and for a question with no options at all, from inside the model:Why this is more than message quality
The two
RuntimeErrors are not rendering failures: achoice/scorequestion with no options reachesDecisionModel.forwardwith zero markers andp.topk(1, -1)raisesselected index k out of range. That is a crash reported from insidelaya, on a caller's typo, and it is the same call #96/#103 fixed for the one-option case.The rest identify nothing:
'NoneType' object has no attribute 'items'says neither which question nor thatcriteriais missing, andRouter.predictgoes straight here, so this is what an HTTP handler sees — after the state was already tokenized. The one message that does name a question (question %r options exceed head_max_len=%d) is the shape the new errors follow.The fix
Agent._check_question(qid, qdef)runs per question before anything is rendered or tokenized, and raises aValueErrornaming the id, the type and the shape expected:It also rejects the one malformed shape that did not raise on
main: ascorequestion givencriteriaas a dict, where the keys were rendered as levels (level 0: low) and every description was dropped, so the caller got an answer to a question they did not write. That is the single behavior change beyond turning a crash into an error, and it is a one-line revert if you would rather keep accepting it — I flagged it in #182 rather than deciding unilaterally.Valid shapes are untouched:
choiceas a dict or a list of labels,scoreas a list,noulwith or without atrue/falsedict, non-stringinstructions(JSON-encoded as before), and odd-but-legal criterion values (0,False, dicts), which the rendering checks above already cover.Verification
Offline, no weights — the tests build a tiny from-config encoder, so
system_onereally runs (the repo had no way to exercise it without a checkpoint;tests/test_local_e2e.pystill covers the real weights).The new section in
tests/test_criteria.pygoes from 34 passed onmainto 70 passed / 0 failed here, and it is a genuine regression suite rather than a mirror of the implementation: against unfixedagent.pyit fails 14 checks, one per case in the table, each naming the old exception class. It also asserts the valid shapes still answer — choice label and probabilities, score range and legend, noul probability, usage — so validation-only coverage would not pass it.tests/test_router.py,test_shortlist.py,test_decision_model.py,test_packaging.pyandtest_email.pyare unchanged and pass;ruffwith CI's rule set is clean andpython -m compileall -q laya/ tests/passes.I put the regression checks in
test_criteria.pyinstead of a new file so they run in the existing CI lanes without touching.github/workflows/*(my token cannot push workflow changes, and #175 edits those same files).Not addressed: the empty-questions case (
{}) is #144's, and the dictionary-shapedscorecriteria are now an error rather than silently accepted, which is the call I flagged above.🤖 Generated with Codebuff
Co-Authored-By: Codebuff noreply@codebuff.com