Repository navigation
fix(agent): comprehensive input validation for questions, instructions, and state (#707) - #716
Swatantra-66 wants to merge 1 commit into
Conversation
Bruce-Yii
left a comment
There was a problem hiding this comment.
This is the one I would take from the pair, and the reason is structural rather than stylistic: it is a strict superset of #708, and it is the only one of the two that closes an issue. Fixes #707 points at a real open issue from @debjit02-rgb asking for exactly this class of work, while #708 closes nothing. Whoever picks between them is really picking which one carries #707.
I checked the behaviour rather than the diff, by exercising both this head and main across the input classes the change touches. Thirteen of them behave differently, and all thirteen are improvements:
typeas a list or dict went fromTypeError: unhashable typeraised from inside the membership test, to a cleanValueErrornaming the question and listing the valid types. This is the case that matters most, because an unhashabletypecurrently escapes as a server error rather than a client error.- a
None, empty, whitespace-only or non-str/intquestion id went from being accepted to being refused with a message that names it. instructionsasNone,"",[]or{}went from being accepted to being refused.predict_longwith aNonestate, orquestionsthat is not a dict, went from proceeding into the model path to being refused up front.
The suites hold up. At this head, test_criteria.py 182 passed, test_onnx_batch.py 57 passed, test_serve.py 61 passed. Against main's source with only the test files from this head, they go red in the places they should: 5 named failures in test_onnx_batch, 8 failures in test_serve including the parametrised named-422 cases for both malformed instructions and malformed question ids, and 8 named failures in test_criteria that include the router-side equivalents, so the three surfaces are genuinely being held to the same behaviour rather than just Agent.
Two things worth knowing, neither of which I think should hold this up.
The predict_long guards raise TypeError, while every guard in _check_question raises ValueError — and laya/serve.py keys its 422 mapping on except ValueError specifically, sending everything else to 500 "inference failed". That is not a live problem today, because predict_long is not reachable from either entry point: serve.py calls router.predict, and the MCP tools call router.predict or router.predict_batch. But the two predict_long inputs are also pre-empted at the HTTP boundary by the 400s on state and questions, so those guards are currently protecting a path nothing routes to. If predict_long is ever exposed through serve, or the pre-validation is refactored away, these become 500s where a client error was intended. ValueError would keep the contract consistent and cost nothing today.
Second, the instructions guard accepts int and float (isinstance(ins, (str, dict, list, int, float))) but the message it raises says must be a string, dict, or list. A caller passing an integer passes validation and is then told integers are not allowed if they hit a neighbouring check. Either widen the message or narrow the tuple.
|
Thanks for the thorough review and testing, @Bruce-Yii! I've addressed both suggestions in commit
All CI checks and test suites ( |
…handling fix(agent): comprehensive input validation for questions, instructions, and state (#707) # Conflicts: # laya/onnx_agent.py # tests/test_criteria.py # tests/test_onnx_batch.py # tests/test_serve.py
|
Passed checks and merged. Thank you @Swatantra-66! It ships in 0.3.22. |
|
"Thanks @NandhaKishorM! Happy to see it merged in 626bedb and released in v0.3.22. |
Bruce-Yii
left a comment
There was a problem hiding this comment.
Both non-blocking notes from my review are now implemented, and the new head is strictly better than the one I approved. Re-approving on 104e7bf2.
Note 1 — predict_long raising TypeError where the contract is ValueError
Fixed across all three surfaces, consistently:
Agent.predict_long—state is Noneand non-dictquestionsnow raiseValueErrorONNXAgent.predict_long— same two guards, same exception typeRouter.routeandRouter.predict— non-dictquestionsnow raiseValueError
That closes the gap I flagged. It matters because laya/serve.py maps its 422 onto except ValueError: had predict_long ever been wired into the serve path, those two guards would have surfaced as a 500 instead of a 422. Every entry point now fails the same way, so the mapping holds whichever one is called.
Note 2 — the instructions guard accepting int/float while the message said otherwise
_check_question was (str, dict, list, int, float) against a message reading "must be a string, dict, or list" — self-contradictory. The guard is now (str, dict, list), so the code and the message agree, and a numeric instructions is refused rather than silently stringified.
Two tests pin it, and both are new rather than adjusted:
tests/test_criteria.py:342 ("instructions is integer", {"type": "noul", "instructions": 123})
tests/test_serve.py:698 @pytest.mark.parametrize("bad_ins", [None, "", " ", [], 123])
tests/test_criteria.py → 184 passed, 0 failed. tests/test_serve.py → 62 passed, 1 deselected (the deselected one is the socket test I always bound out on this host, not a failure).
Neither change is load-bearing on its own, and both are the kind that quietly rot — a TypeError/ValueError split across three surfaces is exactly the thing a future refactor re-introduces. Worth having pinned while it is fresh.
Re-approving.
|
Thanks for the thorough re-review and approval, @Bruce-Yii! Glad to hear the |
|
Status note, because this PR is in a confusing state and that is on me. The first round shipped in 0.3.22, which is why the error messages and the question-id and instructions guards are already on I tried to take that round in this pass and backed it out, because it collides with two pins that disagree with each other:
In other words the exception type for "questions is not a dict" is pinned two different ways in the tree, and completing your change means picking one and retiring the other pin. That is a contract decision rather than a merge-time fix, so I am not going to make it inside a merge commit. The direction you and Bruce were arguing for is the right one: Thanks for the work already in 0.3.22, and sorry for leaving this half-landed. |
104e7bf to
9ea668b
Compare
9ea668b to
e83288c
Compare
|
Thanks for the clear roadmap and guidance, @NandhaKishorM! I've picked this up and finished the second round across all five sites in commit
All CI checks (Python 3.10–3.13, Windows, TypeScript, ONNX, and Docs strict build) are completely green. |
|
Closing this, because the main change in it runs against the convention the rest of the library follows, and that convention is deliberate. Reading the diff, almost all of it is The split in this codebase is the ordinary Python one: a wrong type is a Several suites pin that distinction, which is why #1089 is also held: the same conversion there turned five suite invocations red, including One piece of it is a real finding and worth its own PR. Narrowing the accepted instruction types from r.predict("refund please", {"q": {"type": "noul", "instructions": 42}})
# -> {'type': 'noul', 'noul': 0.2026, 'confidence': 0.7974, ...}An integer instruction is accepted and answered with full confidence. The number is rendered into the sequence and the model answers whatever that reads as, so the caller gets a confident answer to a question they did not ask. That is worth refusing, as a Thank you for the issue work on #707 either way. The validation gap was real; it is the exception taxonomy that has to stay as it is. |
What
Add comprehensive input validation across
Agent,ONNXAgent, andRouterbefore tokenization, routing, or inference:None, empty strings, or non-scalar keys with descriptiveValueError("question id must be a non-empty string, got %r" % (qid,)).[],{}) safely in_check_questionbefore looking up inQTYPES."instructions"is present, notNone, non-empty (for strings, lists, or dicts), and of a valid supported type (str,dict,list), raising a clearValueErrornaming the question and what to fix.ONNXAgent&Router:ONNXAgent.predict_longandAgent.predict_long: reject non-dictquestionsandNonestates with cleanValueErrors, ensuring consistent HTTP 422 client error mapping underserve.py.Router.routeandRouter.predict: validate thatquestionsis adictandstateis notNonewithValueErrorbefore routing or tokenization.serve.pycatchingValueErrorfor HTTP 422 while allowing any internalTypeError(e.g. in hooks or model inference) to bubble to the HTTP 500 handler without leaking internal details.Why
Malformed or incomplete inputs previously surfaced as unhelpful low-level exceptions:
questions = {"q": {"type": []}}raisedTypeError: unhashable type: 'list'from dictionary lookup.questions = {"q": {"type": "noul", "instructions": None}}or""was silently serialized into"null"or empty prompt strings and sent to the model.questions = {None: ...}or""produced invalid answer dictionaries or crashed serialization.predict_long(None, ...)silently tokenized"null"across multiple windows.questionsinONNXAgentraised bareAttributeError: 'list' object has no attribute 'keys'.All invalid decision inputs now produce clear, descriptive error messages and return 422 over HTTP.
Fixes #707.
How it was verified
Added comprehensive regression test suites across SDK, ONNX, and HTTP serve layers:
tests/test_criteria.py: verified that malformed types, missing/None/empty/whitespace/invalid instructions (including integers), and invalid question IDs (None,"", whitespace) raiseValueErrornaming the question.tests/test_onnx_batch.py: verified thatONNXAgent.predict_batchrejects bad question types, malformed instructions, non-dict questions, andNonestates before session runs.tests/test_serve.py: verified that malformed question types, bad instructions (including non-prose integers), and invalid question IDs return HTTP 422 with the exact descriptive message.tests/test_hooks_api.pypasses (359 passed, 0 failed).Checklist
main