Skip to content

fix(agent): comprehensive input validation for questions, instructions, and state (#707) - #716

Closed
Swatantra-66 wants to merge 1 commit into
NandhaKishorM:mainfrom
Swatantra-66:fix/input-validation-error-handling
Closed

Swatantra-66 wants to merge 1 commit into
NandhaKishorM:mainfrom
Swatantra-66:fix/input-validation-error-handling

Conversation

@Swatantra-66

@Swatantra-66 Swatantra-66 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What

Add comprehensive input validation across Agent, ONNXAgent, and Router before tokenization, routing, or inference:

  1. Question ID Validation: Reject None, empty strings, or non-scalar keys with descriptive ValueError("question id must be a non-empty string, got %r" % (qid,)).
  2. Question Type Validation: Reject non-string and unhashable question types (e.g. [], {}) safely in _check_question before looking up in QTYPES.
  3. Question Instructions Validation: Validate that "instructions" is present, not None, non-empty (for strings, lists, or dicts), and of a valid supported type (str, dict, list), raising a clear ValueError naming the question and what to fix.
  4. Questions & State Validation in ONNXAgent & Router:
    • ONNXAgent.predict_long and Agent.predict_long: reject non-dict questions and None states with clean ValueErrors, ensuring consistent HTTP 422 client error mapping under serve.py.
    • Router.route and Router.predict: validate that questions is a dict and state is not None with ValueError before routing or tokenization.
  5. Preserve HTTP 422 vs 500 Boundary: Keep serve.py catching ValueError for HTTP 422 while allowing any internal TypeError (e.g. in hooks or model inference) to bubble to the HTTP 500 handler without leaking internal details.

Why

Malformed or incomplete inputs previously surfaced as unhelpful low-level exceptions:

  • questions = {"q": {"type": []}} raised TypeError: unhashable type: 'list' from dictionary lookup.
  • questions = {"q": {"type": "noul", "instructions": None}} or "" was silently serialized into "null" or empty prompt strings and sent to the model.
  • questions = {None: ...} or "" produced invalid answer dictionaries or crashed serialization.
  • predict_long(None, ...) silently tokenized "null" across multiple windows.
  • Non-dict questions in ONNXAgent raised bare AttributeError: 'list' object has no attribute 'keys'.

All invalid decision inputs now produce clear, descriptive error messages and return 422 over HTTP.

Fixes #707.

How it was verified

Added comprehensive regression test suites across SDK, ONNX, and HTTP serve layers:

  1. tests/test_criteria.py: verified that malformed types, missing/None/empty/whitespace/invalid instructions (including integers), and invalid question IDs (None, "", whitespace) raise ValueError naming the question.
  2. tests/test_onnx_batch.py: verified that ONNXAgent.predict_batch rejects bad question types, malformed instructions, non-dict questions, and None states before session runs.
  3. tests/test_serve.py: verified that malformed question types, bad instructions (including non-prose integers), and invalid question IDs return HTTP 422 with the exact descriptive message.
  4. Verified CI gates:
    ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
    python -m compileall -q laya/ tests/
  5. Verified public API contract:
    tests/test_hooks_api.py passes (359 passed, 0 failed).

Checklist

  • Focused on one change
  • Based on the latest main
  • All tests and CI gates pass locally
  • No breaking public API signature changes

@Swatantra-66 Swatantra-66 changed the title fix(agent): comprehensive input validation for questions, instructions, and state fix(agent): comprehensive input validation for questions, instructions, and state (#707) Sep 29, 2026

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one I would take from the pair, and the reason is structural rather than stylistic: it is a strict superset of #708, and it is the only one of the two that closes an issue. Fixes #707 points at a real open issue from @debjit02-rgb asking for exactly this class of work, while #708 closes nothing. Whoever picks between them is really picking which one carries #707.

I checked the behaviour rather than the diff, by exercising both this head and main across the input classes the change touches. Thirteen of them behave differently, and all thirteen are improvements:

  • type as a list or dict went from TypeError: unhashable type raised from inside the membership test, to a clean ValueError naming the question and listing the valid types. This is the case that matters most, because an unhashable type currently escapes as a server error rather than a client error.
  • a None, empty, whitespace-only or non-str/int question id went from being accepted to being refused with a message that names it.
  • instructions as None, "", [] or {} went from being accepted to being refused.
  • predict_long with a None state, or questions that is not a dict, went from proceeding into the model path to being refused up front.

The suites hold up. At this head, test_criteria.py 182 passed, test_onnx_batch.py 57 passed, test_serve.py 61 passed. Against main's source with only the test files from this head, they go red in the places they should: 5 named failures in test_onnx_batch, 8 failures in test_serve including the parametrised named-422 cases for both malformed instructions and malformed question ids, and 8 named failures in test_criteria that include the router-side equivalents, so the three surfaces are genuinely being held to the same behaviour rather than just Agent.

Two things worth knowing, neither of which I think should hold this up.

The predict_long guards raise TypeError, while every guard in _check_question raises ValueError — and laya/serve.py keys its 422 mapping on except ValueError specifically, sending everything else to 500 "inference failed". That is not a live problem today, because predict_long is not reachable from either entry point: serve.py calls router.predict, and the MCP tools call router.predict or router.predict_batch. But the two predict_long inputs are also pre-empted at the HTTP boundary by the 400s on state and questions, so those guards are currently protecting a path nothing routes to. If predict_long is ever exposed through serve, or the pre-validation is refactored away, these become 500s where a client error was intended. ValueError would keep the contract consistent and cost nothing today.

Second, the instructions guard accepts int and float (isinstance(ins, (str, dict, list, int, float))) but the message it raises says must be a string, dict, or list. A caller passing an integer passes validation and is then told integers are not allowed if they hit a neighbouring check. Either widen the message or narrow the tuple.

@Swatantra-66

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review and testing, @Bruce-Yii!

I've addressed both suggestions in commit 104e7bf:

  1. Exception Consistency: Switched predict_long (across Agent and ONNXAgent) and Router state/question guards to raise ValueError instead of TypeError, ensuring consistent HTTP 422 mapping under serve.py.
  2. Instructions Type Guard: Narrowed isinstance(ins, ...) to (str, dict, list) to match the error message and added test coverage in test_criteria.py and test_serve.py.

All CI checks and test suites (test_criteria.py, test_onnx_batch.py, test_serve.py) continue to pass cleanly.

NandhaKishorM added a commit that referenced this pull request Sep 29, 2026
…handling

fix(agent): comprehensive input validation for questions, instructions, and state (#707)

# Conflicts:
#	laya/onnx_agent.py
#	tests/test_criteria.py
#	tests/test_onnx_batch.py
#	tests/test_serve.py
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @Swatantra-66! It ships in 0.3.22.

@Swatantra-66

Copy link
Copy Markdown
Contributor Author

"Thanks @NandhaKishorM! Happy to see it merged in 626bedb and released in v0.3.22.

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both non-blocking notes from my review are now implemented, and the new head is strictly better than the one I approved. Re-approving on 104e7bf2.

Note 1 — predict_long raising TypeError where the contract is ValueError

Fixed across all three surfaces, consistently:

  • Agent.predict_long — state is None and non-dict questions now raise ValueError
  • ONNXAgent.predict_long — same two guards, same exception type
  • Router.route and Router.predict — non-dict questions now raise ValueError

That closes the gap I flagged. It matters because laya/serve.py maps its 422 onto except ValueError: had predict_long ever been wired into the serve path, those two guards would have surfaced as a 500 instead of a 422. Every entry point now fails the same way, so the mapping holds whichever one is called.

Note 2 — the instructions guard accepting int/float while the message said otherwise

_check_question was (str, dict, list, int, float) against a message reading "must be a string, dict, or list" — self-contradictory. The guard is now (str, dict, list), so the code and the message agree, and a numeric instructions is refused rather than silently stringified.

Two tests pin it, and both are new rather than adjusted:

tests/test_criteria.py:342   ("instructions is integer", {"type": "noul", "instructions": 123})
tests/test_serve.py:698      @pytest.mark.parametrize("bad_ins", [None, "", "   ", [], 123])

tests/test_criteria.py → 184 passed, 0 failed. tests/test_serve.py → 62 passed, 1 deselected (the deselected one is the socket test I always bound out on this host, not a failure).

Neither change is load-bearing on its own, and both are the kind that quietly rot — a TypeError/ValueError split across three surfaces is exactly the thing a future refactor re-introduces. Worth having pinned while it is fresh.

Re-approving.

@Swatantra-66

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough re-review and approval, @Bruce-Yii! Glad to hear the ValueError mapping and instruction guard tests are all passing cleanly.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Status note, because this PR is in a confusing state and that is on me.

The first round shipped in 0.3.22, which is why the error messages and the question-id and instructions guards are already on main. What never landed is the second round you pushed in 104e7bf after @Bruce-Yii's re-review: the TypeError to ValueError change across Agent.predict_long, ONNXAgent.predict_long and the Router guards, plus the narrowed isinstance(ins, (str, dict, list)).

I tried to take that round in this pass and backed it out, because it collides with two pins that disagree with each other:

  • tests/test_predict_long.py (from fix(agent): predict_long skips part of the document and reports that it read it #689) requires predict_long and system_one to raise the same error for the same malformed questions. Your change moves predict_long to ValueError but leaves Agent.predict_batch, which system_one goes through, on TypeError, so the pin goes red.
  • tests/test_onnx_batch.py pins TypeError explicitly for ONNXAgent.predict_batch's questions guard, so making that one ValueError to satisfy the first pin turns the second one red.

In other words the exception type for "questions is not a dict" is pinned two different ways in the tree, and completing your change means picking one and retiring the other pin. That is a contract decision rather than a merge-time fix, so I am not going to make it inside a merge commit.

The direction you and Bruce were arguing for is the right one: laya/serve.py maps ValueError to 422 and a TypeError escapes as a 500, so a caller's malformed questions should be a 422 on every surface. If you want to finish it, the scope is all five sites that carry that message (Agent.predict_batch, Agent.predict_long, ONNXAgent.predict_batch, ONNXAgent.predict_long, both Router guards) moved together, tests/test_onnx_batch.py's pin updated in the same commit with a line saying why, and tests/test_predict_long.py left alone so it keeps proving the two paths agree. Add a laya-serve check that the malformed body is a 422 and the change documents its own reason.

Thanks for the work already in 0.3.22, and sorry for leaving this half-landed.

@Swatantra-66
Swatantra-66 force-pushed the fix/input-validation-error-handling branch from 9ea668b to e83288c Compare October 1, 2026 17:46
@Swatantra-66

Copy link
Copy Markdown
Contributor Author

Thanks for the clear roadmap and guidance, @NandhaKishorM!

I've picked this up and finished the second round across all five sites in commit e83288c:

  1. Exception Consistency: Moved the questions dict guard to ValueError across all five sites (Agent.predict_batch, Agent.predict_long, ONNXAgent.predict_batch, ONNXAgent.predict_long, and both Router entry points) so a caller's malformed questions consistently maps to HTTP 422 under laya-serve rather than leaking as a 500.
  2. Reconciling the Test Pins:
    • Updated tests/test_onnx_batch.py to pin ValueError with a comment explaining the HTTP 422 mapping rationale.
    • Updated tests/test_empty_questions.py to assert ValueError for non-dict questions while preserving TypeError for state=None.
    • Kept tests/test_predict_long.py untouched, confirming predict_long and system_one remain in complete agreement.
  3. HTTP 422 Test: Added test_malformed_question_dict_is_a_named_422 in tests/test_serve.py.
  4. Rebase: Cleanly rebased on top of main (0.3.23).

All CI checks (Python 3.10–3.13, Windows, TypeScript, ONNX, and Docs strict build) are completely green.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Closing this, because the main change in it runs against the convention the rest of the library follows, and that convention is deliberate.

Reading the diff, almost all of it is TypeError becoming ValueError:

-            raise TypeError("state must not be None; pass a string, dict, or list")
+            raise ValueError("state must not be None; pass a string, dict, or list")
-            raise TypeError("questions must be a dict of question id -> definition, got %s" ...)
+            raise ValueError("questions must be a dict of question id -> definition, got %s" ...)

The split in this codebase is the ordinary Python one: a wrong type is a TypeError and a wrong value is a ValueError. Today's main already refuses both of your cases, on the right side of that line:

state=None       -> TypeError: state must not be None; pass a string, dict, or list
instructions=''  -> ValueError: question 'q': 'instructions' must not be empty; add the text

Several suites pin that distinction, which is why #1089 is also held: the same conversion there turned five suite invocations red, including tests/test_batch.py asserting TypeError for a bare string and tests/test_hooks.py the same for a bare dict. So this is not a rebase away from merging; it is a change I would have to revert.

One piece of it is a real finding and worth its own PR. Narrowing the accepted instruction types from (str, dict, list, int, float) to (str, dict, list) closes something I confirmed on main just now:

r.predict("refund please", {"q": {"type": "noul", "instructions": 42}})
# -> {'type': 'noul', 'noul': 0.2026, 'confidence': 0.7974, ...}

An integer instruction is accepted and answered with full confidence. The number is rendered into the sequence and the model answers whatever that reads as, so the caller gets a confident answer to a question they did not ask. That is worth refusing, as a TypeError naming the question id and the type, and it is a five-line change against today's main with a test in tests/test_criteria.py. If you open that, I will take it.

Thank you for the issue work on #707 either way. The validation gap was real; it is the exception taxonomy that has to stay as it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature]: Improve error handling for invalid decision inputs

3 participants