Skip to content

feat(mcp): expose core's min_confidence abstention control on the decision tools - #657

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-mcp-min-confidence
Sep 29, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-mcp-min-confidence

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

min_confidence -- core's opt-in abstention gate from #361 -- is now a keyword on the MCP single-call decision tools (laya_predict, laya_shortlist, laya_preset, laya_decide). It is validated in the tool layer and forwarded to the answering forward pass; on laya_decide a field whose answer falls below the threshold comes back null in values while its raw confidence survives.

Why

min_confidence is already a validated keyword of Agent.predict, Router.predict/Router.predict_batch and laya.decide on main -- the 0.3.21 tagline itself ships "opt-in abstention ... on every surface". But grep -c min_confidence laya/mcp/ is 0: the MCP layer never forwarded it. #567 carried every other per-call control (task/lang/max_len/head_max_len) across to these tools, so a client over MCP could pin a checkpoint and size its token budget, yet could not ask Laya to abstain on an unsure answer -- the documented capability had no reachable lever on this surface.

Concrete use case: a caller that pipes laya_decide's values straight into an action (route a ticket, set a priority) needs "not confident" to be distinguishable from a confident pick, without parsing the confidence map itself. Over MCP that was impossible before this change.

How it was verified

python tests/test_mcp.py            # 746 passed, 0 failed
python tests/test_hooks_api.py      # 387 passed, 0 failed
python tests/test_structured_api.py # 35 passed, 0 failed
python tests/test_confidence.py     # 59 passed, 0 failed
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!
python -m compileall -q laya/ tests/

Before/after measured on main (9d95567): laya.mcp.tools.laya_predict params [..., 'head_max_len', 'router', 'agent'] (no min_confidence) -> now [..., 'head_max_len', 'min_confidence', ...], while core laya.decide/Router.predict/Router.predict_batch already carry it -- so this closes a surface gap, not a core behavior change.

test_min_confidence_control() drives it at the seam: a recording ControlRouter proves the kwarg is forwarded only when set (a no-abstention call reaches core exactly as before), laya_decide(min_confidence=0.9) keeps the 0.94-confidence field and nulls the 0.8/0.89 ones with confidence intact and byte-parity to laya.decide(..., min_confidence=0.9), and bad values (-0.1/1.1/True/"0.5"/nan/inf) become invalid_min_confidence before any forward pass runs. Three faithful mutants (drop the predict forward, drop the decide forward, neuter the validator) were each killed by a named assertion.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Overlaps note: this touches laya/mcp/tools.py, laya/mcp/server.py and tests/test_mcp.py -- the same files my open #640 (batch-request token budget) and #636 (shortlist k default) touch, but in different functions (laya_predict/laya_shortlist/laya_preset/laya_decide vs the *_batch and k surfaces), so they merge independently. No open PR claims MCP min_confidence; #644 is the laya-ts port of #361, a different surface.

Fixes #

…ision tools

min_confidence (NandhaKishorM#361) is a validated keyword of Agent.predict, Router.predict/
predict_batch and laya.decide on main, but the MCP layer never forwarded it: a
client could pin a checkpoint and size its token budget over the wire yet could
not ask Laya to abstain on an unsure answer.

laya_predict, laya_shortlist, laya_preset and laya_decide now take it and
forward it to core, validated in the tool layer via laya.confidence
.check_min_confidence so a bad value is an invalid_min_confidence ToolError
rather than a forward pass that never runs. Only set values are forwarded, so a
no-abstention call stays byte-identical to one before the keyword existed. On
laya_decide a field below the threshold comes back null in values while its
confidence survives -- parity with laya.decide(min_confidence=...).
@aashish254

aashish254 commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author

The only failing required check here is dependency CVEs, and it can't be cleared from a PR branch, so I don't want it read as a problem with this change.

security.yml builds requirements-audit.txt from the pyproject.toml dependencies plus every optional extra, then runs pip-audit --strict, which fails on any finding and has no ignore list. Run 36377656414 reports 7 rows over 3 packages:

  • json-repair 0.25.2, GHSA-xf7x-x43h-rpqh, fix listed as 0.60.1
  • chromadb 1.1.1, PYSEC-2026-311 (listed twice), -3813, -3814, -3815, no fix version listed
  • nltk 3.10.3, PYSEC-2026-3740, no fix version listed

This PR changes no dependency file, and all three packages arrive transitively through the crewai / langgraph / llama-index-core extras, so there's no pin here to bump. Even if there were, 6 of the 7 rows have no fix to move to. Since pip-audit checks the live OSV data, the same tree that passed on main on 2026-09-25 (run 36130226758) fails today only because those advisories were published since, so any PR opened from current main fails the same way.

Your call on the fix, since I'd rather not loosen a security gate from a contributor PR: either add ignore entries for the unpatched IDs and bump json-repair to >=0.60.1, or hold the gate until patched chromadb / nltk land. I can send a small separate PR against security.yml for whichever you prefer.

@NandhaKishorM
NandhaKishorM merged commit 263e314 into NandhaKishorM:main Sep 29, 2026
22 of 23 checks passed
NandhaKishorM pushed a commit that referenced this pull request Sep 29, 2026
`env_device()` is what `laya/mcp/server.py` hands to `Router(device=...)`, and it returned
the environment value with only surrounding whitespace stripped. `resolve_device()`
lower-cased the same value, and that is what `laya_status` reports.

torch's device parser is case-sensitive. Measured with torch 2.13.0+cpu:

```
LAYA_DEVICE   env_device   reported    torch.device(env_device)   torch.device(reported)
'CPU'         CPU          cpu         REJECTED                    ok
'CUDA'        CUDA         cuda        REJECTED                    ok
'CUDA:0'      CUDA:0       cuda:0      REJECTED                    ok
'cpu'         cpu          cpu         ok                          ok
'cuda:0'      cuda:0       cuda:0      ok                          ok
```

So for one environment, `laya_status` reported a device as configured and working while the
Router was being built with a value torch refuses. The failure surfaced as
`ToolError("internal_error", "router construction failed: ...")`, which classifies a
misconfigured server environment as a server fault rather than a bad setting.

`env_device()` now normalises the device type to lower case, and `resolve_device()` returns
that same string instead of lower-casing a second time, so the value handed to torch and the
value reported are one string by construction. Splitting on the first colon keeps the index
intact -- `CUDA:0 -> cuda:0` -- because torch wants the type lower-cased and does not care
about the index.

Unset and whitespace-only still mean "not asked for" and still let `resolve_device()`
auto-detect; that difference is the point of the two functions and is unchanged. Blank
input, padding, and an explicit `resolve_device(force=...)` keep their behaviour.

The existing test asserted the old contract by name (`device/env_raw_for_torch`, with a
comment saying the value goes "verbatim to torch"). That contract is what rejected
`CUDA`, so the assertion is updated to the new one and extended to cover the case that
motivated it, rather than left passing on a value that happens to be lower-case already.

`README.md:929` documented the old wording ("the value is passed straight to torch") and now
states the normalisation.

```
python tests/test_mcp.py                      683 passed, 0 failed  (was 679)
python -m compileall -q laya tests
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
git diff --check
```

Not run here, and not in this diff: `tests/test_serve.py` exceeded a 400 s timeout in this
environment, and `tests/test_mcp_local_e2e.py` needs a real checkpoint from the Hub
(`ModuleNotFoundError` for `laya` with no install metadata). Both are pre-existing
environment limits rather than results, so I am flagging them instead of implying they passed.

`laya/mcp/device.py` is touched by no open PR. Four open PRs edit `tests/test_mcp.py`
(#657, #640, #636, #445) and eighteen edit `README.md`; I read the patches and none adds or
removes the lines changed here -- #640 only has the device import as surrounding context,
#387 matches `LAYA_DEVICE` without changing that row, and #53 adds an unrelated mention.

Found during a cross-repo contract review of how a configuration value is normalised across
the entry points that read it; the defect and the fix are this repository's own.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @aashish254! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants