Repository navigation
fix(serve): forward and validate per-request token budget overrides (… - #566
Conversation
|
Heads-up for whoever reviews first: this and #550 are the same change, opened a few hours apart — both forward The only behavioural difference I can find is the So the combination that cannot work is already refused by core with a named error, and what the new rule additionally rejects — The measurements behind the issue (#549) are per-checkpoint if either PR wants them: state budget 312 tokens on |
|
Thanks for the great feedback and thorough review @AlKor13! You were completely right about Since you offered and have no preference between the two, I would really appreciate having #566 merged if you and @NandhaKishorM are happy with this update! |
|
Thanks for the PR. Capping per-request A few points:
Otherwise a clean, well-tested guard. |
|
Thanks for taking the point — and to settle @rahul05ranjan's question 4 with measurements rather than opinion, since it is the same question: what does core do when So the combination that genuinely cannot work is already refused by core with a named error, which One review note on the update, because it costs you a regression pin: the shared def test_no_budget_keeps_the_call_unchanged(monkeypatch):
router = FakeRouter() # predict(self, state, questions, model=None)
client = TestClient(create_app(router=router))
for body in (REQ, dict(REQ, max_len=None, head_max_len=None)):
assert client.post("/v1/systemone", json=body).status_code == 200That test is in #550 if it is easier to lift than to rewrite. On the duplicate: I meant it, I have no preference, and this one reads fine. I would rather not close #550 until @NandhaKishorM says which he is taking, though — if I close mine first and this one stalls, the fix is gone entirely, and a closed PR is easy to reopen but easy to forget. The moment he indicates this is the one, I will close mine and say so here. Also worth knowing: |
…-arg router regression pin
|
Thanks for the thorough reviews and guidance @rahul05ranjan and @AlKor13! I've pushed an update addressing all points:
All 41 server tests pass cleanly (39 passed, 2 skipped — torch-dependent). Doc table checks and lint/compile CI gates green! |
…efusing them #530 refused a token budget on a remote node because laya-serve had no field for it. #566 added the field (bounded by LAYA_MAX_TOKEN_BUDGET), so _call_remote now puts the override in the request body. Also puts the budget and hooks sections of docs/langchain.md back at the end of the page in order (5-8). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…and CI guards - nix/laya-serve.nix: maxLoaded, maxTokenBudget and revision options for LAYA_MAX_LOADED (#603), LAYA_MAX_TOKEN_BUDGET (#566) and LAYA_REVISION (#605); test_packaging counts mcp/device.py's _ENV_KEY as the LAYA_DEVICE read serve now goes through (#574). - docs/docker.md documents LAYA_REVISION. - test_audit_scope knows the llamaindex/crewai extras and onnxscript. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Thanks @Swatantra-66, merged. Now that serve takes the fields, LangChain remote nodes forward the budget too (follow-up to #530). It ships in 0.3.21. |
What
POST /v1/systemoneinlaya/serve.pyto extract optionalmax_lenandhead_max_lenfields from incoming request JSON payloads and pass them via**predict_kwargsintorouter.predict().LAYA_MAX_TOKEN_BUDGET):DEFAULT_MAX_TOKEN_BUDGET = 8192as the default upper ceiling for any single request._resolve_max_token_budget()to readLAYA_MAX_TOKEN_BUDGETfrom the environment, allowing server operators to safely customize the maximum permissible budget.422 Unprocessable Entity):_validate_budget_param()to validate thatmax_lenandhead_max_lenare positive integers.{"max_len": true}and floating-point values) with HTTP 422 ("<key> must be an integer").<= 0) with HTTP 422 ("<key> must be a positive integer")."<key> exceeds server limit (<val> > <cap>)").head_max_len < max_lenwhenever both parameters are supplied, raising HTTP 422 ifhead_max_len >= max_len("head_max_len (<head>) must be less than max_len (<max>)").**predict_kwargsthrough the async event loop executor (loop.run_in_executor(pool, lambda: router.predict(state, questions, model=model, **predict_kwargs))), maintaining non-blocking HTTP serving without modifying the synchronous signature ofRouter.predict().LAYA_MAX_TOKEN_BUDGETto the environment variables table inlaya/serve.py's module docstring.LAYA_MAX_TOKEN_BUDGETtodocs/docker.mdunder Server Configuration.tests/test_serve.pyexercising parameter pass-through, type rejection, negative/zero rejection, server cap enforcement, relative slice validation, and environment variable override. UpdatedFakeRouterandGatedRouterstubs to record**kwargs.Why this change
In
laya/serve.py, inference requests to/v1/systemonepreviously unpacked onlystate,questions, andmodelfrom the JSON body. Even though the underlying engine (Router.predict()andAgent.system_one()) fully supports custommax_lenandhead_max_lentoken budgets, the HTTP shim silently dropped them.When sending large states (for instance, a 50,000-character support ticket or browser DOM snapshot), the inference engine remained locked to the default 312-token limit. Consequently:
usageblock reported the clamped token count rather than the caller's requested budget.Allowing clients to specify
max_lenandhead_max_lensolves this issue, while enforcingLAYA_MAX_TOKEN_BUDGETensures remote clients cannot provoke memory exhaustion or GPU OOM conditions on the host.Specific design decisions:
isinstance(True, int)evaluates toTrue. The validator checksnot isinstance(val, int) or isinstance(val, bool)to ensure boolean literals like{"max_len": true}are rejected with HTTP 422 instead of coerced to1.head_max_len < max_lencatches misconfigured caller requests at the HTTP boundary before any tensor collation or forward pass begins.predict_kwargsremains empty, preserving the default router and checkpoint behavior with zero overhead.How it was verified
Output:
All 39 tests passed, including all 6 new regression tests:
test_token_budget_forwarded: Verifies validmax_lenandhead_max_lenare received by the router.test_token_budget_validation_type: Verifies strings, booleans, and floats return HTTP 422.test_token_budget_validation_positive: Verifies0and negative values return HTTP 422.test_token_budget_exceeds_server_cap: Verifies values above the server ceiling return HTTP 422.test_token_budget_head_max_len_exceeds_max_len: Verifieshead_max_len >= max_lenreturns HTTP 422.test_token_budget_env_cap_override: Verifies customLAYA_MAX_TOKEN_BUDGETenvironment settings are respected.Output:
ruff check:All checks passed!python -m compileall: Clean, 0 errors.git diff --check: Clean, no whitespace or formatting regressions.Review scope
Changes are strictly confined to the HTTP serving shim, documentation, and corresponding tests:
laya/serve.py(budget resolution, validation logic, docstrings, and prediction forwarding)docs/docker.md(server configuration environment variable table)tests/test_serve.py(unit test suite and test stub updates)Checklist
main.ruff) and compilation (compileall) gates pass without error.laya/serve.pydocstrings anddocs/docker.md.Fixes #549