Skip to content

fix(serve): forward and validate per-request token budget overrides (… - #566

Merged
NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
Swatantra-66:fix/serve-token-budget
Sep 27, 2026
Merged

NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
Swatantra-66:fix/serve-token-budget

Conversation

@Swatantra-66

Copy link
Copy Markdown
Contributor

What

  1. Forward Token Budget Overrides to Router: Updated POST /v1/systemone in laya/serve.py to extract optional max_len and head_max_len fields from incoming request JSON payloads and pass them via **predict_kwargs into router.predict().
  2. Server-Side Token Budget Cap (LAYA_MAX_TOKEN_BUDGET):
    • Added DEFAULT_MAX_TOKEN_BUDGET = 8192 as the default upper ceiling for any single request.
    • Added _resolve_max_token_budget() to read LAYA_MAX_TOKEN_BUDGET from the environment, allowing server operators to safely customize the maximum permissible budget.
  3. Strict Request Validation (422 Unprocessable Entity):
    • Added _validate_budget_param() to validate that max_len and head_max_len are positive integers.
    • Explicitly rejected non-integer types (including JSON booleans {"max_len": true} and floating-point values) with HTTP 422 ("<key> must be an integer").
    • Rejected non-positive integers (<= 0) with HTTP 422 ("<key> must be a positive integer").
    • Rejected values exceeding the server cap with HTTP 422 ("<key> exceeds server limit (<val> > <cap>)").
    • Enforced head_max_len < max_len whenever both parameters are supplied, raising HTTP 422 if head_max_len >= max_len ("head_max_len (<head>) must be less than max_len (<max>)").
  4. Thread Pool Executor Integration: Safely threaded **predict_kwargs through the async event loop executor (loop.run_in_executor(pool, lambda: router.predict(state, questions, model=model, **predict_kwargs))), maintaining non-blocking HTTP serving without modifying the synchronous signature of Router.predict().
  5. Documentation & Env Spec:
    • Added LAYA_MAX_TOKEN_BUDGET to the environment variables table in laya/serve.py's module docstring.
    • Added LAYA_MAX_TOKEN_BUDGET to docs/docker.md under Server Configuration.
  6. Regression Tests: Added 6 targeted unit tests to tests/test_serve.py exercising parameter pass-through, type rejection, negative/zero rejection, server cap enforcement, relative slice validation, and environment variable override. Updated FakeRouter and GatedRouter stubs to record **kwargs.

Why this change

In laya/serve.py, inference requests to /v1/systemone previously unpacked only state, questions, and model from the JSON body. Even though the underlying engine (Router.predict() and Agent.system_one()) fully supports custom max_len and head_max_len token budgets, the HTTP shim silently dropped them.

When sending large states (for instance, a 50,000-character support ticket or browser DOM snapshot), the inference engine remained locked to the default 312-token limit. Consequently:

  • Long states were prematurely clamped to their first 312 tokens, causing decisions to ignore the bulk of the prompt text.
  • The returned usage block reported the clamped token count rather than the caller's requested budget.

Allowing clients to specify max_len and head_max_len solves this issue, while enforcing LAYA_MAX_TOKEN_BUDGET ensures remote clients cannot provoke memory exhaustion or GPU OOM conditions on the host.

Specific design decisions:

  • Type safety for booleans: In Python, isinstance(True, int) evaluates to True. The validator checks not isinstance(val, int) or isinstance(val, bool) to ensure boolean literals like {"max_len": true} are rejected with HTTP 422 instead of coerced to 1.
  • Relative slice integrity: Laya's attention mechanism allocates the head slice inside the broader sequence budget. Enforcing head_max_len < max_len catches misconfigured caller requests at the HTTP boundary before any tensor collation or forward pass begins.
  • Fail-open defaults: If neither parameter is specified, predict_kwargs remains empty, preserving the default router and checkpoint behavior with zero overhead.

How it was verified

  1. Serve Test Suite Execution:
python -m pytest tests/test_serve.py

Output:

   39 passed in 4.30s

All 39 tests passed, including all 6 new regression tests:

  • test_token_budget_forwarded: Verifies valid max_len and head_max_len are received by the router.
  • test_token_budget_validation_type: Verifies strings, booleans, and floats return HTTP 422.
  • test_token_budget_validation_positive: Verifies 0 and negative values return HTTP 422.
  • test_token_budget_exceeds_server_cap: Verifies values above the server ceiling return HTTP 422.
  • test_token_budget_head_max_len_exceeds_max_len: Verifies head_max_len >= max_len returns HTTP 422.
  • test_token_budget_env_cap_override: Verifies custom LAYA_MAX_TOKEN_BUDGET environment settings are respected.
  1. Repository Quality Gates:
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
git diff --check

Output:

  • ruff check: All checks passed!
  • python -m compileall: Clean, 0 errors.
  • git diff --check: Clean, no whitespace or formatting regressions.

Review scope

Changes are strictly confined to the HTTP serving shim, documentation, and corresponding tests:

  • laya/serve.py (budget resolution, validation logic, docstrings, and prediction forwarding)
  • docs/docker.md (server configuration environment variable table)
  • tests/test_serve.py (unit test suite and test stub updates)

Checklist

  • One logical change, on a fresh branch based on latest main.
  • All 39 server tests pass cleanly.
  • Repository CI lint (ruff) and compilation (compileall) gates pass without error.
  • Documentation updated in laya/serve.py docstrings and docs/docker.md.
  • No changes to checkpoint weights or core ONNX/model architecture.

Fixes #549

@AlKor13

AlKor13 commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

Heads-up for whoever reviews first: this and #550 are the same change, opened a few hours apart — both forward max_len/head_max_len from the /v1/systemone body into router.predict, both cap it from an env var (LAYA_MAX_TOKEN_BUDGET here, LAYA_MAX_TOKENS there, both defaulting to 8,192), both 422 on a non-integer, a bool, a non-positive value and a value over the cap, and both leave the call unchanged when neither field is sent. @NandhaKishorM, one of them should just be closed; I have no stake in which, and I am happy to close #550 if this one is the one you want.

The only behavioural difference I can find is the head_max_len < max_len rule here. I measured what core does without it, laya checkpoint, 20 options, a long state:

max_len=192  head_max_len=512  -> ValueError: question 'intent' options exceed head_max_len=512   (serve maps this to 422 today)
max_len=256  head_max_len=256  -> 200, answered, top_p 0.592
max_len=512  head_max_len=512  -> 200, answered, top_p 0.323
max_len=1024 head_max_len=900  -> 200, answered, usage input_tokens 795

So the combination that cannot work is already refused by core with a named error, and what the new rule additionally rejects — head_max_len == max_len, or a head that merely starves the state — is accepted by Agent.system_one and Router.predict today. That makes it a policy change rather than a validation fix, and if the rule is right it probably belongs in core, so the Python API and the HTTP surface agree rather than the server being stricter than the library it wraps.

The measurements behind the issue (#549) are per-checkpoint if either PR wants them: state budget 312 tokens on laya, a 21,692-character state answered from 7.2% of itself, usage reporting the clamped 512 as if it were the whole input, and the same request answered correctly once max_len reaches the wire.

@Swatantra-66

Copy link
Copy Markdown
Contributor Author

Thanks for the great feedback and thorough review @AlKor13!

You were completely right about head_max_len — keeping the HTTP surface consistent with core's existing behavior makes total sense. I've pushed a commit removing that artificial restriction and added a test confirming head_max_len == max_len is forwarded cleanly without rejection.

Since you offered and have no preference between the two, I would really appreciate having #566 merged if you and @NandhaKishorM are happy with this update!

@rahul05ranjan

Copy link
Copy Markdown
Contributor

Thanks for the PR. Capping per-request max_len/head_max_len overrides server-side is a sensible guard against a client asking for an unbounded budget.

A few points:

  1. _resolve_max_token_budget fallback: on any parse failure it silently returns DEFAULT_MAX_TOKEN_BUDGET. That's consistent with _resolve_max_concurrent, but a misconfigured LAYA_MAX_TOKEN_BUDGET=abc would silently become 8192 with no warning. Worth a log line on the fallback path so operators notice the misconfiguration.

  2. _validate_budget_param bool check: isinstance(val, bool) is correctly rejected (since bool is a subclass of int in Python, isinstance(True, int) is True). Good catch — this is a common footgun.

  3. val <= 0 vs val > max_cap: the ordering means 0 and negatives get "must be a positive integer" while oversized gets "exceeds server limit". Clear and correct.

  4. head_max_len vs max_len relationship: the cap applies to each independently, but is there a constraint that head_max_len <= max_len? If a client sets head_max_len above max_len, does the agent handle it, or should the server reject that combination? Worth confirming.

  5. Doc table alignment: the serve.py docstring table row for LAYA_MAX_TOKEN_BUDGET looks slightly misaligned (the ==== separator column widths). Cosmetic, but the doc tables are auto-checked by test_doc_tables.py per the CI — worth confirming it passes.

Otherwise a clean, well-tested guard.

@AlKor13

AlKor13 commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

Thanks for taking the point — and to settle @rahul05ranjan's question 4 with measurements rather than opinion, since it is the same question: what does core do when head_max_len is not below max_len? laya checkpoint, 20 options, a long state, on main:

max_len=192  head_max_len=512   -> ValueError: question 'intent' options exceed head_max_len=512
max_len=256  head_max_len=256   -> 200, answered, top_p 0.592
max_len=512  head_max_len=512   -> 200, answered, top_p 0.323
max_len=1024 head_max_len=900   -> 200, answered, usage input_tokens 795

So the combination that genuinely cannot work is already refused by core with a named error, which serve maps to 422 today; head_max_len == max_len is accepted by Agent.system_one and Router.predict and merely starves the state of room. Rejecting it on the HTTP surface alone would make the server stricter than the library it wraps, which is why removing the rule was the right call — and if that rule is wanted, it belongs in core so both agree.

One review note on the update, because it costs you a regression pin: the shared FakeRouter.predict now takes **kwargs. That fake was the only thing in tests/test_serve.py asserting that serve calls router.predict without budget keywords when the body carries none — which matters because create_app(router=...) accepts an injected router, and a third-party or predating one has the three-argument signature. Both PRs only pass the fields when present, so the behaviour is right; it is just no longer pinned. Keeping one fake with the old signature alongside the new one covers it:

def test_no_budget_keeps_the_call_unchanged(monkeypatch):
    router = FakeRouter()          # predict(self, state, questions, model=None)
    client = TestClient(create_app(router=router))
    for body in (REQ, dict(REQ, max_len=None, head_max_len=None)):
        assert client.post("/v1/systemone", json=body).status_code == 200

That test is in #550 if it is easier to lift than to rewrite.

On the duplicate: I meant it, I have no preference, and this one reads fine. I would rather not close #550 until @NandhaKishorM says which he is taking, though — if I close mine first and this one stalls, the fix is gone entirely, and a closed PR is easy to reopen but easy to forget. The moment he indicates this is the one, I will close mine and say so here.

Also worth knowing: gh pr checks reports no checks on this branch, so CI has not run on the update yet — probably the first-time-contributor approval gate. Worth a nudge, since the maintainer reviews against green runs.

@Swatantra-66

Swatantra-66 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough reviews and guidance @rahul05ranjan and @AlKor13!

I've pushed an update addressing all points:

  1. Warning log on fallback (@rahul05ranjan): Added _log.warning(...) when LAYA_MAX_TOKEN_BUDGET is unparseable or <= 0 before falling back to default 8192. Added a unit test verifying both warnings in server logs.
  2. Doc table alignment (@rahul05ranjan): Widened column separators to 25 chars (=========================) so all rows align cleanly. Verified python tests/test_doc_tables.py passes (109 passed).
  3. Injected router regression pin (@AlKor13): Great catch on preserving the 3-arg signature. I restored FakeRouter and GatedRouter to their original 3-argument predict(self, state, questions, model=None) signature, added a BudgetRouter subclass for kwargs tracking, and added test_no_budget_keeps_the_call_unchanged to guarantee older/injected routers are never broken.
  4. head_max_len vs max_len: As AlKor13 noted with measurements, core Agent accepts head_max_len == max_len and only raises ValueError if the options exceed the slice budget (which serve.py maps to 422). The previous commit removed the artificial < check so the HTTP surface stays strictly aligned with core library behavior.
  5. Torch skip guards: Added pytest.importorskip("torch") to two pre-existing tests (test_thread_limit, test_a_nested_choice_label) that fail without torch installed, matching the existing test_fast.py pattern.

All 41 server tests pass cleanly (39 passed, 2 skipped — torch-dependent). Doc table checks and lint/compile CI gates green!

@NandhaKishorM
NandhaKishorM merged commit d8c0ccb into NandhaKishorM:main Sep 27, 2026
NandhaKishorM added a commit that referenced this pull request Sep 27, 2026
…efusing them

#530 refused a token budget on a remote node because laya-serve had no field for it. #566
added the field (bounded by LAYA_MAX_TOKEN_BUDGET), so _call_remote now puts the override in
the request body. Also puts the budget and hooks sections of docs/langchain.md back at the
end of the page in order (5-8).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
NandhaKishorM added a commit that referenced this pull request Sep 27, 2026
…and CI guards

- nix/laya-serve.nix: maxLoaded, maxTokenBudget and revision options for LAYA_MAX_LOADED
  (#603), LAYA_MAX_TOKEN_BUDGET (#566) and LAYA_REVISION (#605); test_packaging counts
  mcp/device.py's _ENV_KEY as the LAYA_DEVICE read serve now goes through (#574).
- docs/docker.md documents LAYA_REVISION.
- test_audit_scope knows the llamaindex/crewai extras and onnxscript.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @Swatantra-66, merged. Now that serve takes the fields, LangChain remote nodes forward the budget too (follow-up to #530). It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/v1/systemone answers a 50,000-character state from its first 312 tokens: max_len is ignored and usage reports the clamped count

4 participants