Skip to content

perf(router): bypass predict context and OS entropy when no hooks are active - #1386

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
HamidRezaLS:perf/router-bypass-context-entropy
Oct 31, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
HamidRezaLS:perf/router-bypass-context-entropy

Conversation

@HamidRezaLS

@HamidRezaLS HamidRezaLS commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What

in laya/router.py:

  • checks if any active hooks implement on_route before allocating a PredictContext.
  • when no on_route hooks are active (standard path), skips uuid.uuid4().hex (reading OS entropy via /dev/urandom syscall) and time.perf_counter(), directly returning the routing decision.
  • hoists hook composition and timeout validation outside the batch loop in route_batch.

Why

in Router.route and route_batch, every request previously instantiated a full PredictContext. on Linux, uuid.uuid4() invokes the getrandom(2) kernel syscall to read 16 bytes of entropy from /dev/urandom.

when zero on_route hooks are registered (the default for production inference), generating random UUIDs and timer timestamps is completely unused overhead.

bypassing context creation when hooks are idle eliminates all /dev/urandom syscalls and speeds up batch routing by ~12% to 19% (and single pinned routes by 2.1x), with zero changes to routing decisions or hook lifecycle behavior.

tested on Router.route_batch before and after (no hooks registered):

Batch Size Before After Speedup /dev/urandom Syscalls
16 0.90 ms 0.75 ms +16.1% 16 to 0 (-100%)
64 3.65 ms 2.95 ms +19.4% 64 to 0 (-100%)
128 6.89 ms 6.04 ms +12.3% 128 to 0 (-100%)
Single Pinned Route 5.58 µs 2.61 µs 2.14x 1 to 0 (-100%)

How it was verified

python tests/test_doc_citations.py
python tests/test_router.py
python tests/test_hooks.py
python tests/test_hooks_api.py
python tests/test_router_batch.py
python tests/test_state_budget.py
python tests/test_example_server_limits.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/

all passed:

  • test_doc_citations.py: 22 passed, 0 failed
  • test_router.py: 837 passed, 0 failed
  • test_hooks.py: 311 passed, 0 failed
  • test_hooks_api.py: 645 passed, 0 failed
  • test_router_batch.py: 34 passed, 0 failed
  • test_state_budget.py: 32 passed, 0 failed
  • test_example_server_limits.py: 155 passed, 0 failed
  • ruff and compilation checks: clean


  • preserves public contract for subclasses: resolves use_fast = type(self).route is Router.route once per call so subclasses overriding route retain their overrides during route_batch.
  • de-duplicates request dictionary validation ahead of the dispatch branches.
  • added regression test asserting route_batch invokes subclass route overrides.

@HamidRezaLS
HamidRezaLS force-pushed the perf/router-bypass-context-entropy branch from f602705 to 31e3bac Compare October 26, 2026 19:51
NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window
attribution preserved through end hooks (#1369), RemoteRouter identifying
itself with a configurable User-Agent (#1396), ten more inputs refused where
they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404,
#1405, #1406), per-language temperatures and per-call budgets in .NET and
Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle,
SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation
(#1399, #1407), and parity-lane sharding that replaces the budget headroom
added for 0.4.16 (#1373, #1375).

Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff
clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean.
Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0:
Java 20 families match laya (9 unverified), UnicodeTables matches CPython
15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match,
both laya-cpp .inc files match. Version agrees across pyproject.toml,
laya.__version__ and nix/package.nix. README carries 0.4.17 only.

Security audit of the merge diff, 362766 added lines over 480 files: zero
obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or
symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm
install hooks unchanged, and no third-party host in any added line
(example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12
dynamic-exec hits are the C++ test harness invoking its own checker through
sys.executable, the Go sample generator reading laya's own example by AST at
development time, and one comment in rust.yml.

Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)]
but pub Rust hook that would ship as public API able to replace the forward
pass), #1386 (route_batch switched from self.route to self._route, silently
bypassing subclass overrides the docstring promises), and nine stacked port
PRs that conflict with each other and need linearizing (#1377, #1378, #1380,
#1382, #1383, #1385, #1387, #1388, #1389, #1390).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Holding this, and as with #1381 the performance work is sound and I would like it in. Driving the /dev/urandom syscalls to zero is a real fix, and 2.14x on a single pinned route is worth having. I also checked the thing I was most suspicious about and it holds up: validate_timeout still runs before the early return, so a bad timeout is still refused rather than skipped on the fast path.

The blocker is one line. route_batch moves from self.route(...) to self._route(...), which bypasses any subclass override of route. The docstring on route_batch promises "the same optional routing overrides accepted by :meth:route" and behaviour "as on :meth:route", so a subclass that overrides route to add logging, a policy check or a decision rewrite currently has that applied to batches and would silently stop having it applied. Silently is the problem: no error, no warning, just a batch that no longer goes through the caller's code.

Nothing in the tree catches this. RemoteRouter is the only in-tree subclass of Router and it does not override route, so the suite passes either way. That makes it a change in the public contract that the tests cannot see, which is the category I will not merge on a green run alone.

The fix I would suggest is cheap: resolve it once per call rather than per item.

use_fast = type(self).route is Router.route

Then take self._route when that is true and self.route when it is not. A subclass keeps its override, an unmodified Router keeps your speedup, and the check costs one attribute comparison per batch instead of per row.

Separately, and not a blocker: the request validation is now duplicated across the two branches, about fifteen lines. Once the override check above decides which path to take, that validation should be able to sit in one place ahead of the split.

A test with a subclass that overrides route and asserts the override is still reached through route_batch would make the contract enforceable rather than documented, and would have caught this.

Please also say in the PR description that route_batch behaviour is unchanged for subclasses once this is in, so the next person reading the history knows it was considered.

@HamidRezaLS
HamidRezaLS force-pushed the perf/router-bypass-context-entropy branch from 31e3bac to a6fbbbc Compare October 28, 2026 15:42
@HamidRezaLS
HamidRezaLS force-pushed the perf/router-bypass-context-entropy branch from a6fbbbc to 79abd9c Compare October 28, 2026 15:48
@HamidRezaLS

Copy link
Copy Markdown
Contributor Author

updated!
resolved use_fast = type(self).route is Router.route once per batch so any subclass overriding route keeps going through self.route, while standard routers take the fast path. also unified the request validation ahead of the branches and added a regression test in tests/test_router_batch.py to guard it.
updated the pr description as well thanks!

@NandhaKishorM
NandhaKishorM merged commit afb8a3b into NandhaKishorM:main Oct 31, 2026
63 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged, and I owe you an account of the review because I nearly rejected this on a mistake of my own.

I first read the diff, then merged a local pr1386 ref and found six failures in tests/test_mcp_diagnostics.py, all of the form

assert len(router.calls) == 2 * len(requests)
AssertionError: assert 0 == (2 * 4)

The cause looked damning: route_batch calling self._route directly means a Router subclass that overrides route is bypassed, and RecordingRouter in that file overrides exactly that. A user subclass wrapping route for logging or caching would have stopped being called.

That was a stale checkout, not your code. My git fetch origin refs/pull/1386/head:pr1386 was writing into a branch left over from an earlier session, which git refused as a non fast-forward, and I had redirected stderr to /dev/null so the rejection was invisible. I was reviewing 31e3bacb while your head was 79abd9cb. Entirely my error, and the second time in two days I have been bitten by hiding git's stderr.

On your actual head the guard is already there:

use_fast = type(self).route is Router.route

which is the right test, and it is the thing I would have asked for. Full gate on the correct head: 785 tests across 93 suites, exit 0, and the MCP diagnostics suite green.

What I verified beyond the suite, because the whole change rests on _has_on_route never returning False for a hook that does implement it. normalise_hooks accepts any duck-typed object carrying a hook event, not only BaseHook subclasses, and AsyncHook defines its own on_route, so a naive getattr would report True for every wrapped hook. Nine shapes, all correct:

hook shape detected
BaseHook without on_route False
BaseHook with on_route True
duck-typed with on_route True
duck-typed without on_route False
AsyncHook(with) True
AsyncHook(without) False
AsyncHook(duck with) True
AsyncHook(AsyncHook(with)) True
mixed, no-route first True

And the behaviour the gate protects: a subclass overriding route still records 2 calls through route_batch, the fast and slow paths agree on both model and reason, an on_route hook still mutates every decision in a batch, and an empty batch returns [] without touching anything.

The unwrap loop rather than a single .hook dereference is also right, since AsyncHook(AsyncHook(x)) is constructible and I tested it.

Shipping in 0.4.19.

@HamidRezaLS

Copy link
Copy Markdown
Contributor Author

haha no worries at all those silent git fails happen to everyone... glad everything checked out with the hook edge cases and thanks for merging!

@HamidRezaLS
HamidRezaLS deleted the perf/router-bypass-context-entropy branch October 31, 2026 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants