Skip to content

feat(go): router model lifecycle (factory, leases, LRU, single-flight) - #1374

Merged
NandhaKishorM merged 7 commits into
NandhaKishorM:mainfrom
keemsisi:feat/go-router-lifecycle-main
Oct 28, 2026
Merged

NandhaKishorM merged 7 commits into
NandhaKishorM:mainfrom
keemsisi:feat/go-router-lifecycle-main

Conversation

@keemsisi

Copy link
Copy Markdown
Contributor

Based on main (R1 #1310 and R2 #1311 are merged). Branch feat/go-router-lifecycle-main, rebased from the earlier stacked feat/go-router-lifecycle.

What and why

Part What
router/lifecycle.go AgentFactory func(ctx, Checkpoint) (*agent.Agent, error), WithAgentFactory, WithMaxLoaded (default 2). Load, Lease (refcounted: an evicted or unloaded agent the Router built closes only when its last lease is closed), Attach, Preload, Unload, UnloadAll, Loaded, MaxLoaded, Close, ErrClosed. Single-flight build with waiters; a failed build wakes them and is not cached; builds are serialised by one token (peak memory), a resident load never waits behind a cold one; Close waits for a build in flight
router/hooks.go LoadHook/EvictHook, LoadContext{Model, Agent, RunID, Router}, EvictContext, BaseHook and HookFuncs gain OnLoad/OnEvict. Fired over the R2 dispatcher after every internal lock is released, evictions before the load, stopping at the first raising hook as Python does
scripts/gen_go_fixtures.py new lifecycle family (275 cases, 2,145 ops) recorded from real laya.router with a stub Agent: loaded() and max_loaded after every op, events with the agent label, errors, warnings. Only the new lifecycle key is added to go_extra.json
CI floor 3869 -> 4552 (comment and both --min-tests)

Python semantics kept: LRU order with touch on every resident load; eviction after publishing the new agent, on_evict before on_load; Attach raises max_loaded to fit and never evicts or fires hooks; Preload raises it to fit the request plus everything resident and leaves a resident checkpoint where it is in the order; a build that loses a race with Attach keeps the attached agent (and still fires on_load for it); unknown names fail before anything changes.

Differences from Python

Python Go Why
load() returns an Agent that lives as long as someone references it Load returns the agent, valid only while resident (eviction closes what the Router built). Lease keeps it open until Close a native session needs an explicit close; the GC cannot decide for us. Same split as Java
eviction drops a reference eviction/unload retires the slot; the agent closes at its last lease release, outside every lock a prediction in flight never meets a closed session
max_loaded below 1 is clamped to 1 WithMaxLoaded(n<1) is an error Java's rule; a silent clamp hides a typo
attach of an unknown name registers it *UnknownModelError (names are the three built-ins; register is R5-optional) no custom checkpoints in Go
attach never closes the agent (GC) the Router never closes an attached agent; attaching the agent already resident hands ownership to the caller Java's rule
preload([]) loads nothing Preload(ctx) with no names loads all three variadic
agent built from a hub repo and agent_kwargs, revision, digests the factory owns all of it; no hub download R5 (config) is separate
the exception from the build propagates as is, to waiters too router: cannot load the english checkpoint: <cause> with %w; waiters get loading english failed on the call that was building it: <cause>. A builder's context cancellation is not inherited: the waiter builds for itself Go errors wrap; one caller's context must not fail another's
a waiter blocks on an Event a waiter also honours its own ctx ctx on I/O
unload() returns None Unload/UnloadAll return the checkpoints let go convenience; events are the same
hook text laya: hook Boom.on_load failed laya: hook <GoTypeName>.on_load failed; timeout text laya: hook <GoTypeName> exceeded 0.1s (as R2) Go has no bound-method name; tests rewrite and compare the rest
hooks_concurrent=False lock is an RLock not re-entrant: a serial hook must not call Load/Lease/Preload/Unload on its own Router when that fires a hook (waits until a timeout). Documented no goroutine identity
unload(K) racing a build of K can fire on_evict before that build's on_load same (Python does too); documented in hooks.go the build publishes, then the hooks fire outside the lock
a factory may call back into the Router a factory must not call Load/Lease/Preload/Unload/UnloadAll/Close on its own Router: it holds the build token and would wait for itself. Documented on AgentFactory one build at a time, as Python's build lock (which is not re-entrant either)
an on_load hook that unloads or closes for the same checkpoint Load returns the already-closed agent and no error (documented); after a hook timeout the call gives its lease back, so an abandoned hook's agent may close (documented) the hook ran outside the lock by design
no close() Close seals the Router, frees everything (waits for builds), is idempotent; loads after it return ErrClosed Java's rule
loaded_revisions, register/unregister/registered not ported R5 / optional

Evidence

Check Result
goldens (lifecycle): 35 hand-chosen sequences + 240 random, each op checks loaded(), max_loaded, events, errors, warnings all equal Python, including hook raise/warn/timeout on on_load and on_evict, build failure, attach/preload/unload interleavings
go_extra.json only the lifecycle key added; --check matches
differential fuzz vs Python (seeds 7, 11, 12; 1,500 sequences each of up to 25 ops: load/preload/attach/unload/fail/heal, hooks that raise or not, hooks_raise, serial hooks), -race 4,500 sequences, 58,527 ops, 0 mismatches
-race tests (deterministic fake factory with gates) concurrent loads of one key build once; waiters get the error and none rebuilds; waiter does not inherit cancellation; panicking factory wakes waiters; lease keeps an evicted agent open and closes it at the last release; double Lease.Close under a race counts once; Close during an in-flight build (builder keeps its lease, the agent closes at release); a hook calls Loaded, MaxLoaded, Load while hooks run (outside the lock); a slow Close of an evicted agent holds no lock; a resident load is not stalled by a cold build; eviction close error fails the call but delivers the events; a failing on_load gives the lease back; 8-goroutine stress over every API, every built agent closed exactly once, attached agents never
lane: race 1.27.1, 1.22.12, committed goldens, private output dir 4552 ran, 0 skipped, 0 failed on each; floor+1 exits 1
gofmt, vet, staticcheck, gen_go_fixtures --check, gen_tables --check, workflow folding, ruff clean

Mutants

41 compiled, 41 killed (at most 3 per behavior).

Behavior Mutants Killed by
lease refcount release closes with leases outstanding; last release never closes; Lease.Close not once-only; retire closes under a lease TestLeaseOutlivesEviction, TestClosingALeaseTwiceUnderARaceCountsOnce, TestConcurrentStress
evict order newest first; resident load does not touch; one too many TestEvictionOrderIsLeastRecentlyUsed, TestLoadedOrderIsLRU, TestLifecycleGolden
single-flight waiters build their own; builds not serialised TestFailedBuildIsNotCachedAndWakesWaiters, TestColdBuildsAreSerialised
failure waiters not woken; waiters get no error; waiter inherits cancellation; panic does not fail the build TestFailedBuildIsNotCachedAndWakesWaiters, TestWaiterDoesNotInheritTheBuildersCancellation, TestPanickingFactoryWakesWaiters
Close does not wait for builds; does not seal; flag never set TestCloseWaitsForAnInFlightBuild, TestCloseFreesAndSeals
hooks dispatched under the lock; load before evict; lease released before on_load; failing hook strands the lease; on_load agent is the discarded build; first hook error does not stop TestHooksRunOutsideTheLock, TestLifecycleGolden, TestLeaseHeldAcrossTheLoadHook, TestFailingHookKeepsTheCheckpointAndGivesBackTheLease, TestAttachWhileBuildingKeepsTheAttachedAgent
attach / preload / max max not raised (both); Router owns attached agent; same agent retired; duplicate build kept open; resident touched by preload; zero accepted TestAttachRaisesMaxLoaded, TestPreloadFitsEverythingAndStages, TestAttachedAgentsAreNotClosed, TestAttachingTheSameInstanceIsSafe, TestPreloadResidentIsNotTouched, TestMaxLoadedMustBePositive
close errors / locks close error ignored; close under the lock; nil agent accepted; unload order not trimmed; unload waits for every build TestCloseErrorFailsTheLoadButNotTheEvents, TestASlowCloseHoldsNoLock, TestFactoryReturningNothing, TestUnloadFrees, TestUnloadWaitsForItsOwnBuildOnly

Review fixes (Opus): a panic in an agent's Close is turned into an error per agent, so it cannot wedge the Router (a double finish re-closed a channel under the lock; finish is now idempotent), skip the other closes or skip on_evict; Attach of an agent the Router built but evicted while leased hands it to the caller (shared ownership record), so the last lease release no longer closes a resident attached agent; Load and Preload keep their lease across on_load; a build queued on the build token rechecks Close and Attach before calling the factory. Each has a test and a mutant (7 more, 41 total, all killed).

The mutant run found one real gap, fixed: on_load saw the discarded duplicate when Attach won the race with a build (TestAttachWhileBuildingKeepsTheAttachedAgent now pins it).

Real-weights e2e (scratch under laya-e2e/go-lane/e2e-router-lifecycle/, not in CI)

Factory over the nested onnx module (onnx.Open + agent.Open) for the three checkpoints, WithMaxLoaded(1), Python reference = ONNXAgent on the same fused graph, -race.

Registered Run Skipped Failed
1 (TestZZRouterLifecycle) 1 0 0
Measure Result
checkpoints english (graph exported for this run from checkpoints/english, export self-verified vs torch), multilingual (onnx/fixed-multilingual), typed-decisions (laya-review-scratch/work, not in LAYA_FIXTURE_CHECKPOINTS)
predictions vs Python 15 (5 checks x 3 states): 297 numbers, 132 labels, max deviation 0.00e+00 (tolerance 8e-4)
leased evicted agent multilingual evicted by typed-decisions while leased: still answers (equal to Python), closes when the lease is closed
sessions 4 opened, 4 closed (english, multilingual, typed-decisions, english); never more than 2 live at once; 0 live and 0 runtime references after Close
hook events load:english, evict:english, load:multilingual, evict:multilingual, load:typed-decisions, evict:typed-decisions, load:english, evict:english
tamper (english factory serves the multilingual weights) fails on the first prediction (go 1, py 0.9334)

Scanners (CodeQL, Semgrep, CVE gate, actionlint) and review: pending.

Related PRs

PR What
#1310 router R1 (routing)
#1311 router R2 (on_route hooks)
this PR router R3 (lifecycle)
#1306, #1307, #1308, #1309 calibration, presets and shortlist, email, samples (each on main)

Real laya.router with a stub Agent: eviction order, loaded() and max_loaded after every
op of 38 fixed and 240 random sequences (load, preload, attach, unload, build failure),
and the on_load/on_evict events, raise, warn and timeout text. Only the new lifecycle key is
added to go_extra.json.
…/on_evict)

AgentFactory, Load, Lease (an evicted agent closes after its last lease), WithMaxLoaded,
Attach, Preload, Unload, UnloadAll, Loaded, Close. One build per checkpoint with waiters;
a failed build wakes them and is not cached; a waiter does not inherit the builder's
cancellation; Close waits for a build in flight. Hooks fire after every lock is released,
evictions before the load, over the shared hook dispatcher.
…tach race, gated build released on failure)

Mutants found: on_load saw the discarded build when Attach won the race.
A panic in an agent's Close is an error per agent (it wedged the Router through a double
finish, and skipped the other closes and on_evict); finish is idempotent. Attach of an agent
the Router built hands ownership to the caller even when an earlier slot is still leased.
Load keeps its lease across on_load. A build queued on the build token rechecks Close and
Attach before calling the factory. Documented: Unload racing a build can fire on_evict first;
a factory must not call back into its Router. Floor 4190 -> 4197.
…hable finish guards

Preload released its lease before the hooks, so a concurrent Unload or Close could close the
agent under OnLoad. The idempotent finish and the mine.err guard are unreachable now that
closes cannot panic. Documented: Load after an OnLoad hook unloads, and hook timeouts.
Floor 4197 -> 4198.
@NandhaKishorM
NandhaKishorM merged commit 99ec92b into NandhaKishorM:main Oct 28, 2026
50 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window
attribution preserved through end hooks (#1369), RemoteRouter identifying
itself with a configurable User-Agent (#1396), ten more inputs refused where
they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404,
#1405, #1406), per-language temperatures and per-call budgets in .NET and
Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle,
SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation
(#1399, #1407), and parity-lane sharding that replaces the budget headroom
added for 0.4.16 (#1373, #1375).

Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff
clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean.
Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0:
Java 20 families match laya (9 unverified), UnicodeTables matches CPython
15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match,
both laya-cpp .inc files match. Version agrees across pyproject.toml,
laya.__version__ and nix/package.nix. README carries 0.4.17 only.

Security audit of the merge diff, 362766 added lines over 480 files: zero
obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or
symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm
install hooks unchanged, and no third-party host in any added line
(example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12
dynamic-exec hits are the C++ test harness invoking its own checker through
sys.executable, the Go sample generator reading laya's own example by AST at
development time, and one comment in rust.yml.

Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)]
but pub Rust hook that would ship as public API able to replace the forward
pass), #1386 (route_batch switched from self.route to self._route, silently
bypassing subclass overrides the docstring promises), and nine stacked port
PRs that conflict with each other and need linearizing (#1377, #1378, #1380,
#1382, #1383, #1385, #1387, #1388, #1389, #1390).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged. The lifecycle piece I checked hardest was the lease rule, because it is where this design either holds or corrupts state: an evicted agent closing only after its last lease is released is the correct ordering. The alternative, closing on eviction, hands an in-flight request a closed session, and the resulting failure appears at the call site rather than at the eviction that caused it.

Single-flight on the build path is the other thing that had to be there. Without it, a burst of first requests for the same checkpoint each start their own load, and the cost shows up as memory rather than as an error, which is the hardest kind of load problem to attribute afterwards.

Pairing OnLoad and OnEvict with the existing OnRoute hooks keeps one dispatcher rather than a second parallel notion of hooks, and WithAgentFactory gating the loading behaviour means the routing-only path stays as cheap as it was.

Shipped in 0.4.17.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants