Repository navigation
feat(go): router model lifecycle (factory, leases, LRU, single-flight) - #1374
Conversation
Real laya.router with a stub Agent: eviction order, loaded() and max_loaded after every op of 38 fixed and 240 random sequences (load, preload, attach, unload, build failure), and the on_load/on_evict events, raise, warn and timeout text. Only the new lifecycle key is added to go_extra.json.
…/on_evict) AgentFactory, Load, Lease (an evicted agent closes after its last lease), WithMaxLoaded, Attach, Preload, Unload, UnloadAll, Loaded, Close. One build per checkpoint with waiters; a failed build wakes them and is not cached; a waiter does not inherit the builder's cancellation; Close waits for a build in flight. Hooks fire after every lock is released, evictions before the load, over the shared hook dispatcher.
…tach race, gated build released on failure) Mutants found: on_load saw the discarded build when Attach won the race.
A panic in an agent's Close is an error per agent (it wedged the Router through a double finish, and skipped the other closes and on_evict); finish is idempotent. Attach of an agent the Router built hands ownership to the caller even when an earlier slot is still leased. Load keeps its lease across on_load. A build queued on the build token rechecks Close and Attach before calling the factory. Documented: Unload racing a build can fire on_evict first; a factory must not call back into its Router. Floor 4190 -> 4197.
…hable finish guards Preload released its lease before the hooks, so a concurrent Unload or Close could close the agent under OnLoad. The idempotent finish and the mine.err guard are unreachable now that closes cannot panic. Documented: Load after an OnLoad hook unloads, and hook timeouts. Floor 4197 -> 4198.
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window attribution preserved through end hooks (#1369), RemoteRouter identifying itself with a configurable User-Agent (#1396), ten more inputs refused where they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404, #1405, #1406), per-language temperatures and per-call budgets in .NET and Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle, SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation (#1399, #1407), and parity-lane sharding that replaces the budget headroom added for 0.4.16 (#1373, #1375). Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean. Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0: Java 20 families match laya (9 unverified), UnicodeTables matches CPython 15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match, both laya-cpp .inc files match. Version agrees across pyproject.toml, laya.__version__ and nix/package.nix. README carries 0.4.17 only. Security audit of the merge diff, 362766 added lines over 480 files: zero obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm install hooks unchanged, and no third-party host in any added line (example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12 dynamic-exec hits are the C++ test harness invoking its own checker through sys.executable, the Go sample generator reading laya's own example by AST at development time, and one comment in rust.yml. Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)] but pub Rust hook that would ship as public API able to replace the forward pass), #1386 (route_batch switched from self.route to self._route, silently bypassing subclass overrides the docstring promises), and nine stacked port PRs that conflict with each other and need linearizing (#1377, #1378, #1380, #1382, #1383, #1385, #1387, #1388, #1389, #1390). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged. The lifecycle piece I checked hardest was the lease rule, because it is where this design either holds or corrupts state: an evicted agent closing only after its last lease is released is the correct ordering. The alternative, closing on eviction, hands an in-flight request a closed session, and the resulting failure appears at the call site rather than at the eviction that caused it. Single-flight on the build path is the other thing that had to be there. Without it, a burst of first requests for the same checkpoint each start their own load, and the cost shows up as memory rather than as an error, which is the hardest kind of load problem to attribute afterwards. Pairing Shipped in 0.4.17. |
Based on
main(R1 #1310 and R2 #1311 are merged). Branchfeat/go-router-lifecycle-main, rebased from the earlier stackedfeat/go-router-lifecycle.What and why
router/lifecycle.goAgentFactory func(ctx, Checkpoint) (*agent.Agent, error),WithAgentFactory,WithMaxLoaded(default 2).Load,Lease(refcounted: an evicted or unloaded agent the Router built closes only when its last lease is closed),Attach,Preload,Unload,UnloadAll,Loaded,MaxLoaded,Close,ErrClosed. Single-flight build with waiters; a failed build wakes them and is not cached; builds are serialised by one token (peak memory), a resident load never waits behind a cold one;Closewaits for a build in flightrouter/hooks.goLoadHook/EvictHook,LoadContext{Model, Agent, RunID, Router},EvictContext,BaseHookandHookFuncsgainOnLoad/OnEvict. Fired over the R2 dispatcher after every internal lock is released, evictions before the load, stopping at the first raising hook as Python doesscripts/gen_go_fixtures.pylifecyclefamily (275 cases, 2,145 ops) recorded from reallaya.routerwith a stubAgent:loaded()andmax_loadedafter every op, events with the agent label, errors, warnings. Only the newlifecyclekey is added togo_extra.json--min-tests)Python semantics kept: LRU order with touch on every resident load; eviction after publishing the new agent,
on_evictbeforeon_load;Attachraisesmax_loadedto fit and never evicts or fires hooks;Preloadraises it to fit the request plus everything resident and leaves a resident checkpoint where it is in the order; a build that loses a race withAttachkeeps the attached agent (and still fireson_loadfor it); unknown names fail before anything changes.Differences from Python
load()returns an Agent that lives as long as someone references itLoadreturns the agent, valid only while resident (eviction closes what the Router built).Leasekeeps it open untilClosemax_loadedbelow 1 is clamped to 1WithMaxLoaded(n<1)is an errorattachof an unknown name registers it*UnknownModelError(names are the three built-ins;registeris R5-optional)attachnever closes the agent (GC)preload([])loads nothingPreload(ctx)with no names loads all threeagent_kwargs,revision, digestsrouter: cannot load the english checkpoint: <cause>with%w; waiters getloading english failed on the call that was building it: <cause>. A builder's context cancellation is not inherited: the waiter builds for itselfctxunload()returns NoneUnload/UnloadAllreturn the checkpoints let golaya: hook Boom.on_load failedlaya: hook <GoTypeName>.on_load failed; timeout textlaya: hook <GoTypeName> exceeded 0.1s(as R2)hooks_concurrent=Falselock is anRLockLoad/Lease/Preload/Unloadon its own Router when that fires a hook (waits until a timeout). Documentedunload(K)racing a build of K can fireon_evictbefore that build'son_loadhooks.goLoad/Lease/Preload/Unload/UnloadAll/Closeon its own Router: it holds the build token and would wait for itself. Documented onAgentFactoryon_loadhook that unloads or closes for the same checkpointLoadreturns the already-closed agent and no error (documented); after a hook timeout the call gives its lease back, so an abandoned hook's agent may close (documented)close()Closeseals the Router, frees everything (waits for builds), is idempotent; loads after it returnErrClosedloaded_revisions,register/unregister/registeredEvidence
lifecycle): 35 hand-chosen sequences + 240 random, each op checks loaded(), max_loaded, events, errors, warningson_loadandon_evict, build failure, attach/preload/unload interleavingsgo_extra.jsonlifecyclekey added;--checkmatcheshooks_raise, serial hooks),-race-racetests (deterministic fake factory with gates)Lease.Closeunder a race counts once;Closeduring an in-flight build (builder keeps its lease, the agent closes at release); a hook callsLoaded,MaxLoaded,Loadwhile hooks run (outside the lock); a slowCloseof an evicted agent holds no lock; a resident load is not stalled by a cold build; eviction close error fails the call but delivers the events; a failingon_loadgives the lease back; 8-goroutine stress over every API, every built agent closed exactly once, attached agents nevergen_go_fixtures --check,gen_tables --check, workflow folding, ruffMutants
41 compiled, 41 killed (at most 3 per behavior).
Lease.Closenot once-only; retire closes under a leaseTestLeaseOutlivesEviction,TestClosingALeaseTwiceUnderARaceCountsOnce,TestConcurrentStressTestEvictionOrderIsLeastRecentlyUsed,TestLoadedOrderIsLRU,TestLifecycleGoldenTestFailedBuildIsNotCachedAndWakesWaiters,TestColdBuildsAreSerialisedTestFailedBuildIsNotCachedAndWakesWaiters,TestWaiterDoesNotInheritTheBuildersCancellation,TestPanickingFactoryWakesWaitersTestCloseWaitsForAnInFlightBuild,TestCloseFreesAndSealson_load; failing hook strands the lease;on_loadagent is the discarded build; first hook error does not stopTestHooksRunOutsideTheLock,TestLifecycleGolden,TestLeaseHeldAcrossTheLoadHook,TestFailingHookKeepsTheCheckpointAndGivesBackTheLease,TestAttachWhileBuildingKeepsTheAttachedAgentTestAttachRaisesMaxLoaded,TestPreloadFitsEverythingAndStages,TestAttachedAgentsAreNotClosed,TestAttachingTheSameInstanceIsSafe,TestPreloadResidentIsNotTouched,TestMaxLoadedMustBePositiveTestCloseErrorFailsTheLoadButNotTheEvents,TestASlowCloseHoldsNoLock,TestFactoryReturningNothing,TestUnloadFrees,TestUnloadWaitsForItsOwnBuildOnlyReview fixes (Opus): a panic in an agent's
Closeis turned into an error per agent, so it cannot wedge the Router (a doublefinishre-closed a channel under the lock;finishis now idempotent), skip the other closes or skipon_evict;Attachof an agent the Router built but evicted while leased hands it to the caller (shared ownership record), so the last lease release no longer closes a resident attached agent;LoadandPreloadkeep their lease acrosson_load; a build queued on the build token rechecksCloseandAttachbefore calling the factory. Each has a test and a mutant (7 more, 41 total, all killed).The mutant run found one real gap, fixed:
on_loadsaw the discarded duplicate whenAttachwon the race with a build (TestAttachWhileBuildingKeepsTheAttachedAgentnow pins it).Real-weights e2e (scratch under
laya-e2e/go-lane/e2e-router-lifecycle/, not in CI)Factory over the nested
onnxmodule (onnx.Open+agent.Open) for the three checkpoints,WithMaxLoaded(1), Python reference =ONNXAgenton the same fused graph,-race.TestZZRouterLifecycle)checkpoints/english, export self-verified vs torch), multilingual (onnx/fixed-multilingual), typed-decisions (laya-review-scratch/work, not inLAYA_FIXTURE_CHECKPOINTS)Closeload:english, evict:english, load:multilingual, evict:multilingual, load:typed-decisions, evict:typed-decisions, load:english, evict:englishgo 1, py 0.9334)Scanners (CodeQL, Semgrep, CVE gate, actionlint) and review: pending.
Related PRs
main)