Repository navigation
feat(java): add Router.predictBatch, so batched decisions can run on a router - #1060
Merged
NandhaKishorM merged 7 commits intoOct 10, 2026
Merged
Conversation
c0bcb17 moved `Router.Builder.defaultCheckpoint` and `LayaRouter.Default` to multilingual with the reference, but four places describing them did not move: Router.withDefaults() javadoc "English as default" Builder.defaultCheckpoint javadoc "English by default, which is the reference's choice" laya-java/README.md advised setting MULTILINGUAL, now the default LayaRouter.Default <summary> "Default LayaCheckpoint.English, matching Python's default" Each now names multilingual, and the README's advice is inverted: set ENGLISH when traffic is known to be English. The `"default"` alias still resolves to english on both ports, which is what laya/router.py's alias table says, and the README now says so next to the default, because `load("default")` returning english would otherwise read as a bug. Comment and doc only: no code line changed.
… review findings
A second adversarial review, this time of the PR description against the
code, found the description right about the numbers and wrong about five
behaviours. Each is fixed here rather than reworded.
Event order. laya/router.py fires on_evict and then on_load; dispatchOutcome
fired load then evict, and its javadoc said "in that order". A hook keeping a
resident-set gauge saw one checkpoint too many between the two. Evictions
first now; evictionDispatches asserts the reference's sequence.
lease lost both events when closing the evicted agent threw. build publishes
the new slot, then closeAll throws, then releases and rethrows -- so a load
and an eviction happened, and lease never reached dispatchOutcome. load had
been fixed with a finally; lease and Router.predict (which leases) had not.
They now dispatch what is owed and rethrow, a hook failure attached as
suppressed rather than masking the close failure. preload gets the same
guard, though it raises maxLoaded first, so only a concurrent load can make
its build evict; no test, because one call cannot reach that path.
AsyncHook abandoned a pooled thread for the rest of the call. Hooks abandons
a thread it made for one hook; AsyncHook abandons an executor's thread, which
the executor reuses. On a single-thread executor, after one overrun every
later callback of the same call was refused:
end refused: laya: hook thread pool-1-thread-1 exceeded its deadline and
was abandoned; this call has moved on, so ctx.results() is refused
The task now releases its thread when it returns, and abandon and release
are decided under one gate, so a task finishing as the deadline fires cannot
leave its thread abandoned. aPooledThreadIsReleasedWhenTheOverrunEnds.
Javadoc that contradicted the shipped code: Hook still argued AsyncHook was
deliberately not ported and said a hook always runs on the calling thread;
PredictContext said nothing dispatches on_route.
README: concurrent(false) with an AsyncHook deadlocks only when the callback
calls back into the router or agent, not always; on_route's context has no
decision, so "correlate on ctx.decision" named a field that does not exist,
and the hook cannot replace the route as the reference's can; and the two
hazards the last commit recorded but did not document -- load() can return
an agent a hook unloaded during on_load, and a router-level
on_predict_start calling router.predict recurses.
A hook that throws on that owed dispatch is attached to the close failure as
suppressed, not substituted for it: anOwedHookFailureDoesNotMaskTheCloseFailure.
Mutants, each against this commit, unmutated control green:
load dispatched before evict 2 tests red
lease skips the owed dispatch 1 red
AsyncHook never releases 1 red
hook failure replaces close failure 1 red
abandon ignores the finished flag survives -- it guards a window between
the task's finally and the timeout firing
that no deterministic test can open
model-free suite 983 registered, 966 run, 0 failed
floors java.yml 966 and 967, release-java.yml 966
javadoc clean
check_workflow_folding 10 workflows clean
…troduced A third adversarial review, of 1acb2e9 itself, found that commit made one thing worse and left three of its own claims only partly true. The regression. 1acb2e9's AsyncHook released its thread unconditionally when the task ended. On a direct or caller-runs executor that thread is the caller's own -- including Hooks' `laya-hook-timeout` thread for a hook that already overran `hooks_timeout` -- so the release lifted an abandonment AsyncHook never made, and the overrunning hook's late writes were accepted again. Reviewer's probe: "late write ACCEPTED" at 1acb2e9, "refused" before it. AsyncHook now releases only an abandonment it made (`abandonedHere`). PredictContext also counts abandonments per thread here; that alone does not fix it -- an unconditional release still takes the count from 1 to 0 -- and the next commit removes it. anOuterAbandonmentSurvivesAnInnerAsyncHook pins it. The same gate now also covers the task STARTING: a callback that had not started when the deadline fired never runs. Previously the timeout path could read `runner == null` just before the task set it, skip the abandon, and let the callback write into a finished call. A sweep separating writes made before and after the caller gave up saw that once in 20,000 calls at a 1 ms deadline, and zero after this change. "A hook failure is attached, not substituted" now holds on every path, not only lease: load, unload and unloadAll dispatched in a `finally`, so a throwing on_evict replaced the close failure and skipped the remaining evictions. Errors count as well as RuntimeExceptions. "In the reference's order" now holds for predict too. laya/router.py routes and loads BEFORE on_predict_start, inside its try, and starts elapsed_ms after the load. Java took the lease inside Hooks.around, so the order was route, start, evict, load, end, and elapsed included a cold load. Router.predict now routes and leases first; a routing or loading failure is reported to on_error and on_predict_end with no start event, through the new Hooks.failedBeforeStart, where before a routing failure reached no hook. on_load fires when an attach wins the race with a build, as the reference's does (it dispatches after either branch). Tests that could not fail, now can: the pooled-thread test asserts the SAME thread ran both callbacks, and has a variant whose overrun throws (a release outside the finally survived before); an Error from a hook on the owed path. Not tested, and why: the start/expiry and finish/expiry races are timing windows no deterministic test can open. preload's owed dispatch is reachable only when another load evicts mid-preload. model-free suite 991 registered, 974 run, 0 failed floors java.yml 974 and 975, release-java.yml 974 javadoc clean check_workflow_folding 10 workflows clean
…reviewing 743fc47 found A fourth adversarial review, of 743fc47, found one regression and six gaps. The regression. 743fc47 moved the lease ahead of PredictContext's construction, so a hooked Router.predict with null questions routed, cold- loaded and could evict a resident checkpoint under maxLoaded, firing on_load, before the context constructor refused it -- a caller-controlled way to thrash loads. When routing or the load failed instead, the catch block's own PredictContext threw and the real failure was lost. Now state and questions are checked first, on both paths, as laya/router.py does before it routes. elapsedMs on a failure before start covered nothing: the context was built after the failure. The reference creates its context before route(), so a failed call's elapsed_ms covers routing and the failed load. Timed from before routing now; success is unchanged (after the load, before start). A hook that rethrows ctx.error() -- log and rethrow -- replaced the failure with "Self-suppression not permitted", because a throwable cannot suppress itself. 743fc47 put routing and load failures through that path under the default raising policy. Every addSuppressed site in Hooks and Router now skips that case; this includes Hooks.around, where it predates the branch. An interrupted wait left the callback writing into the finished call, both in AsyncHook (timed wait) and in Hooks' timeout thread. Both now cut it off exactly as a deadline does. An untimed AsyncHook wait ignores interrupts, so it always ends with the callback and has nothing to cut off; the javadoc says so. Per-thread abandonment counts are removed. `abandonedHere` is what fixes the nesting case; counts never exceed one, because a second abandonment of one thread needs an inner timeout on that same thread, and a direct executor runs synchronously and cannot time out. Hooks.failedBeforeStart is public, because Router is outside the hooks package and the setters it needs are not. It now takes the start time, refuses a null problem, and refuses a context that already has results or an error. Tests for six mutants that survived the whole suite: the hooked predict's lease released on success and on a throwing start hook; unloadAll's catch path, with two evictions so a skipped second one is visible; the model a load failure reports; elapsed and ctx.error on a failure before start; a rethrowing on_error; an Error from the factory. Plus null questions, and an interrupted wait on each of the two paths. model-free suite 1000 registered, 983 run, 0 failed floors java.yml 983 and 984, release-java.yml 983 javadoc clean check_workflow_folding 10 workflows clean
A fifth adversarial review, of 1d1b656, found no behavioural defect and verified its ten claims, but eight mutants on guards it added still passed the whole suite. Each now has a test that fails without it: null state refused before routing nullsAreRefusedBeforeAnyLoad both checks ahead of the no-hooks fast path (same test, both paths) self-suppression in around's on_error, rethrowingCtxErrorNeverReplaces- around's on_predict_end, and TheFailure failedBeforeStart's on_predict_end elapsedMs timed from before routing upper bound added: <= the call's own wall time (a start of 0 reported nanoTime's arbitrary origin, ~1.9 h, and passed) failedBeforeStart's two refusals failedBeforeStartRefusesMisuse The two interrupt tests no longer race a 300 ms sleep: the hook waits on a latch the test releases only after the caller has been interrupted and returned, so a descheduled CI thread cannot reorder them. Wording. Abandonment now happens on a deadline or an interrupt, and for an AsyncHook without any Policy timeout, so PredictContext's field doc, six @throws lines and the refusal message ("exceeded its deadline") said something false; they now say the call stopped waiting for the thread. closeAll's guard, and Router's own self-suppression check on lifecycle dispatch, are defensive and no test reaches them: a lifecycle hook is never shown the close failure, so it cannot rethrow it. The null-state message keeps the reference's guidance, as Agent's does: "pass a string, a map or a list". failedBeforeStart's javadoc no longer claims to detect a context mid-call. closeAll's one unguarded addSuppressed is guarded, which makes 1d1b656's "every addSuppressed site" true. model-free suite 1002 registered, 985 run, 0 failed floors java.yml 985 and 986, release-java.yml 985 javadoc clean check_workflow_folding 10 workflows clean
Python has Router.predict_batch; the Java Router had no batch path and did not implement BatchPredictor, so Decisions.decideBatch could not take a router at all. Stacked on NandhaKishorM#1057, whose Router.predict hook order this follows. Design - Router.Request(state, questions, options, maxLen, headMaxLen) carries the per-request keys the reference reads off each request dict (model, task, lang, lang_guess via RouteOptions; max_len, head_max_len). - predictBatch(List<Request>[, batchSize, sortByLength]) validates every request, routes them all, groups by checkpoint in order of first appearance, and within a checkpoint splits on question schema (order-sensitive), token budget and, only when the checkpoint has per-language temperatures, language. One lease per checkpoint, results restored to request order. - predictBatch(states, questions) implements BatchPredictor; predictBatch(states, questions, options) applies one pin to all, so decideBatch can be pinned with a lambda runner. - Hook order is the reference's predict_batch and NandhaKishorM#1057's predict: every on_route, then per checkpoint on_evict/on_load, then one start per request in order, inference, ends in reverse. Hooks.aroundGroup is the multi-request form of Hooks.around (usage totalled before any end; on a failure every started request gets onError then onPredictEnd; hook failures attached through NandhaKishorM#1057's self-suppression guard). - A routing failure reaches its request, and a load failure every request of that checkpoint, through NandhaKishorM#1057's Hooks.failedBeforeStart. - Agent gets a package-private predictBatch overload that seeds the budgets on its context, so agent-level hooks can still replace them. Deliberate differences from Python - predict_batch routes and loads outside its try, so a routing or load failure fires no predict events there; here it is reported as NandhaKishorM#1057's predict reports it. - No min_confidence/hooks/hooks_raise/hooks_timeout keywords: Router.predict has none either. Prediction carries no routing block. - Router-level ctx.maxLen/headMaxLen are honoured here but still not by Router.predict (Agent.predict takes no budget); documented in the README. - A null state is refused when the Request is built rather than by the agent. Evidence - Model-free: 1021 registered, 1003 run, 0 failed (18 checkpoint aborts). RouterBatchTest: 18 tests on stub agents (order and equality to predict, one pass per checkpoint, schema/budget/language splits, pins incl. alias, decideBatch on a router and pinned, lifecycle order with eviction, load and route failures before start, skip, failure path, refusals). - Real weights (multilingual + typed-decisions graphs): 1438 registered, 1437 run, 0 failed. RouterBatchParityTest: 7 mixed requests (auto-routed Hindi/Chinese, pins to both checkpoints, two schemas, one maxLen=32) equal per-request predict within 1e-4, usage exact. - Mutants, each killed: per-request groups, reversed result assignment, dropped shared pin, dropped per-request pin, load after the starts (lifecycleOrder), silent load failure, silent route failure, dropped budget, language never split, forward end order. - Floors: model-free 985 -> 1003, JDK matrix 986 -> 1004, release 985 -> 1003; 1003 exits 0 and 1004 exits 1.
…in what review found Review of d76eacb found four gaps. - Process-wide default hooks fired once per router request and again for the agent's forward pass: one default start hook over three batched requests fired four times, and Router.predict fired it twice. The reference runs both inner passes under _SKIP_DEFAULTS (router.py, predict and predict_batch), so Router.predict and Router.predictBatch now run the agent call inside Hooks.withoutDefaultHooks. The agent's installed hooks still run. The README's "default hook sees two predict pairs" row no longer applies and is dropped. - Nothing pinned that a start hook's replacement questions are the ones its request is asked; asking the request's original questions passed every test. hookQuestionsHonoured kills it. - Nothing pinned that a request whose own start hook throws is ended: registering it after its start dispatch passed every test. startHookFailure checks each started request, the throwing one included, gets onError then onPredictEnd exactly once and nothing reaches the model. - The README said batched results "equal" per-request predict; they match within 1e-4, which is what the real-weights test asserts. Evidence - Model-free: 1025 registered, 1007 run, 0 failed. Real weights: 1442 registered, 1441 run, 0 failed (RouterBatchParityTest passes). - Mutants, each killed by the named test: original questions (hookQuestionsHonoured), started.add after dispatch (startHookFailure), no default skip in predictBatch (defaultsOncePerRequestBatch), no default skip in predict (defaultsOncePerPredict). - Floors: model-free 1003 -> 1007, JDK matrix 1004 -> 1008, release 1003 -> 1007; 1007 exits 0 and 1008 exits 1.
Owner
|
Merged.
All 86 suites pass on the merged tree. The Java suite runs in CI, where both lanes stay advisory. See my note on #1059 about the test floor: four Java PRs each measured their own count on main, so the floor on |
This was referenced Oct 10, 2026
NandhaKishorM
added a commit
that referenced
this pull request
Oct 10, 2026
14 pull requests from 8 contributors. * the TypeScript tree parses again: an unterminated template literal shipped in 0.4.1 and stopped 22 of 34 test files loading (#1056) * x_jev_confidence on /v1/systemone, computed locally (#1066, #302) * a narrowed shortlist drops option_order with a warning instead of failing (#1063) * registering a name one edit from a built-in warns (#1055, #919) * compose forwards four knobs the runtime already read (#1053, #1054) * Java Router.predictBatch and Shortlist.predictTournament (#1060, #1061) * teacher-agreement evals, with the self-agreement ceiling (#1071, #919) Security: no malicious content found; the audit is recorded locally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM
pushed a commit
that referenced
this pull request
Oct 11, 2026
…e counts 1007/1008 were the pre-#1061 measurements, kept on purpose because #1057, #1059, #1060 and #1061 were each branched off main independently and none measured the merged tree, which carries all four. That measurement now exists: on merged main 6880462 (release 0.4.2), Actions run 38041041968: - job 114182848558 (jvm build + model-free tests) registered 1192, aborted 24, ran 1168, 0 failed, so this lane's floor moves 1007 -> 1168; - job 114182848539 (jvm model-free tests, jdk 17) registered 1192, aborted 23, ran 1169, 0 failed, so the matrix lane's floor moves 1008 -> 1169. check_test_results.py counts what ran (tests - skipped), so the floors are 1168/1169 and not 1192. The matrix lane's floor is one higher on purpose: it passes -PtestJavaVersion, which enables TestJvmVersionTest there; the --require-passed guard for it is unchanged, as are --allow-aborted and every unrelated parity floor. This tightens a loss-detection guard; it fixes no code failure, and the old floors sat 161 below what actually runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1057; review
7900dba...HEAD. This PR makes the batch path use the same hook order that #1057 givesRouter.predict, and reuses its helpers (Hooks.failedBeforeStartand the self-suppression guard). It also changesRouter.predictitself: process-wide default hooks used to fire there twice, and now fire once.Gap
Python has
Router.predict_batch(laya/router.py). The JavaRouterhad no batch path and did not implementBatchPredictor, soDecisions.decideBatchcould not take a router at all.Design
Router.Request(state, questions, options, maxLen, headMaxLen)holds the per-request keys the reference reads off each request dict:model,task,langandlang_guessthroughRouteOptions, plusmax_lenandhead_max_len. It is built withRequest.of(state, questions)and withers.predictBatch(List<Request>[, batchSize, sortByLength])works in these steps:Agent.predictBatchcall when they share a question schema (order-sensitive) and a token budget. Language splits a call only when the checkpoint has per-language temperatures, which is the reference'slang_keyrule.Each checkpoint is leased once, and results come back in request order.
predictBatch(states, questions)implementsBatchPredictor, with each state routed on its own.predictBatch(states, questions, options)applies one set of routing options to every state. That covers what upstream 98a7b4d did fordecide_batch(model=...):Decisions.decideBatch((s, q) -> router.predictBatch(s, q, options), states, schema).Hook order matches the reference's
predict_batchand fix(java): router hooks review fixes that #1046's merge did not include #1057'spredict:on_route.on_evict/on_load.Because the checkpoint is loaded before the starts,
elapsedMsdoes not include a cold load.Hooks.aroundGroupis the multi-request form ofHooks.around. Usage is totalled before any request ends. If the checkpoint's inference fails, every started request getsonErrorand thenonPredictEnd. Hook failures are attached to the original failure using fix(java): router hooks review fixes that #1046's merge did not include #1057's guard.Failures before any start: a routing failure is reported to its own request, and a load failure to every request of that checkpoint, through fix(java): router hooks review fixes that #1046's merge did not include #1057's
Hooks.failedBeforeStart.Process-wide default hooks fire once per request. The reference runs the agent's forward pass under
_SKIP_DEFAULTSin bothpredictandpredict_batch.Router.predict(an existing gap on fix(java): router hooks review fixes that #1046's merge did not include #1057's base) andpredictBatchnow run the agent call insideHooks.withoutDefaultHooks. Before this change, one default start hook fired 2 times perpredictand 4 times over a 3-request batch. The agent's own installed hooks still run once per forward pass. This retires the README warning that a default hook "sees two predict pairs" and double-counts.Agentgets a package-privatepredictBatchoverload that puts the budgets on its own context, so agent-level hooks can still replace them.Deliberate differences from Python
predict_batchroutes and loads outside itstry, so a routing or load failure there fires no predict events. Here such failures are reported the way fix(java): router hooks review fixes that #1046's merge did not include #1057'spredictreports them.min_confidence,hooks,hooks_raiseorhooks_timeoutkeywords, because Java'sRouter.predicthas none either.Predictionhas noroutingblock.ctx.maxLen/ctx.headMaxLenare honoured here.Router.predictstill ignores them, becauseAgent.predicttakes no budget. The README documents this.Requestis built. Python rejects it later, inside the agent.decideBatchtakes no routing keywords. A pin is a lambda runner, and a budget is aRequestlist.Evidence
./gradlew test javadoc --rerun-tasks, Corretto 17): 1025 registered, 1007 ran, 0 failed. The 18 that did not run are tests that need a checkpoint.RouterBatchTesthas 22 tests on stub agents, each checkpoint with its own recording session:predictexactly on stubs; on real weights they match within 1e-4modelpin, including an alias, overrides detectiondecideBatchon a router, plain and pinnedlifecycleOrder)hookQuestionsHonoured)startHookFailure)predictBatchand once perpredict(defaultsOncePerRequestBatch,defaultsOncePerPredict)LAYA_CHECKPOINTS+ multilingual graph + typed-decisions graph): 1,442 registered, 1,441 ran, 1 skipped (TestJvmVersionTest), 0 failed.RouterBatchParityTestsends 7 mixed requests: auto-routed Hindi and Chinese, pins to both checkpoints, two schemas, and one withmaxLen=32that is shown to truncate. Each result equals per-requestpredictwithin 1e-4, and usage matches exactly; the budgeted request is compared with its checkpoint's agent at the same budget, sinceRouter.predicttakes none. The test aborts without the environment variables. No CI parity cell has both graphs, so CI does not run it.oneCallPerCheckpointorderPreservedAndEqualToPredict,languageSplitsOnlyWhenTempered; alsoRouterBatchParityTeston the pre-rebase commitpinForwarded,decideBatchOnRouterpinForwardedlifecycleOrderand othersloadFailureBeforeStartrouteFailureBeforeStartbudgetForwardedlanguageSplitsOnlyWhenTemperedhookQuestionsHonouredstartHookFailurepredictBatchdefaultsOncePerRequestBatchpredictdefaultsOncePerPredicthooksPerRequest,lifecycleOrder,failureEndsEveryStartedRequestEnd-to-end on real weights
Rig. Corretto 17. Checkpoints
english,multilingualandtyped-decisions(typed-decisions at pinned revision55cf4c4), the exportedfixed-multilingual/laya.onnxgraph and the typed-decisions graph, run through./gradlew test --rerun-taskswithLAYA_CHECKPOINTS,LAYA_ONNX_GRAPHandLAYA_TYPED_ONNX_GRAPHset. The only skip in the run isTestJvmVersionTest, which is enabled only in the JDK-matrix lane.TestJvmVersionTest)RouterBatchParityTestsends 7 mixed requests through oneRouter.predictBatchcall, routed across the multilingual and typed-decisions checkpoints. Each result is compared with the same request sent alone throughRouter.predict(themaxLen=32one through its checkpoint's agent, sinceRouter.predicttakes no budget): every answer matches within 1e-4, andusage()is identical. An earlier revision also ran the ordering mutant (results returned out of request order) on real weights, and this test failed on it. CI has no parity cell with both graphs, so this test runs only on a local rig with both.Floors
Re-derived on #1057's tree, which ran 985 model-free tests. Each floor is the exact count of tests that ran:
java.ymlmodel-free: 985 → 1007java.ymlJDK matrix: 986 → 1008release-java.yml: 985 → 1007--min-tests 1007exits 0 and--min-tests 1008exits 1. Parsing the workflow YAML shows the shell receives the full commands, andcheck_workflow_folding.pyreports the workflows clean.Diff (
7900dba...HEAD, 2 commits)