Skip to content

feat(laya-ts): batch inference, tensor-plumbing perf fixes, and DX helpers - #340

Closed
nnlgsakib wants to merge 6 commits into
NandhaKishorM:mainfrom
nnlgsakib:feat/laya-ts-perf-usability
Closed

nnlgsakib wants to merge 6 commits into
NandhaKishorM:mainfrom
nnlgsakib:feat/laya-ts-perf-usability

Conversation

@nnlgsakib

Copy link
Copy Markdown
Contributor

Summary

Makes laya-ts faster for bulk workloads and easier to integrate, without changing any existing API semantics. Six themed commits, each building and passing tests in isolation:

  1. fix(ts) — loop-based tensor plumbing, cures spread overflow
  2. perf(ts) — BPE, question-prefix, and hooks fast-path caches
  3. feat(ts) — Agent.predictMany / Router.predictMany batch inference
  4. feat(ts) — q builders, result helpers, shortlist auto-embed
  5. feat(ts) — retrying loads with abort signal and progress
  6. docs(ts) — working examples and batch README

Motivation

  • Math.max(...lengths) throws RangeError past ~100k args, so large batches crashed instead of running.
  • Bulk triage ran N sequential forward passes for N states; the encoder/decoder round-trip also allocated the [B,L,H] hidden tensor up to 3x per call.
  • predictShortlist forced callers to hand-roll embed_fn despite the agent carrying an encoder.
  • Examples hardcoded ./model-ml, which exists in no checkout, so every example failed out of the box.

What changed

Perf / fixes (common.ts, providers.ts, shortlist.ts, tokenizer.ts, hooks.ts)

  • New maxOf(values, fallback) loop helper; collateItems, feed, feedHead, and shortlist embed widths use it.
  • toNested copies once with precomputed steps; i64, hidden_states, and marker_mask flatten straight into typed arrays (no flat(Infinity).map intermediates). Values and dims verified identical by tests/flat-parity.test.ts.
  • Per-merges BPE word cache (20k cap), per-tokenizer question-prefix cache (1k cap), composeHooks shared-empty fast path. All transparent to callers.

Batch API (agent.ts, router.ts)

  • predictMany/systemMany on both classes. Agent runs one encoder+head pass over all SxQ rows; _systemOneCore delegates to the same core, so single and batch paths can't diverge. Hook semantics unchanged (rewrite/skip/usage aggregation preserved).
  • Router routes each state (per-state onRoute), groups by model, loads each checkpoint once, preserves input order, attaches per-state routing. Agents without predictMany fall back to looped systemOne.

DX (dx.ts, index.ts, shortlist.ts)

  • New q.choice / q.score / q.noul factories (output passes checkQuestion), best(), topK(), isConfident(); all exported from the package root alongside maxOf.
  • predictShortlist(agent, state, questions) now defaults to embedFnFromAgent(agent) when shortlisting is actually needed; explicit embedFn callers unaffected.

Load robustness (providers.ts, agent.ts)

  • fetchWithRetry (2 retries on network errors and 429/5xx, 404s fail fast, aborts never retried); signal + onProgress(done, total, file) through loadNodeBundle, loadWebBundle, and Agent.load; versioned CacheStorage key (laya-ts-v1).

Examples / docs

  • New examples/try-batch.mjs (one-pass batch + routed batch with per-state choice/confidence output); try-ml.mjs and snake.mjs resolve the shipped weights via --model plus first-existing-dir fallback; README batch snippet.

Verification

  • Suite grows 165 -> 181 tests (6 new files: maxlen, flat-parity, prefix-cache, batch, dx, load-robustness); every commit was checked out in an isolated worktree with a clean tsc build and a green suite.
  • Live run on the shipped multilingual weights: batched: billing/billing/other, routing: english/multilingual/multilingual; try-ml.mjs reports Hindi refund -> billing 0.99.

Notes / deliberate non-changes

  • No measured wall-clock benchmark is included — the batch win is structural (1 forward pass vs N).
  • Kept QuestionPrefix.nOptions (exported type) and the PredictHookArg/HookArg split (callables valid as onPredictStart but correctly rejected as hooks); skipped a lang single-pass and cache TTL as not worth it at these sizes.

nnlgsakib added 6 commits September 24, 2026 11:29
Math.max(...lengths) throws RangeError past ~100k args, so any large
batch (bulk triage, big shortlist embed) crashed instead of running.
Replace every spread-max on batch-controlled lengths with explicit
loops, and while there remove the flat(Infinity)+map intermediates on
the encoder->head path, which allocated the [B,L,H] hidden tensor up
to three times per forward pass.

- common.ts: new maxOf(values, fallback) loop helper; collateItems
  derives L/K with loops instead of Math.max(...maps)
- providers.ts: feed/feedHead derive L/k with loops; toNested copies
  once with precomputed steps (no per-node slice/reduce); i64,
  hidden_states and marker_mask flatten straight into their typed
  arrays instead of flat(Infinity).map + TypedArray.from
- shortlist.ts: embed batch width via maxOf

No behavior change: tensor values and dims are identical, verified by
tests/flat-parity.test.ts. New tests/maxlen.test.ts proves a 300k-row
length list no longer throws.

Tests: 167 passed (incl. 4 new), tsc build clean.
Three hot paths redo identical work on every call: BPE merges run
O(L^2) per word even for words seen a thousand times, question
prefixes re-encode the same instructions per state, and every predict
allocates hook arrays that are empty in the common no-hooks case.
All three are transparent to callers.

- tokenizer.ts: per-merges word->pieces cache in bpeWord (cap 20k,
  cleared on overflow); single-char inputs keep keys collision-free
- common.ts: per-tokenizer question-prefix cache keyed by the full
  question JSON (cap 1k); sequenceWithState never mutates prefixes,
  so sharing the cached object is safe
- hooks.ts: composeHooks returns a shared empty list when no default,
  installed, or per-call hooks exist; dispatch only reads it

Correctness is guarded by tests/prefix-cache.test.ts (repeat call
returns the cached prefix) plus the existing BPE, sequence-prefix,
and hooks suites, which all pass unchanged.

Tests: 170 passed (incl. 1 new), tsc build clean.
Bulk triage ran N sequential forward passes for N states. Now one
encoder+head run covers all S*Q rows: prefixes build once per
question, per-state tokens are summed for usage, and answers decode
per (state, question) row. Hook semantics are unchanged -- start hooks
still see and may rewrite all states, skip() still short-circuits,
and end hooks still see aggregated usage.

- agent.ts: predictMany/systemMany; _predictHooked batches via the
  new _systemManyCore; _systemOneCore delegates to it, so single and
  batch paths share one implementation and stay identical
- router.ts: predictMany/systemMany route each state (firing onRoute
  per state), group indices by model, load each checkpoint once, keep
  input order, and attach per-state routing; agents without
  predictMany fall back to looped systemOne; a start hook that changes
  the state count triggers re-routing so decisions stay aligned
- tests/batch.test.ts: single encoder run for N states, batch equals
  looped systemOne answers, empty inputs, router grouping with one
  load per model and order preservation

Verified live on the shipped multilingual weights (try-batch.mjs):
3 states in one pass, routing english/multilingual/multilingual.

Tests: 175 passed (incl. 5 new), tsc build clean.
Two integration papercuts: building a valid question dict means
remembering the exact type/criteria shapes, and predictShortlist
required callers to hand-roll an embed_fn even though the agent
already carries an encoder. Both now have one-line answers.

- dx.ts (new, exported from the package root alongside maxOf):
  q.choice/q.score/q.noul factories whose output passes
  checkQuestion; best() argmax over choice/score/noul answers;
  topK() probability ranking; isConfident() gate (default 0.8)
- shortlist.ts: predictShortlist embedFn becomes optional; when a
  choice question actually needs shortlisting (k < n), it defaults to
  embedFnFromAgent(agent), which throws honestly if the agent has no
  encoder access. Passthrough questions never touch an encoder.
- tests/dx.test.ts: builder validity, helper semantics, and
  auto-embed end to end with a fake agent

Backwards compatible: explicit embedFn callers are unaffected.

Tests: 179 passed (incl. 4 new), tsc build clean.
Model downloads failed hard on the first transient error, could not
be cancelled, and gave no feedback while multi-hundred-MB weights
transferred. Loads now take signal/onProgress end to end, and the
browser cache key is versioned.

- providers.ts: fetchWithRetry (2 retries on network errors and
  429/5xx, 404s fail fast, aborts never retried); signal and
  onProgress(done, total, file) plumbed through loadNodeBundle,
  loadWebBundle, and Agent.load; CacheStorage key laya-ts-v1;
  shared TOKENIZER_CANDIDATES for the node/web tokenizer layouts
  (full loader unification skipped: 8 lines of duplication cost less
  than an IO-injection abstraction)
- agent.ts: Agent.load accepts and forwards signal/onProgress
- tests/load-robustness.test.ts: 503-then-success retry plus
  progress callbacks; pre-aborted signal throws without retrying

Skipped deliberately: cache TTL (online-first plus the browser HTTP
cache already covers freshness).

Tests: 181 passed (incl. 2 new), tsc build clean.
try-ml.mjs and try-batch.mjs hardcoded ./model-ml, which exists in no
checkout: running either failed with "Incompatible model" and fell
into a Hugging Face download of a bogus path. snake.mjs had the same
default with no fallback.

- examples/try-batch.mjs (new): predictMany over English/Hindi/German
  states in one forward pass, then Router.predictMany with per-state
  routing; prints choice + confidence + full distribution per state;
  attaches the local weights for both languages so the demo runs
  offline
- examples/try-ml.mjs, examples/snake.mjs: --model flag plus first-
  existing-dir fallback (./model-ml, laya-ts/examples/model-ml,
  ./examples/model-ml) so every example finds the shipped weights
  from either the repo root or the laya-ts dir
- README.md: try-batch run line and a 5-line predictMany/q-helper
  snippet

Verified live on the shipped multilingual weights: batched
billing/billing/other with routing english/multilingual/multilingual;
snake --ticks 5 runs on model weights; try-ml reports billing 0.99
on the Hindi refund request.
@nnlgsakib nnlgsakib closed this Sep 24, 2026
@nnlgsakib nnlgsakib reopened this Sep 24, 2026
@nnlgsakib

Copy link
Copy Markdown
Contributor Author

@NandhaKishorM take a look

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for all the work here, several of these pieces are genuinely valuable. We are going with #329 for batch inference because it mirrors Python's predict_batch names and per-request hook behaviour, so please drop predictMany and systemMany. Could you split the rest into small focused PRs: first the maxOf spread-overflow fix with the typed-array flattening and its parity test, then the retrying load with abort and progress, then the example path fixes. For the helpers, please have isConfident read answer_confidence (now in main via #386) rather than the entropy confidence, and include a quick measurement with the caches so we can judge them on their own.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Still holding on the split, which has not happened, and I want to restate what I would take so this is not just a stale request.

The decision on batch inference stands: #329 shipped it, because it mirrors Python's predict_batch naming and per-request hook behaviour, and having two spellings of the same thing in the TypeScript client is worse than either one alone. So predictMany and systemMany are out.

The rest I do want, in this order, as separate PRs:

  1. The maxOf spread-overflow fix with the typed-array flattening and its parity test. This is a correctness bug, not a perf change: a spread over a large enough tensor throws, so it is the one piece that fixes something broken rather than making it faster. I would merge this on its own today.
  2. The retrying load with an abort signal and progress. Useful, self-contained, and easy to review against the Python loader's behaviour.
  3. The caches (BPE, question prefix, hooks fast path), with a measured before and after. A cache is a correctness risk as well as a speed win, so this one wants the numbers and a test that the cached and uncached paths agree.
  4. The q builders and result helpers. DX surface, so mostly an API-shape conversation, and worth having after the others.

Note that laya-ts has moved since: #644, #664, #685, #723, #746 and #749 all touched src/agent.ts, and #750 is open against it too. So each of these needs a rebase onto current main rather than a rebase of this branch.

The branch conflicts now and splitting it is less work than resolving it. Thanks for the volume of work in here; the sequencing is the only thing in the way.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Closing this one. It is 2,187 commits behind main, the furthest of any open PR, and laya-ts is not the package this branch was written against: src/agent.ts, src/router.ts, src/common.ts, src/hooks.ts, src/providers.ts and src/shortlist.ts have all been rewritten or heavily extended since, and several of the things this PR adds now exist.

Specifically, on batch inference: the TypeScript SDK has predictBatch, and decideBatch plus answerConfidence landed in 0.4.11 through #750. Hooks are a full registry with timeouts and panic isolation. So the gap this PR opened against is closed, by different code.

I am not going to pretend the rest of it was absorbed: the DX helpers in src/dx.ts and the tensor-plumbing changes were your own design and nothing merged is a port of them. If any of them still buys something against today's laya-ts, the useful form is one small PR per helper against current main, because that is reviewable and this is not.

This was one of the earliest external contributions to the TypeScript port and it sat open while that port changed underneath it, which is on the repository rather than on you. Sorry it went that way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants