Skip to content

feat(ts): dx helpers, shortlist auto-embed, and encode caches - #414

Merged
NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
nnlgsakib:ts-helpers
Sep 29, 2026
Merged

NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
nnlgsakib:ts-helpers

Conversation

@nnlgsakib

@nnlgsakib nnlgsakib commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Small usability batch: one-line question builders and result
helpers, predictShortlist without a hand-rolled embed function,
plus the question-prefix cache, with measurements.

Changes

  • dx.ts (new, exported from package root): q.choice / q.score /
    q.noul factories (output passes checkQuestion); best(),
    topK(), isConfident(). isConfident gates on
    answer_confidence alone — one scale across types, no entropy
    fallback.
  • shortlist.ts: predictShortlist embedFn is now optional and
    defaults to embedFnFromAgent(agent) when a choice question
    actually needs shortlisting (k < n). Explicit callers unaffected.
  • common.ts: per-tokenizer question-prefix cache keyed by
    [type, instructions, renderOptions, maxLen, headMaxLen, optionOrder] — i.e. what actually builds the prefix, since
    JSON.stringify on raw criteria drops undefined descriptions
    (silently losing an option) and throws on BigInt. Regression test
    included for the undefined case.
  • hooks.ts: composeHooks shared-empty fast path over a frozen
    EMPTY_HOOKS array.

Measurements (shipped mmBERT tokenizer)

  • 200 identical buildQuestionPrefix calls: 1.0ms (miss) ->
    0.1ms (hit), ~7x.
  • Hooks fast-path saves one array spread per predict: below
    measurement granularity at predict level, kept as a one-line win.

Verification

  • New tests/dx.test.ts (builders, helpers incl. the
    answer_confidence-only gate, auto-embed end to end) and
    tests/prefix-cache.test.ts (cache hit, undefined-description
    separation, BigInt tolerance).
  • Full suite 378 passed, tsc build clean.

Notes

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks for splitting these out and for including measurements, and isConfident reading answer_confidence is exactly right. One real bug in the prefix cache: JSON.stringify drops undefined values, so criteria { a: undefined, b: 'x', c: 'y' } share a cache entry with { b: 'x', c: 'y' } and the model silently loses option 'a'. It also throws on BigInt descriptions that used to render fine. Keying on what actually builds the prefix, for example JSON.stringify([q.ins, renderOptions(q), maxLen, headMaxLen, optionOrder]), fixes both. Please add a test for the undefined-description case. Could you also Object.freeze the shared EMPTY_HOOKS array, and consider dropping the entropy fallback in isConfident so the gate always uses one scale?

nnlgsakib added 4 commits September 25, 2026 17:36
Two hot paths redo identical work on every call: question prefixes
re-encode the same instructions per state, and every predict
allocates hook arrays that are empty in the common no-hooks case.
Both are transparent to callers.

- common.ts: per-tokenizer question-prefix cache keyed by the full
  question JSON (cap 1k); sequenceWithState never mutates prefixes,
  so sharing the cached object is safe
- hooks.ts: composeHooks returns a shared empty list when no default,
  installed, or per-call hooks exist; dispatch only reads it

(BPE word caching was part of this change originally and is dropped:
upstream NandhaKishorM#449 replaced the quadratic merge with a heap, subsuming
it. Measured below.)

Correctness is guarded by tests/prefix-cache.test.ts (repeat call
returns the cached prefix) plus the existing sequence-prefix and
hooks suites, which pass unchanged.
Two integration papercuts: building a valid question dict means
remembering the exact type/criteria shapes, and predictShortlist
required callers to hand-roll an embed_fn even though the agent
already carries an encoder. Both now have one-line answers.

- dx.ts (new, exported from the package root alongside maxOf):
  q.choice/q.score/q.noul factories whose output passes
  checkQuestion; best() argmax over choice/score/noul answers;
  topK() probability ranking; isConfident() gate (default 0.8)
- shortlist.ts: predictShortlist embedFn becomes optional; when a
  choice question actually needs shortlisting (k < n), it defaults to
  embedFnFromAgent(agent), which throws honestly if the agent has no
  encoder access. Passthrough questions never touch an encoder.
- tests/dx.test.ts: builder validity, helper semantics, and
  auto-embed end to end with a fake agent

Backwards compatible: explicit embedFn callers are unaffected.

Tests: 179 passed (incl. 4 new), tsc build clean.
Per review: the gate must use answer_confidence (max(p), comparable
across question types) rather than the entropy confidence, which is
not comparable across choice/score/noul. Falls back to confidence
when answer_confidence is absent so the helper stays correct on
checkpoints without the field. Covered by the new
"prefers answer_confidence" test in tests/dx.test.ts.
Prefix cache keys on rendered options instead of raw criteria: JSON.stringify drops undefined descriptions (silently merging {a: undefined, b} with {b}) and throws on BigInt. Key is now [type, instructions, renderOptions, maxLen, headMaxLen, optionOrder] with a regression test for the undefined case.

Object.freeze the shared EMPTY_HOOKS array. isConfident gates on answer_confidence alone (one scale across types, no entropy fallback).
@nnlgsakib

Copy link
Copy Markdown
Contributor Author

Thanks for splitting these out and for including measurements, and isConfident reading answer_confidence is exactly right. One real bug in the prefix cache: JSON.stringify drops undefined values, so criteria { a: undefined, b: 'x', c: 'y' } share a cache entry with { b: 'x', c: 'y' } and the model silently loses option 'a'. It also throws on BigInt descriptions that used to render fine. Keying on what actually builds the prefix, for example JSON.stringify([q.ins, renderOptions(q), maxLen, headMaxLen, optionOrder]), fixes both. Please add a test for the undefined-description case. Could you also Object.freeze the shared EMPTY_HOOKS array, and consider dropping the entropy fallback in isConfident so the gate always uses one scale?

Implemented the requested review follow-ups:

  • Fixed the prefix cache key to use the values that actually build the prefix.
  • Added a regression test for undefined descriptions so cache entries stay separate.
  • BigInt descriptions are now handled without the cache key throwing.
  • Froze the shared EMPTY_HOOKS array.
  • Updated isConfident() to use answer_confidence only, with no entropy fallback.
  • Full suite passes: 378 tests, with a clean TypeScript build.

@NandhaKishorM
NandhaKishorM merged commit 6482abb into NandhaKishorM:main Sep 29, 2026
20 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @nnlgsakib! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants