Repository navigation
feat(ts): dx helpers, shortlist auto-embed, and encode caches - #414
Conversation
|
Thanks for splitting these out and for including measurements, and isConfident reading answer_confidence is exactly right. One real bug in the prefix cache: JSON.stringify drops undefined values, so criteria { a: undefined, b: 'x', c: 'y' } share a cache entry with { b: 'x', c: 'y' } and the model silently loses option 'a'. It also throws on BigInt descriptions that used to render fine. Keying on what actually builds the prefix, for example JSON.stringify([q.ins, renderOptions(q), maxLen, headMaxLen, optionOrder]), fixes both. Please add a test for the undefined-description case. Could you also Object.freeze the shared EMPTY_HOOKS array, and consider dropping the entropy fallback in isConfident so the gate always uses one scale? |
Two hot paths redo identical work on every call: question prefixes re-encode the same instructions per state, and every predict allocates hook arrays that are empty in the common no-hooks case. Both are transparent to callers. - common.ts: per-tokenizer question-prefix cache keyed by the full question JSON (cap 1k); sequenceWithState never mutates prefixes, so sharing the cached object is safe - hooks.ts: composeHooks returns a shared empty list when no default, installed, or per-call hooks exist; dispatch only reads it (BPE word caching was part of this change originally and is dropped: upstream NandhaKishorM#449 replaced the quadratic merge with a heap, subsuming it. Measured below.) Correctness is guarded by tests/prefix-cache.test.ts (repeat call returns the cached prefix) plus the existing sequence-prefix and hooks suites, which pass unchanged.
Two integration papercuts: building a valid question dict means remembering the exact type/criteria shapes, and predictShortlist required callers to hand-roll an embed_fn even though the agent already carries an encoder. Both now have one-line answers. - dx.ts (new, exported from the package root alongside maxOf): q.choice/q.score/q.noul factories whose output passes checkQuestion; best() argmax over choice/score/noul answers; topK() probability ranking; isConfident() gate (default 0.8) - shortlist.ts: predictShortlist embedFn becomes optional; when a choice question actually needs shortlisting (k < n), it defaults to embedFnFromAgent(agent), which throws honestly if the agent has no encoder access. Passthrough questions never touch an encoder. - tests/dx.test.ts: builder validity, helper semantics, and auto-embed end to end with a fake agent Backwards compatible: explicit embedFn callers are unaffected. Tests: 179 passed (incl. 4 new), tsc build clean.
Per review: the gate must use answer_confidence (max(p), comparable across question types) rather than the entropy confidence, which is not comparable across choice/score/noul. Falls back to confidence when answer_confidence is absent so the helper stays correct on checkpoints without the field. Covered by the new "prefers answer_confidence" test in tests/dx.test.ts.
Prefix cache keys on rendered options instead of raw criteria: JSON.stringify drops undefined descriptions (silently merging {a: undefined, b} with {b}) and throws on BigInt. Key is now [type, instructions, renderOptions, maxLen, headMaxLen, optionOrder] with a regression test for the undefined case.
Object.freeze the shared EMPTY_HOOKS array. isConfident gates on answer_confidence alone (one scale across types, no entropy fallback).
008f654 to
ba4904a
Compare
Implemented the requested review follow-ups:
|
|
Passed checks and merged. Thank you @nnlgsakib! It ships in 0.3.22. |
Summary
Small usability batch: one-line question builders and result
helpers,
predictShortlistwithout a hand-rolled embed function,plus the question-prefix cache, with measurements.
Changes
dx.ts(new, exported from package root):q.choice/q.score/q.noulfactories (output passescheckQuestion);best(),topK(),isConfident().isConfidentgates onanswer_confidencealone — one scale across types, no entropyfallback.
shortlist.ts:predictShortlistembedFnis now optional anddefaults to
embedFnFromAgent(agent)when a choice questionactually needs shortlisting (k < n). Explicit callers unaffected.
common.ts: per-tokenizer question-prefix cache keyed by[type, instructions, renderOptions, maxLen, headMaxLen, optionOrder]— i.e. what actually builds the prefix, sinceJSON.stringifyon raw criteria dropsundefineddescriptions(silently losing an option) and throws on BigInt. Regression test
included for the
undefinedcase.hooks.ts:composeHooksshared-empty fast path over a frozenEMPTY_HOOKSarray.Measurements (shipped mmBERT tokenizer)
buildQuestionPrefixcalls: 1.0ms (miss) ->0.1ms (hit), ~7x.
measurement granularity at predict level, kept as a one-line win.
Verification
tests/dx.test.ts(builders, helpers incl. theanswer_confidence-only gate, auto-embed end to end) andtests/prefix-cache.test.ts(cache hit,undefined-descriptionseparation, BigInt tolerance).
tscbuild clean.Notes
predictMany/systemMany) were dropped per review — feat(ts): port Agent.predict_batch and Router.route_batch/predict_batch #329 owns batch inferencewith
predict_batchnaming.dropped: perf(ts): BPE merging is quadratic in word length; 5.5 s -> 3 ms (1722x) at 16k chars #449 replaced the quadratic merge with a heap, which
subsumes it.