Repository navigation
fix(ts): cut stateText budgets at code points like Python - #1446
NandhaKishorM merged 1 commit into
Conversation
stateText sliced its 4000-character budget with String.slice, which counts UTF-16 units, so a state whose 4000th unit was the high half of a surrogate pair returned text ending in an orphaned surrogate -- malformed text Python's state_text never produces, and the same lone-surrogate shape laya-serve already refuses as a request (NandhaKishorM#454, NandhaKishorM#814). The segment scan and the 4000-character leaf samples cut the same way. cutCodePoints mirrors the email port's helper, added there when a budget cut an emoji in half. Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
jev_confidence no longer returns a value outside [0, 1] when finite weights sum to infinity (#1442), language detection flattens large states lazily (#1448), laya-ts counts and cuts code points like Python (#1445, #1446), and Router.predictBatch honours per-request token budgets (#1441). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged into 0.4.22, right after #1445. The argument I found most convincing is the one about shape rather than rate: an orphaned high surrogate is a string Python's Worth knowing for your next one: in Same note as on #1445 about the conflict between the two: resolved by keeping both helpers, nothing needed from you. |
|
Thanks for merging this one right alongside #1445 — and for confirming the cutCodePoints parity with email.ts down to the negative-count semantics. The port matrix being code-point-consistent everywhere now is a great state to leave it in. |
What
laya-ts/src/lang.tssliced its character budgets withString.slice(UTF-16 code units) at four sites:stateText's 4000-character cutnonEnglishSegment's per-segment cap (seg.slice(0, maxChars - seen)andseen += capped.length)leafNonEnglish, thealphaCountranking inanalyse)A local
cutCodePoints()helper — mirroring the helper of the same name and rule already shipped inlaya-ts/src/email.ts— now cuts at code points, so a surrogate pair is never split in half. A regression test is added in the file's established style.Why
Fixes #1444. When the 4000th UTF-16 unit was the high half of an emoji,
stateTextreturned text ending in an orphaned high surrogate: ill-formed detection text Python'sstate_textnever produces, the same lone-surrogate shape the serve side already refuses (#454, #814), and the same "budget cut an emoji in half" case the email port's helper was added for. The cut text is whatanalyse()/Routerread and whatmixedSegmentcan return to a caller.How it was verified
laya-tsneedsvitestto run its suite, which this host cannot install, so the change was verified by driving the patchedlang.tsdirectly (bun import, no installs) against Python'slaya.lang(imported standalone, stdlib-only):stateText("A".repeat(3999) + "😀" + "tail text that will be cut")now ends with the whole emoji (last code pointU+1F600), matching Python; onmainit ended in an orphaned\uD83Dmain, 0 mismatches with this patchcleanEmailBodyover 1200 generated emails: 0 mismatches before and after)Checklist
diacriticRatecounting fix is a separate PR)main(branched from0565524)vitestcannot be installed on this host; the standalone differential above is the verification offered, and CI runs the suite)Fixes #1444