Skip to content

fix(ts): cut stateText budgets at code points like Python - #1446

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
rahul05ranjan:fix/ts-lang-slice-code-points
Nov 3, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
rahul05ranjan:fix/ts-lang-slice-code-points

Conversation

@rahul05ranjan

Copy link
Copy Markdown
Contributor

What

laya-ts/src/lang.ts sliced its character budgets with String.slice (UTF-16 code units) at four sites:

  • stateText's 4000-character cut
  • nonEnglishSegment's per-segment cap (seg.slice(0, maxChars - seen) and seen += capped.length)
  • the two 4000-character leaf samples (leafNonEnglish, the alphaCount ranking in analyse)

A local cutCodePoints() helper — mirroring the helper of the same name and rule already shipped in laya-ts/src/email.ts — now cuts at code points, so a surrogate pair is never split in half. A regression test is added in the file's established style.

Why

Fixes #1444. When the 4000th UTF-16 unit was the high half of an emoji, stateText returned text ending in an orphaned high surrogate: ill-formed detection text Python's state_text never produces, the same lone-surrogate shape the serve side already refuses (#454, #814), and the same "budget cut an emoji in half" case the email port's helper was added for. The cut text is what analyse()/Router read and what mixedSegment can return to a caller.

How it was verified

laya-ts needs vitest to run its suite, which this host cannot install, so the change was verified by driving the patched lang.ts directly (bun import, no installs) against Python's laya.lang (imported standalone, stdlib-only):

  • The issue's case: stateText("A".repeat(3999) + "😀" + "tail text that will be cut") now ends with the whole emoji (last code point U+1F600), matching Python; on main it ended in an orphaned \uD83D
  • Differential over 400 generated states with surrogate pairs landing on and around the 4000-unit boundary: every in-pair cut diverged on main, 0 mismatches with this patch
  • The email port's own behavior is untouched (differential of cleanEmailBody over 1200 generated emails: 0 mismatches before and after)

Checklist

  • Focused on one change (the related diacriticRate counting fix is a separate PR)
  • Rebased on the latest main (branched from 0565524)
  • Tests pass locally (vitest cannot be installed on this host; the standalone differential above is the verification offered, and CI runs the suite)
  • Docs or examples updated when the public API changed (no public API change)

Fixes #1444

stateText sliced its 4000-character budget with String.slice, which
counts UTF-16 units, so a state whose 4000th unit was the high half of
a surrogate pair returned text ending in an orphaned surrogate --
malformed text Python's state_text never produces, and the same
lone-surrogate shape laya-serve already refuses as a request (NandhaKishorM#454,
NandhaKishorM#814). The segment scan and the 4000-character leaf samples cut the
same way. cutCodePoints mirrors the email port's helper, added there
when a budget cut an emoji in half.

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
@NandhaKishorM
NandhaKishorM merged commit b939072 into NandhaKishorM:main Nov 3, 2026
65 checks passed
NandhaKishorM added a commit that referenced this pull request Nov 3, 2026
jev_confidence no longer returns a value outside [0, 1] when finite
weights sum to infinity (#1442), language detection flattens large
states lazily (#1448), laya-ts counts and cuts code points like
Python (#1445, #1446), and Router.predictBatch honours per-request
token budgets (#1441).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged into 0.4.22, right after #1445. cutCodePoints is byte-identical to the helper email.ts already carries, negative-count semantics included, so stateText(text, -1) keeps matching Python's text[:-1]; I checked that against CPython rather than assuming it.

The argument I found most convincing is the one about shape rather than rate: an orphaned high surrogate is a string Python's state_text can never return, and laya-serve already refuses that exact shape as malformed input (#454, #814). So the old cut could hand a caller, through mixedSegment, a string the rest of the stack treats as invalid.

Worth knowing for your next one: in nonEnglishSegment you now write seen += [...capped].length, and after #1445 merged there is a width() helper in the same file doing exactly that count. Both are correct and I left yours verbatim rather than edit a contributor's merge, but width(capped) would be the one to reach for later. Also note the negative branch is unreachable at that call site, since the loop returns when seen >= maxChars before the subtraction, so it is defensive rather than load-bearing there.

Same note as on #1445 about the conflict between the two: resolved by keeping both helpers, nothing needed from you.

@rahul05ranjan

Copy link
Copy Markdown
Contributor Author

Thanks for merging this one right alongside #1445 — and for confirming the cutCodePoints parity with email.ts down to the negative-count semantics. The port matrix being code-point-consistent everywhere now is a great state to leave it in.

@xinpingzh xinpingzh mentioned this pull request Nov 6, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

laya-ts: stateText cuts its 4000-character budget inside a surrogate pair, returning an orphaned high surrogate

2 participants