Skip to content

fix(ts): count code points in latinProfile like Python - #1445

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
rahul05ranjan:fix/ts-diac-rate-code-points
Nov 3, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
rahul05ranjan:fix/ts-diac-rate-code-points

Conversation

@rahul05ranjan

Copy link
Copy Markdown
Contributor

What

laya-ts/src/lang.ts counted characters with String.length (UTF-16 code units) at four sites where Python's lang.py counts code points:

  • latinProfile's diacRate denominator (the main routing-relevant one)
  • the two shouted-word bars in shoutedEvidence
  • the all-caps run bar in analyseText

A local width() helper (mirroring email.ts's helper of the same name and rule) now counts code points at those sites, and a regression test is added in the file's established style with expectations generated from Python's laya.lang.

Why

Fixes #1443. Every astral character counted as two, understating diacriticRate on emoji-bearing text — at the NON_EN_DIACRITIC_RATE threshold that flipped analyse()'s verdict against Python and routed a state to the English checkpoint that Python keeps off it. Rust, Go, Java and C++ all count code points here; the TS port was the outlier, and the email port already documents this exact class in its header.

How it was verified

laya-ts needs vitest to run its suite, which this host cannot install, so the change was verified by driving the patched lang.ts directly (bun import, no installs) against Python's laya.lang (imported standalone, stdlib-only):

  • Before/after at the threshold case from the issue:
    • before: analyse("é😀😀" + "-".repeat(46)) → isEnglish: true, diacriticRate: 0.0196
    • after: → isEnglish: false, diacriticRate: 0.0204, matching Python exactly
  • Differential over 800 generated mixed-script/alphabet states (script, state text, latin profile, language guess): 26 mismatches on main, 0 with this patch
  • Differential over 600 astral-heavy texts on the full analyse() output: 257 rate mismatches on main, 0 with this patch
  • The three new test expectations were generated with Python's laya.lang.latin_profile/analyse on the same inputs and re-checked against the patched TS (0.0204 / 0.1739 / 0.1)

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (branched from 0565524)
  • Tests pass locally (vitest cannot be installed on this host; the standalone differential above is the verification offered, and CI runs the suite)
  • Docs or examples updated when the public API changed (no public API change)

Fixes #1443

diacriticRate divided by lowered.length, which counts UTF-16 code units,
so every astral character counted as two and understated the rate the
way the email port's header already warns about. Python divides by
len(lowered) -- code points -- and so do the Go, Rust, Java and C++
ports. On text at the NON_EN_DIACRITIC_RATE threshold the understated
rate flipped the verdict: analyse() answered isEnglish: true for a
one-diacritic state that Python routes off the English checkpoint, and
any text carrying emoji reported a lower rate than the reference. The
shouted-word length bars counted units the same way.

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
@NandhaKishorM
NandhaKishorM merged commit c0d5479 into NandhaKishorM:main Nov 3, 2026
65 checks passed
NandhaKishorM added a commit that referenced this pull request Nov 3, 2026
jev_confidence no longer returns a value outside [0, 1] when finite
weights sum to infinity (#1442), language detection flattens large
states lazily (#1448), laya-ts counts and cuts code points like
Python (#1445, #1446), and Router.predictBatch honours per-request
token budgets (#1441).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged into 0.4.22. I checked the central claim, that this port was the only one counting UTF-16 units, and it holds exactly:

  • Python lang.py: diac / max(1, len(lowered)), code points
  • Rust: lowered.chars().count().max(1)
  • Go: utf8.RuneCountInString(lowered)
  • Java: codePointLength(lowered)
  • laya-ts before this: lowered.length, the outlier

The width() helper is also byte-identical to the one email.ts already ships, so this adds no new convention.

The threshold case is the part that earns the merge: two emoji moving the rate from 1/51 to 1/49 across NON_EN_DIACRITIC_RATE is enough to flip analyse() and route a state onto the English checkpoint that Python keeps off it, and that is a wrong answer rather than a cosmetic difference.

On the unchecked box: that is fine, CI ran the suite here and it is green, and your standalone differential was the right substitute to offer rather than claiming a run you could not do. Note that this and #1446 both add a helper at the same point in lang.ts, so they conflicted on merge; I resolved it by keeping both helpers verbatim, yours first. Nothing needed from you.

@rahul05ranjan

Copy link
Copy Markdown
Contributor Author

Thank you for the careful verification and the merge! Knowing you cross-checked the central claim against the other four ports independently means a lot — that was exactly the standard the differential harness tried to hold itself to. Glad it landed into 0.4.22.

@xinpingzh xinpingzh mentioned this pull request Nov 6, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

laya-ts: diacriticRate counts UTF-16 code units, so emoji flip analyse()'s English verdict against Python

2 participants