Repository navigation
fix(ts): count code points in latinProfile like Python - #1445
Conversation
diacriticRate divided by lowered.length, which counts UTF-16 code units, so every astral character counted as two and understated the rate the way the email port's header already warns about. Python divides by len(lowered) -- code points -- and so do the Go, Rust, Java and C++ ports. On text at the NON_EN_DIACRITIC_RATE threshold the understated rate flipped the verdict: analyse() answered isEnglish: true for a one-diacritic state that Python routes off the English checkpoint, and any text carrying emoji reported a lower rate than the reference. The shouted-word length bars counted units the same way. Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
jev_confidence no longer returns a value outside [0, 1] when finite weights sum to infinity (#1442), language detection flattens large states lazily (#1448), laya-ts counts and cuts code points like Python (#1445, #1446), and Router.predictBatch honours per-request token budgets (#1441). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged into 0.4.22. I checked the central claim, that this port was the only one counting UTF-16 units, and it holds exactly:
The The threshold case is the part that earns the merge: two emoji moving the rate from 1/51 to 1/49 across On the unchecked box: that is fine, CI ran the suite here and it is green, and your standalone differential was the right substitute to offer rather than claiming a run you could not do. Note that this and #1446 both add a helper at the same point in |
|
Thank you for the careful verification and the merge! Knowing you cross-checked the central claim against the other four ports independently means a lot — that was exactly the standard the differential harness tried to hold itself to. Glad it landed into 0.4.22. |
What
laya-ts/src/lang.tscounted characters withString.length(UTF-16 code units) at four sites where Python'slang.pycounts code points:latinProfile'sdiacRatedenominator (the main routing-relevant one)shoutedEvidenceanalyseTextA local
width()helper (mirroringemail.ts's helper of the same name and rule) now counts code points at those sites, and a regression test is added in the file's established style with expectations generated from Python'slaya.lang.Why
Fixes #1443. Every astral character counted as two, understating
diacriticRateon emoji-bearing text — at theNON_EN_DIACRITIC_RATEthreshold that flippedanalyse()'s verdict against Python and routed a state to the English checkpoint that Python keeps off it. Rust, Go, Java and C++ all count code points here; the TS port was the outlier, and the email port already documents this exact class in its header.How it was verified
laya-tsneedsvitestto run its suite, which this host cannot install, so the change was verified by driving the patchedlang.tsdirectly (bun import, no installs) against Python'slaya.lang(imported standalone, stdlib-only):analyse("é😀😀" + "-".repeat(46))→isEnglish: true,diacriticRate: 0.0196isEnglish: false,diacriticRate: 0.0204, matching Python exactlymain, 0 with this patchanalyse()output: 257 rate mismatches onmain, 0 with this patchlaya.lang.latin_profile/analyseon the same inputs and re-checked against the patched TS (0.0204 / 0.1739 / 0.1)Checklist
main(branched from0565524)vitestcannot be installed on this host; the standalone differential above is the verification offered, and CI runs the suite)Fixes #1443