Repository navigation
fix(lang): count letters the interpreter's Unicode database does not know - #1334
Merged
NandhaKishorM merged 1 commit intoOct 24, 2026
Merged
Conversation
…know str.isalpha() and unicodedata.category answer from the Unicode database bundled with the host interpreter, and the supported versions disagree: Python 3.10 ships Unicode 13.0, 3.11 14.0, 3.12 15.0, 3.13 15.1. A letter added after that database is category Cn -- *unassigned* -- so isalpha() is False and _script_counts dropped it before its "other" branch could count it, which that branch's own comment exists to prevent. Text written in Kawi, Nag Mundari, Todhri, Vithkuqi, Old Uyghur, CJK extension I, ... then reported script="unknown", i.e. "no letters detected", and Router._route reads that as "use the default checkpoint" rather than as non-Latin text. With Router(default= "english") - the documented setting for a mostly-English deployment - that is a misroute to the checkpoint that collapses off English; and laya.is_english() / analyse() gave different answers for the same input on different supported Pythons. _is_letter() now consults a generated table of letters that Unicode 16.0 assigns and Unicode 14.0 does not (plus the scripts Unicode 14 introduced, so Python 3.10 matches too), gated on the character being Cn here: a character this interpreter already knows is still decided by isalpha() alone. Verified by comparing _script_counts for all 1 114 112 codepoints, old vs new, on three interpreters: Unicode 16.0 -> 0 differences; Unicode 15.1 -> 4302, every one a codepoint 15.1 does not know; Unicode 14.0 -> 9272 of the same kind; and on each, zero codepoints the interpreter knows changed answer.
NandhaKishorM
added a commit
that referenced
this pull request
Oct 24, 2026
perf(lang): optimize unicode script detection with binary search Add/add conflict against #1334, which put its _POST_DATABASE_LETTER_RANGES table at the same point in the file. Both wanted and independent, so this is a union: the flattened bisect tables sit next to the _SCRIPT_RANGES they derive from, then #1334's letter table and _is_letter. The two function bodies merged without conflict and compose correctly. _script_counts now rejects via #1334's _is_letter, then bisects, and #1349's if/else preserves what #1334's for/else did for a letter no range claims, which still lands in counts["other"]. Verified equivalent rather than assumed. A bisect over flattened ranges only matches a first-hit linear scan when no two ranges overlap, so both were checked: 0 overlapping pairs, and a sweep of the entire code space, all 1114112 points from U+0000 to U+10FFFF, finds 0 disagreements with the old scan. _script_counts runs 0.125 ms on 1.5k mixed-script characters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM
added a commit
that referenced
this pull request
Oct 24, 2026
33 PRs. Server hardening (#1331, #1332, #1333), TypeScript parity (#1330, #1336, #1337, #1338, #1339, #1340, #1341, #1342, #1343), fail-closed input handling in evals and the integrations (#1317, #1318, #1319, #1320, #1321, #1322, #1326, #1335), exact and faster script detection (#1334, #1349), JVM and .NET resource handling (#1327, #1328, #1329, #1346), port goldens and clipping (#1324 excluded, #1325, #1344, #1345), CI (#1323, #1347, #1348). Verified before tagging: all 92 suites pass, 610 tests. ruff clean on both lists. laya-ts 1247 tests across 44 files, tsc --noEmit clean. Version agrees across pyproject.toml, laya.__version__ and nix/package.nix. README carries 0.4.15 only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Centaurioun
pushed a commit
to Centaurioun/laya
that referenced
this pull request
Oct 25, 2026
Clears the only job-level failure at 0.4.15. `jvm build + model-free tests` has been red on main since the release, at the step "Model-free fixture drift vs. committed", and the Java lane still reported success because that job is continue-on-error. My own miss. NandhaKishorM#1334 added _POST_DATABASE_LETTER_FLOOR to laya/lang.py and gen_fixtures records laya.lang's constants into the thresholds block, so merging NandhaKishorM#1334 needed a Java fixture regeneration that did not happen. Regenerated rather than hand-edited, on the toolchain the fixtures were recorded with: CPython 3.12.14 with unicodedata 15.0.0, torch 2.14.0, transformers 5.17.0, tokenizers 0.23.2, numpy 2.5.3. The result is one line and reproduces CI's own blob hashes, eb177d8 to d01133a. Nothing else drifted. gen_fixtures reports 20 families matching with 9 unverified for want of a checkpoint, gen_unicode_tables matches CPython 15.0.0, gen_language_tables matches laya/lang.py, and the three Go tables_gen.go files match. A job-level sweep of every run at the release commit finds this one failure and no other. The key sits in thresholds beside _SHOUTED_MIN_STOPWORD and _SHOUTED_MIN_WORD, two internals already recorded there, and LanguageTablesTest reads the five public thresholds by name rather than asserting the key set, so the added key is inert on the Java side. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM
added a commit
that referenced
this pull request
Oct 28, 2026
#1392 added the Vietnamese diacritics to laya/lang.py's _NON_EN_DIACRITICS so Vietnamese stops routing to the English checkpoint, but the generated copies were not re-recorded, so laya-go/lang/tables_gen.go and laya-cpp/src/lang_tables.inc both went stale and their --check gates would have failed on main. Regenerated with laya-go/scripts/gen_tables.py on CPython 3.12 / unicodedata 15.0.0, the pinned generator toolchain. The delta is only U+01A1, U+01B0 and U+1EA1..U+1EF9; the other three generated files re-recorded byte-identically. This is the 0.4.15 failure caught before the tag rather than after: there I merged #1334 without regenerating the Java fixtures and shipped with a red job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
_script_countsdecides "is this a letter" through a new_is_letter, which keepsstr.isalpha()for anything the running interpreter knows and otherwise consults a generated table of letters assigned in Unicode 16.0 but not in 14.0 (plus the five scripts Unicode 14 introduced, so Python 3.10 matches too). The table is only consulted for characters this interpreter reports asCn.Why
str.isalpha()andunicodedata.categoryanswer from the Unicode database bundled with the interpreter, and supported versions disagree (3.10 -> Unicode 13, 3.11 -> 14, 3.12 -> 15.0, 3.13 -> 15.1). A letter added since then isCn, i.e. unassigned, soisalpha()is False and it was dropped before the "other" branch could count it -- the branch whose own comment exists to stop unreadable text reaching the English checkpoint. Kawi, Nag Mundari, Todhri, Vithkuqi, Old Uyghur, CJK extension I, ... then reportedscript="unknown"("no letters detected"), whichRouter._routereads as "use the default checkpoint". WithRouter(default="english")-- the documented setting for a mostly-English deployment -- that is a misroute to the checkpoint that collapses off English, andlaya.is_english()/analyse()gave different answers for the same input on different supported Pythons.How it was verified
Checklist
main(branch cut fromupstream/mainat 0.4.14)