Skip to content

fix(lang): count letters the interpreter's Unicode database does not know - #1334

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
thread3d:fix/lang-newer-unicode-letters
Oct 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
thread3d:fix/lang-newer-unicode-letters

Conversation

@thread3d

Copy link
Copy Markdown
Contributor

What

_script_counts decides "is this a letter" through a new _is_letter, which keeps str.isalpha() for anything the running interpreter knows and otherwise consults a generated table of letters assigned in Unicode 16.0 but not in 14.0 (plus the five scripts Unicode 14 introduced, so Python 3.10 matches too). The table is only consulted for characters this interpreter reports as Cn.

Why

str.isalpha() and unicodedata.category answer from the Unicode database bundled with the interpreter, and supported versions disagree (3.10 -> Unicode 13, 3.11 -> 14, 3.12 -> 15.0, 3.13 -> 15.1). A letter added since then is Cn, i.e. unassigned, so isalpha() is False and it was dropped before the "other" branch could count it -- the branch whose own comment exists to stop unreadable text reaching the English checkpoint. Kawi, Nag Mundari, Todhri, Vithkuqi, Old Uyghur, CJK extension I, ... then reported script="unknown" ("no letters detected"), which Router._route reads as "use the default checkpoint". With Router(default="english") -- the documented setting for a mostly-English deployment -- that is a misroute to the checkpoint that collapses off English, and laya.is_english() / analyse() gave different answers for the same input on different supported Pythons.

How it was verified

python tests/test_lang_guess.py
python tests/test_router.py

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (branch cut from upstream/main at 0.4.14)
  • Tests pass locally
  • Docs or examples updated when the public API changed

…know

str.isalpha() and unicodedata.category answer from the Unicode database bundled with
the host interpreter, and the supported versions disagree: Python 3.10 ships Unicode
13.0, 3.11 14.0, 3.12 15.0, 3.13 15.1. A letter added after that database is category
Cn -- *unassigned* -- so isalpha() is False and _script_counts dropped it before its
"other" branch could count it, which that branch's own comment exists to prevent. Text
written in Kawi, Nag Mundari, Todhri, Vithkuqi, Old Uyghur, CJK extension I, ... then
reported script="unknown", i.e. "no letters detected", and Router._route reads that as
"use the default checkpoint" rather than as non-Latin text. With Router(default=
"english") - the documented setting for a mostly-English deployment - that is a
misroute to the checkpoint that collapses off English; and laya.is_english() /
analyse() gave different answers for the same input on different supported Pythons.

_is_letter() now consults a generated table of letters that Unicode 16.0 assigns and
Unicode 14.0 does not (plus the scripts Unicode 14 introduced, so Python 3.10 matches
too), gated on the character being Cn here: a character this interpreter already knows
is still decided by isalpha() alone.

Verified by comparing _script_counts for all 1 114 112 codepoints, old vs new, on three
interpreters: Unicode 16.0 -> 0 differences; Unicode 15.1 -> 4302, every one a codepoint
15.1 does not know; Unicode 14.0 -> 9272 of the same kind; and on each, zero codepoints
the interpreter knows changed answer.
@NandhaKishorM
NandhaKishorM merged commit e5b6dc2 into NandhaKishorM:main Oct 24, 2026
48 of 52 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 24, 2026
perf(lang): optimize unicode script detection with binary search

Add/add conflict against #1334, which put its _POST_DATABASE_LETTER_RANGES
table at the same point in the file. Both wanted and independent, so this
is a union: the flattened bisect tables sit next to the _SCRIPT_RANGES
they derive from, then #1334's letter table and _is_letter.

The two function bodies merged without conflict and compose correctly.
_script_counts now rejects via #1334's _is_letter, then bisects, and
#1349's if/else preserves what #1334's for/else did for a letter no range
claims, which still lands in counts["other"].

Verified equivalent rather than assumed. A bisect over flattened ranges
only matches a first-hit linear scan when no two ranges overlap, so both
were checked: 0 overlapping pairs, and a sweep of the entire code space,
all 1114112 points from U+0000 to U+10FFFF, finds 0 disagreements with the
old scan. _script_counts runs 0.125 ms on 1.5k mixed-script characters.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM added a commit that referenced this pull request Oct 24, 2026
33 PRs. Server hardening (#1331, #1332, #1333), TypeScript parity
(#1330, #1336, #1337, #1338, #1339, #1340, #1341, #1342, #1343),
fail-closed input handling in evals and the integrations (#1317, #1318,
#1319, #1320, #1321, #1322, #1326, #1335), exact and faster script
detection (#1334, #1349), JVM and .NET resource handling (#1327, #1328,
#1329, #1346), port goldens and clipping (#1324 excluded, #1325, #1344,
#1345), CI (#1323, #1347, #1348).

Verified before tagging: all 92 suites pass, 610 tests. ruff clean on
both lists. laya-ts 1247 tests across 44 files, tsc --noEmit clean.
Version agrees across pyproject.toml, laya.__version__ and nix/package.nix.
README carries 0.4.15 only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Centaurioun pushed a commit to Centaurioun/laya that referenced this pull request Oct 25, 2026
Clears the only job-level failure at 0.4.15. `jvm build + model-free
tests` has been red on main since the release, at the step "Model-free
fixture drift vs. committed", and the Java lane still reported success
because that job is continue-on-error.

My own miss. NandhaKishorM#1334 added _POST_DATABASE_LETTER_FLOOR to laya/lang.py and
gen_fixtures records laya.lang's constants into the thresholds block, so
merging NandhaKishorM#1334 needed a Java fixture regeneration that did not happen.

Regenerated rather than hand-edited, on the toolchain the fixtures were
recorded with: CPython 3.12.14 with unicodedata 15.0.0, torch 2.14.0,
transformers 5.17.0, tokenizers 0.23.2, numpy 2.5.3. The result is one
line and reproduces CI's own blob hashes, eb177d8 to d01133a.

Nothing else drifted. gen_fixtures reports 20 families matching with 9
unverified for want of a checkpoint, gen_unicode_tables matches CPython
15.0.0, gen_language_tables matches laya/lang.py, and the three Go
tables_gen.go files match. A job-level sweep of every run at the release
commit finds this one failure and no other.

The key sits in thresholds beside _SHOUTED_MIN_STOPWORD and
_SHOUTED_MIN_WORD, two internals already recorded there, and
LanguageTablesTest reads the five public thresholds by name rather than
asserting the key set, so the added key is inert on the Java side.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
#1392 added the Vietnamese diacritics to laya/lang.py's _NON_EN_DIACRITICS
so Vietnamese stops routing to the English checkpoint, but the generated
copies were not re-recorded, so laya-go/lang/tables_gen.go and
laya-cpp/src/lang_tables.inc both went stale and their --check gates
would have failed on main.

Regenerated with laya-go/scripts/gen_tables.py on CPython 3.12 /
unicodedata 15.0.0, the pinned generator toolchain. The delta is only
U+01A1, U+01B0 and U+1EA1..U+1EF9; the other three generated files
re-recorded byte-identically.

This is the 0.4.15 failure caught before the tag rather than after:
there I merged #1334 without regenerating the Java fixtures and shipped
with a red job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thread3d
thread3d deleted the fix/lang-newer-unicode-letters branch October 28, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants