Repository navigation
fix(lang): keep Vietnamese off the English checkpoint - #1392
Conversation
Vietnamese holds no stopword list of its own, so the non-English diacritic set is the only signal that routes it to the multilingual checkpoint -- but every tone-marked Vietnamese letter is a precomposed codepoint the set did not hold, and the set already claimed Vietnamese coverage through 'đ' alone. Everyday sentences such as 'Cảm ơn bạn nhiều nhé' carried a diacritic rate of 0.0, stayed language-undecided and were routed to the English checkpoint. Adds the tone-marked lowercase Vietnamese letters (and ơ/ư) to the Python set and to the laya-ts port, with routing tests and English loanword/name controls on both sides. Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
#1392 added the Vietnamese diacritics to laya/lang.py's _NON_EN_DIACRITICS so Vietnamese stops routing to the English checkpoint, but the generated copies were not re-recorded, so laya-go/lang/tables_gen.go and laya-cpp/src/lang_tables.inc both went stale and their --check gates would have failed on main. Regenerated with laya-go/scripts/gen_tables.py on CPython 3.12 / unicodedata 15.0.0, the pinned generator toolchain. The delta is only U+01A1, U+01B0 and U+1EA1..U+1EF9; the other three generated files re-recorded byte-identically. This is the 0.4.15 failure caught before the tag rather than after: there I merged #1334 without regenerating the Java fixtures and shipped with a red job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same root cause as the Go and C++ regeneration one commit back: #1392 added the Vietnamese diacritics to laya/lang.py, and every generated copy downstream of that file went stale. Here that is LanguageTables.java, lang_tables.json and lang_detect.json, all three of which have blocking --check gates in the Java lane. Regenerated on CPython 3.12 / unicodedata 15.0.0. lang_tables.json gains the Vietnamese characters; lang_detect.json moves two diacritic_rate values from 0.2 to 0.6, which is the behaviour #1392 is for. All three gates now report clean: 20 families match laya (9 unverified, the checkpoint-backed ones), UnicodeTables matches CPython 15.0.0, and LanguageTables matches laya/lang.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window attribution preserved through end hooks (#1369), RemoteRouter identifying itself with a configurable User-Agent (#1396), ten more inputs refused where they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404, #1405, #1406), per-language temperatures and per-call budgets in .NET and Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle, SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation (#1399, #1407), and parity-lane sharding that replaces the budget headroom added for 0.4.16 (#1373, #1375). Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean. Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0: Java 20 families match laya (9 unverified), UnicodeTables matches CPython 15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match, both laya-cpp .inc files match. Version agrees across pyproject.toml, laya.__version__ and nix/package.nix. README carries 0.4.17 only. Security audit of the merge diff, 362766 added lines over 480 files: zero obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm install hooks unchanged, and no third-party host in any added line (example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12 dynamic-exec hits are the C++ test harness invoking its own checker through sys.executable, the Go sample generator reading laya's own example by AST at development time, and one comment in rust.yml. Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)] but pub Rust hook that would ship as public API able to replace the forward pass), #1386 (route_batch switched from self.route to self._route, silently bypassing subclass overrides the docstring promises), and nine stacked port PRs that conflict with each other and need linearizing (#1377, #1378, #1380, #1382, #1383, #1385, #1387, #1388, #1389, #1390). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thanks for merging #1392 (fix(lang): keep Vietnamese off the English checkpoint)! 🎉 By the way, I'm currently open to new opportunities (Full Stack (PHP, NodeJs, Python) / Data Science / Applied AI / Devtools, based in India). If you or anyone around is hiring, I'd love to connect: |
…ports #1392 added the precomposed Vietnamese tone-marked letters to `_NON_EN_DIACRITICS` in laya/lang.py and updated the TypeScript port. The generated tables (Go, C++, Java) were regenerated in 0aab6d5 and 4aedb9e. Rust and .NET hold the set as a hand-written literal and were missed, so both stopped at `đ` and scored every Vietnamese tone mark as undiacritised. The Rust differential lane caught it: 75 language and 58 routing mismatches against Python, all of them a diacritic rate reported low (for example 0.1026 where Python now says 0.1795), which then flips `looks_non_english` and the routing decision behind it. The .NET lane caught the same thing through two curated probes, `is_english/vietnamese` and `p0411/vietnamese_combining`. All six ports now carry the same 114 code points, checked by extracting each literal and comparing it against the Python set rather than by eye. The routing goldens are regenerated on CPython 3.12 (unicodedata 15.0.0). The Rust differential corpus moves more than the four probe values because dump_differential_golden.py reads `_NON_EN_DIACRITICS` live to build its words and draws from one seeded RNG, so a larger set shifts every later draw. That is the documented point of reading it live: a port that misses a table change gets caught. The .NET generator does not read the table, so its corpus is frozen and only its curated probes moved. Also raise the Rust router cell's budget from 30 to 60 minutes. It selects two checkpoints, so on a cold export cache it pays for both before a single test runs, and it was CANCELLED 29m22s into the export step. Measured cold on that same run: english 23m58s, multilingual 8m50s, so about 33 minutes of export alone. A cache is only saved by a job that succeeds and the key includes laya/common.py, so the cold path recurs rather than happening once. Verified: cargo test --workspace, 392 passed, 0 failed, 0 skipped, and the differential test that failed in CI now reports lang 0, route 0, email_clean 0, email_state 0. Python 640 passed / 92 suites, ruff clean, every table and fixture drift gate clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged, and this one earned its keep: "Cảm ơn bạn nhiều nhé" scoring a diacritic rate of 0.0 and landing on the English checkpoint is exactly the kind of quiet misroute that never shows up as an error. Adding the precomposed tone-marked letters is the right fix, and I like that you left the comment above One thing to carry forward, and it is the reason the lanes went red after the merge rather than before it. The set lives in seven places, and they do not all update the same way:
So after the merge the Rust lane reported 75 language and 58 routing mismatches against Python, and .NET failed Two details worth knowing, because they explain why the feedback arrived late: The generators must run on CPython 3.12. They bake in
If you touch a table in |
What
Adds the lowercase tone-marked Vietnamese letters (
ả ạ ấ ầ … ơ ư) to the non-English diacritic set inlaya/lang.py, and the same letters toNON_EN_DIACRITICSin thelaya-tsport. Routing tests with English controls added totests/test_router.py; mirrored expectations (generated with Python'slaya.lang.analyse) added tolaya-ts/tests/lang.test.ts.Why
Vietnamese has no stopword list of its own, so the diacritic set is the only signal that keeps it off the English checkpoint — and the set's comment already claims Vietnamese coverage through
đalone. Every other Vietnamese letter is a precomposed tone-marked codepoint the set did not hold, so onmain:The language stays undecided, no non-English marker fires, and undecided-but-not-non-English is read as English — the exact class #35/#42/#172 fixed for Romanian/German/Portuguese. laya-multilingual serves 100+ languages and the README's router contract is "sends non-English text to laya-multilingual"; a fully-accented Vietnamese sentence violates it today.
Fixes #1391
How it was verified
laya.langis dependency-free, so it was loaded directly and exercised on both the un-fixed and fixed file (this host has no numpy/torch for the fullpython tests/test_router.py, which is left to CI):On un-fixed HEAD the first two Vietnamese sentences returned
is_english=True(diacritic_rate=0.0); with the fix all five readFalsewith rates 0.18–0.34. The accented-Nguyễncontrol stays English through the #337 loanword rescue (rate 0.0185, below the 0.02 floor). The TS change is a byte-for-byte mirror of the same letter list; its expectations inlaya-ts/tests/lang.test.tswere computed with Python'sanalyseon the same inputs, matching the file's stated convention. The addedtests/test_router.pyrows assert the same behaviour throughRouter.routeand are left to CI, as are the vitest runs.Checklist
mainlaya.langverification above is exact;python tests/test_router.pyneeds numpy/torch unavailable on this host and is left to CI