Skip to content

fix(lang): keep Vietnamese off the English checkpoint - #1392

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
rahul05ranjan:fix/vietnamese-diacritics
Oct 28, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
rahul05ranjan:fix/vietnamese-diacritics

Conversation

@rahul05ranjan

Copy link
Copy Markdown
Contributor

What

Adds the lowercase tone-marked Vietnamese letters (ả ạ ấ ầ … ơ ư) to the non-English diacritic set in laya/lang.py, and the same letters to NON_EN_DIACRITICS in the laya-ts port. Routing tests with English controls added to tests/test_router.py; mirrored expectations (generated with Python's laya.lang.analyse) added to laya-ts/tests/lang.test.ts.

Why

Vietnamese has no stopword list of its own, so the diacritic set is the only signal that keeps it off the English checkpoint — and the set's comment already claims Vietnamese coverage through đ alone. Every other Vietnamese letter is a precomposed tone-marked codepoint the set did not hold, so on main:

>>> analyse("Cảm ơn bạn nhiều nhé")["is_english"]
True        # -> routes to the English checkpoint
>>> analyse("Cảm ơn bạn nhiều nhé")["diacritic_rate"]
0.0

The language stays undecided, no non-English marker fires, and undecided-but-not-non-English is read as English — the exact class #35/#42/#172 fixed for Romanian/German/Portuguese. laya-multilingual serves 100+ languages and the README's router contract is "sends non-English text to laya-multilingual"; a fully-accented Vietnamese sentence violates it today.

Fixes #1391

How it was verified

laya.lang is dependency-free, so it was loaded directly and exercised on both the un-fixed and fixed file (this host has no numpy/torch for the full python tests/test_router.py, which is left to CI):

import importlib.util
spec = importlib.util.spec_from_file_location("layalang", "laya/lang.py")   # fixed file
m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
# Vietnamese (exact codepoints, no shell mangling):
for t in ["Cảm ơn bạn nhiều nhé", "Cảm ơn bạn rất nhiều", "Mình muốn hỏi về đơn hàng đã đặt",
          "Cho tôi xin hoàn tiền cho đơn hàng này", "CẢM ƠN BẠN RẤT NHIỀU"]:
    assert m.analyse(t)["is_english"] is False      # all False on the fix, True for the first two on main
# English controls (must not move):
for t in ["Please refund the duplicate charge on invoice 4411 today.",
          "The pho was great and the banh mi was the best in town",
          "The ao dai show at the cafe was wonderful",
          "Thanks Nguyen for the meal we had last night",
          "Please help Nguyễn with the invoice for the last order",
          "Please send me the café menu today please",
          "Jose is the account manager and he will handle the refund"]:
    assert m.is_english(t) is True                  # all True

On un-fixed HEAD the first two Vietnamese sentences returned is_english=True (diacritic_rate=0.0); with the fix all five read False with rates 0.18–0.34. The accented-Nguyễn control stays English through the #337 loanword rescue (rate 0.0185, below the 0.02 floor). The TS change is a byte-for-byte mirror of the same letter list; its expectations in laya-ts/tests/lang.test.ts were computed with Python's analyse on the same inputs, matching the file's stated convention. The added tests/test_router.py rows assert the same behaviour through Router.route and are left to CI, as are the vitest runs.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally — the dependency-free laya.lang verification above is exact; python tests/test_router.py needs numpy/torch unavailable on this host and is left to CI
  • Docs or examples updated when the public API changed — no public API change; routing behaviour only

Vietnamese holds no stopword list of its own, so the non-English diacritic
set is the only signal that routes it to the multilingual checkpoint -- but
every tone-marked Vietnamese letter is a precomposed codepoint the set did
not hold, and the set already claimed Vietnamese coverage through 'đ' alone.
Everyday sentences such as 'Cảm ơn bạn nhiều nhé' carried a diacritic rate of
0.0, stayed language-undecided and were routed to the English checkpoint.
Adds the tone-marked lowercase Vietnamese letters (and ơ/ư) to the Python set
and to the laya-ts port, with routing tests and English loanword/name
controls on both sides.

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
@NandhaKishorM
NandhaKishorM merged commit f712b56 into NandhaKishorM:main Oct 28, 2026
48 of 52 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
#1392 added the Vietnamese diacritics to laya/lang.py's _NON_EN_DIACRITICS
so Vietnamese stops routing to the English checkpoint, but the generated
copies were not re-recorded, so laya-go/lang/tables_gen.go and
laya-cpp/src/lang_tables.inc both went stale and their --check gates
would have failed on main.

Regenerated with laya-go/scripts/gen_tables.py on CPython 3.12 /
unicodedata 15.0.0, the pinned generator toolchain. The delta is only
U+01A1, U+01B0 and U+1EA1..U+1EF9; the other three generated files
re-recorded byte-identically.

This is the 0.4.15 failure caught before the tag rather than after:
there I merged #1334 without regenerating the Java fixtures and shipped
with a red job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
Same root cause as the Go and C++ regeneration one commit back: #1392
added the Vietnamese diacritics to laya/lang.py, and every generated
copy downstream of that file went stale. Here that is LanguageTables.java,
lang_tables.json and lang_detect.json, all three of which have blocking
--check gates in the Java lane.

Regenerated on CPython 3.12 / unicodedata 15.0.0. lang_tables.json gains
the Vietnamese characters; lang_detect.json moves two diacritic_rate
values from 0.2 to 0.6, which is the behaviour #1392 is for. All three
gates now report clean: 20 families match laya (9 unverified, the
checkpoint-backed ones), UnicodeTables matches CPython 15.0.0, and
LanguageTables matches laya/lang.py.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window
attribution preserved through end hooks (#1369), RemoteRouter identifying
itself with a configurable User-Agent (#1396), ten more inputs refused where
they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404,
#1405, #1406), per-language temperatures and per-call budgets in .NET and
Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle,
SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation
(#1399, #1407), and parity-lane sharding that replaces the budget headroom
added for 0.4.16 (#1373, #1375).

Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff
clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean.
Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0:
Java 20 families match laya (9 unverified), UnicodeTables matches CPython
15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match,
both laya-cpp .inc files match. Version agrees across pyproject.toml,
laya.__version__ and nix/package.nix. README carries 0.4.17 only.

Security audit of the merge diff, 362766 added lines over 480 files: zero
obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or
symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm
install hooks unchanged, and no third-party host in any added line
(example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12
dynamic-exec hits are the C++ test harness invoking its own checker through
sys.executable, the Go sample generator reading laya's own example by AST at
development time, and one comment in rust.yml.

Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)]
but pub Rust hook that would ship as public API able to replace the forward
pass), #1386 (route_batch switched from self.route to self._route, silently
bypassing subclass overrides the docstring promises), and nine stacked port
PRs that conflict with each other and need linearizing (#1377, #1378, #1380,
#1382, #1383, #1385, #1387, #1388, #1389, #1390).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@rahul05ranjan

Copy link
Copy Markdown
Contributor Author

Thanks for merging #1392 (fix(lang): keep Vietnamese off the English checkpoint)! 🎉

By the way, I'm currently open to new opportunities (Full Stack (PHP, NodeJs, Python) / Data Science / Applied AI / Devtools, based in India). If you or anyone around is hiring, I'd love to connect:
LinkedIn: https://www.linkedin.com/in/rahulswt7/ · GitHub: https://github.com/rahul05ranjan

NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
…ports

#1392 added the precomposed Vietnamese tone-marked letters to
`_NON_EN_DIACRITICS` in laya/lang.py and updated the TypeScript port. The
generated tables (Go, C++, Java) were regenerated in 0aab6d5 and 4aedb9e.
Rust and .NET hold the set as a hand-written literal and were missed, so both
stopped at `đ` and scored every Vietnamese tone mark as undiacritised.

The Rust differential lane caught it: 75 language and 58 routing mismatches
against Python, all of them a diacritic rate reported low (for example 0.1026
where Python now says 0.1795), which then flips `looks_non_english` and the
routing decision behind it. The .NET lane caught the same thing through two
curated probes, `is_english/vietnamese` and `p0411/vietnamese_combining`.

All six ports now carry the same 114 code points, checked by extracting each
literal and comparing it against the Python set rather than by eye.

The routing goldens are regenerated on CPython 3.12 (unicodedata 15.0.0). The
Rust differential corpus moves more than the four probe values because
dump_differential_golden.py reads `_NON_EN_DIACRITICS` live to build its words
and draws from one seeded RNG, so a larger set shifts every later draw. That is
the documented point of reading it live: a port that misses a table change gets
caught. The .NET generator does not read the table, so its corpus is frozen and
only its curated probes moved.

Also raise the Rust router cell's budget from 30 to 60 minutes. It selects two
checkpoints, so on a cold export cache it pays for both before a single test
runs, and it was CANCELLED 29m22s into the export step. Measured cold on that
same run: english 23m58s, multilingual 8m50s, so about 33 minutes of export
alone. A cache is only saved by a job that succeeds and the key includes
laya/common.py, so the cold path recurs rather than happening once.

Verified: cargo test --workspace, 392 passed, 0 failed, 0 skipped, and the
differential test that failed in CI now reports lang 0, route 0, email_clean 0,
email_state 0. Python 640 passed / 92 suites, ruff clean, every table and
fixture drift gate clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged, and this one earned its keep: "Cảm ơn bạn nhiều nhé" scoring a diacritic rate of 0.0 and landing on the English checkpoint is exactly the kind of quiet misroute that never shows up as an error. Adding the precomposed tone-marked letters is the right fix, and I like that you left the comment above đ honest instead of deleting the claim it had already made.

One thing to carry forward, and it is the reason the lanes went red after the merge rather than before it. The set lives in seven places, and they do not all update the same way:

  • laya/lang.py is the source.
  • laya-go/lang/tables_gen.go, laya-cpp/src/lang_tables.inc and LanguageTables.java are generated. laya-go/scripts/gen_tables.py and laya-java/scripts/gen_language_tables.py rebuild them, and each has a --check mode wired into CI.
  • laya-ts/src/lang.ts, laya-rust/src/language_detection.rs and laya-dotnet/src/Laya/LanguageDetection.cs hold the set as a hand-written literal. You updated TypeScript. Rust and .NET were missed.

So after the merge the Rust lane reported 75 language and 58 routing mismatches against Python, and .NET failed is_english/vietnamese and p0411/vietnamese_combining. Both were the same thing: a diacritic rate reported low, for instance 0.1026 where Python now says 0.1795, which then flips looks_non_english and the routing decision sitting behind it. Fixed in dc0916c, so nothing is outstanding for you here.

Two details worth knowing, because they explain why the feedback arrived late:

The generators must run on CPython 3.12. They bake in unicodedata, and 3.12 carries Unicode 15.0.0 while 3.11 carries 14.0.0, so regenerating on 3.11 reports drift in files that were already correct and quietly writes the wrong tables.

laya-rust/tools/dump_differential_golden.py reads _NON_EN_DIACRITICS live to build its corpus and draws from a single seeded RNG. Growing the set from 50 code points to 114 shifts every later draw, so its golden moves far beyond the four values you would expect. That is deliberate, and its own comment says so: a port that missed a table change gets caught. The .NET generator does not read the table, so its corpus is frozen and only its curated probes moved. That asymmetry is why Rust caught the implementation gap and .NET only caught it through two hand-written cases.

If you touch a table in laya/lang.py again, the quickest check is the three --check gates plus a grep for the constant name across the ports. I have added the 114 code point cross-port comparison to my own pre-tag checklist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants