Repository navigation
fix(lang): identify plain-ASCII German instead of labelling it English - #130
Conversation
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The change is narrowly scoped to stopword data with targeted regression tests, and the only requested edits are minor comment wording clarifications.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
Improves the Latin-script stopword heuristic so plain-ASCII German is identified as de (instead of being mislabeled en) and therefore routed to the multilingual checkpoint, addressing the “German labeled English” portion of #54 without retraining.
Changes:
- Expanded the German stopword set in
laya/lang.pywith high-frequency German function words and addedin/wasto prevent them being English-only evidence. - Added regression tests that pin the previously misrouted ASCII-German examples and an English control case.
| File | Description |
|---|---|
laya/lang.py |
Extends German stopwords to strengthen guess_latin_language() on plain-ASCII German. |
tests/test_router.py |
Adds regression tests for ASCII-German language guessing and routing behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| "meinem", "meiner", "diese", "dieser", "diesen", "dieses", "einen", "einem", "einer", | ||
| "wie", "wo", "wann", "welche", "im", "zum", "zur", "aus", "bei", "nach", "noch", "bitte", | ||
| "heute", "jetzt", "kann", "kannst", "habe", "gibt", "wird", | ||
| # shared with English on purpose: counted for English alone, they outvoted short German |
| # No umlaut for the diacritic rate to catch, and `in`/`was` counted for English alone, so these were | ||
| # labelled English and handed to the checkpoint that cannot read them. |
|
Thank you, this is a big win, and I measured it on real data. On MASSIVE German, the share routed to the multilingual checkpoint goes from 46.9% to 76.1% with accents and from 23.3% to 66.4% with accents stripped, with only one English text moving in 20,000. One regression to fix first. Spanish drops from 68.1% to 66.1% (45.2% to 40.4% stripped), and French dips too. The cause is |
|
@NandhaKishorM On it |
41c469f to
f239521
Compare
Short German requests without umlauts were labelled English and routed to the English checkpoint (NandhaKishorM#54). The `de` stopword list held only 23 words and missed most of the function words assistant traffic is made of (ich, wie, mir, bitte, meinen, im, heute), while `in` and `was` counted for English alone. German needs a two-hit margin over English, so a sentence like "wie lautet die temperatur in fulda in hessen" scored de=1 against en=2 and fell through to "en". Add 38 high-frequency German function words that no other list claims, and list `in` and `was` for German too so they no longer count as evidence for English only. test_lang_guess pinned "Mein Konto wurde zweimal belastet, bitte erstatten Sie" to the English checkpoint; it now routes to multilingual. MASSIVE 1.1 test split, share routed to the English checkpoint: de-DE 0.507 -> 0.247 (accents stripped: 0.753 -> 0.364) en-US 1.000 -> 1.000; 0 of 16,521 en-US utterances change label or route every other locale unchanged, except af-ZA (-0.013) and one stripped vi-VN state, which move to multilingual German with no identified language and no umlauts still takes the router's `default` (English unless configured); that case is a separate follow-up agreed on NandhaKishorM#54.
`es` and `du` are German function words, but Spanish and French claim them too, and a word two lists share names neither language. With them in the German list, "que hora es en australia" and "baisse le volume du haut-parleur" are left undecided and routed to the English checkpoint, and the MASSIVE es-ES share routed to multilingual falls from 0.711 to 0.692 (0.488 to 0.447 with accents stripped; fr-FR 0.822 to 0.816). Pin both so a later German addition cannot bring them back.
f239521 to
b22698c
Compare
|
@NandhaKishorM Done: dropped Two things the rebase changed:
Updated numbers are in the description. |
`in` and `was` now count for German as well as English, and `im` and `den` are English tokens too. Pin two English states that must keep routing to English: "turn off smart lamp in den", the en-US MASSIVE utterance nearest to flipping (de=2 against en=1, three needed), and a chat line with `im` that flips if any one of `am`, `an` or `so` joins the German list. Making `in`/`was` German-only flips both.
|
Thank you for dropping |
|
Cross-checked #130 (at 5be6e11) against the 12 open PRs that touch
None of the |
|
Thanks @NandhaKishorM! I'd love to pick up some more tasks from your Slack. Let me know! Feel free to assign me |
NandhaKishorM#130) * fix(lang): identify plain-ASCII German instead of labelling it English Short German requests without umlauts were labelled English and routed to the English checkpoint (NandhaKishorM#54). The `de` stopword list held only 23 words and missed most of the function words assistant traffic is made of (ich, wie, mir, bitte, meinen, im, heute), while `in` and `was` counted for English alone. German needs a two-hit margin over English, so a sentence like "wie lautet die temperatur in fulda in hessen" scored de=1 against en=2 and fell through to "en". Add 38 high-frequency German function words that no other list claims, and list `in` and `was` for German too so they no longer count as evidence for English only. test_lang_guess pinned "Mein Konto wurde zweimal belastet, bitte erstatten Sie" to the English checkpoint; it now routes to multilingual. MASSIVE 1.1 test split, share routed to the English checkpoint: de-DE 0.507 -> 0.247 (accents stripped: 0.753 -> 0.364) en-US 1.000 -> 1.000; 0 of 16,521 en-US utterances change label or route every other locale unchanged, except af-ZA (-0.013) and one stripped vi-VN state, which move to multilingual German with no identified language and no umlauts still takes the router's `default` (English unless configured); that case is a separate follow-up agreed on NandhaKishorM#54. * test(lang): keep es and du out of the German stopwords `es` and `du` are German function words, but Spanish and French claim them too, and a word two lists share names neither language. With them in the German list, "que hora es en australia" and "baisse le volume du haut-parleur" are left undecided and routed to the English checkpoint, and the MASSIVE es-ES share routed to multilingual falls from 0.711 to 0.692 (0.488 to 0.447 with accents stripped; fr-FR 0.822 to 0.816). Pin both so a later German addition cannot bring them back. * test(lang): pin English that shares words with the German list `in` and `was` now count for German as well as English, and `im` and `den` are English tokens too. Pin two English states that must keep routing to English: "turn off smart lamp in den", the en-US MASSIVE utterance nearest to flipping (de=2 against en=1, three needed), and a chat line with `im` that flips if any one of `am`, `an` or `so` joins the German list. Making `in`/`was` German-only flips both.

Addresses #54: plain-ASCII German labelled English. Code-only, no retraining. Undecided text (path 1) is a separate follow-up, as agreed on #54. The earlier routing-reason commit is dropped: #203 already reports undecided text as not identified.
Thanks @gitsupportb for the report and @AlKor13 for checking it against #42; the utterances that still failed after #42 are regression tests here.
Root cause
latin_profile()needs the best non-English stopword list to beat English by two hits.deheld 23 words, missing most function words of short requests, andin/wascounted for English only. Onmain:No umlauts for the diacritic rate to catch, and #178 covered the Romance lists, not German.
Change
laya/lang.py, data only: 38 frequent German function words that no other list claims (ich,wie,mir,bitte,meinen,im,heute, ...), picked from misrouted MASSIVE utterances.inandwasare also listed for German, asnlalready does withis.esanddustay out: Spanish and French claim them, and since #178 a word two lists share names neither language. Two tests pin that, and two English controls pin the other direction.tests/test_lang_guess.py(#231) pinned "Mein Konto wurde zweimal belastet, bitte erstatten Sie", plain-ASCII German, toenglish; it now routes tomultilingual.Results
MASSIVE 1.1 test splits, 2,974 utterances per locale. Share routed to the multilingual checkpoint, accents kept / stripped:
es/dumain, utterance by utterance, except 75 af-ZA states and 1 stripped vi-VN state that move to multilingual. None moves to English.End to end,
scenarioas an 18-waychoicewith real weights (MPS): each utterance is scored on both checkpoints, then each routing's picks are replayed (matchesRouter.predicton 40/40 sampled items). Question format followsresearch/scripts/bench_local.py; the issue's wording was not published, so absolute numbers differ from its 0.330 / 0.535.englishmultilingualGerman +2.0 points, a third of the routing loss recovered; first 200 rows, the issue's slice: 0.365 -> 0.420. The 736 German states still sent to English: 352 undecided (the follow-up), 303 under four words (skipped by the heuristic on purpose), 81 still
en(English song titles,will).Reproduce the routing shares
Data: https://amazon-massive-nlu-dataset.s3.amazonaws.com/amazon-massive-dataset-1.1.tar.gz
Verification
tests/test_router.py: 399 passed, 0 failed (main: 386). The 8 German checks fail onmain; the 2es/duchecks fail with those words back in the list.in/wasbecome German-only; the second fails if any one ofam,an,sojoins the German list.ruff(CI rule set) andcompileall: two clean runs, then a rerun from the committed HEAD in a fresh checkout (tests, routes, both tables). Each commit passes on its own.tests/test_local_e2e.py, which needs the typed-decisions checkpoint in a local layout; no model code changed.Commits
fix(lang): German stopwords, with the German routing tests and thetest_lang_guessupdate.test(lang): pinesandduout of the German list.test(lang): pin English that shares words with the German list.