Skip to content

fix(lang): identify plain-ASCII German instead of labelling it English - #130

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
Rish-it:fix/lang-ascii-german-routing
Sep 23, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
Rish-it:fix/lang-ascii-german-routing

Conversation

@Rish-it

@Rish-it Rish-it commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Addresses #54: plain-ASCII German labelled English. Code-only, no retraining. Undecided text (path 1) is a separate follow-up, as agreed on #54. The earlier routing-reason commit is dropped: #203 already reports undecided text as not identified.

Thanks @gitsupportb for the report and @AlKor13 for checking it against #42; the utterances that still failed after #42 are regression tests here.

Root cause

latin_profile() needs the best non-English stopword list to beat English by two hits. de held 23 words, missing most function words of short requests, and in/was counted for English only. On main:

'trage diesen termin in meinen kalender ein'    de=1  en=1  -> 'en'  -> english
'wie lautet die temperatur in fulda in hessen'  de=1  en=2  -> 'en'  -> english
'schalte das licht im wohnzimmer aus'           de=1  en=0  -> None  -> english
'was ist die aktuelle zeit'                     de=2  en=1  -> 'en'  -> english

No umlauts for the diacritic rate to catch, and #178 covered the Romance lists, not German.

Change

laya/lang.py, data only: 38 frequent German function words that no other list claims (ich, wie, mir, bitte, meinen, im, heute, ...), picked from misrouted MASSIVE utterances. in and was are also listed for German, as nl already does with is. es and du stay out: Spanish and French claim them, and since #178 a word two lists share names neither language. Two tests pin that, and two English controls pin the other direction.

tests/test_lang_guess.py (#231) pinned "Mein Konto wurde zweimal belastet, bitte erstatten Sie", plain-ASCII German, to english; it now routes to multilingual.

Results

MASSIVE 1.1 test splits, 2,974 utterances per locale. Share routed to the multilingual checkpoint, accents kept / stripped:

main with es/du this PR
de-DE 0.493 / 0.247 0.763 / 0.651 0.753 / 0.636
es-ES 0.711 / 0.488 0.692 / 0.447 0.711 / 0.488
fr-FR 0.822 / 0.622 0.816 / 0.606 0.822 / 0.622
en-US 0.000 / 0.000 0.000 / 0.000 0.000 / 0.000
  • Every locale other than de-DE routes exactly as on main, utterance by utterance, except 75 af-ZA states and 1 stripped vi-VN state that move to multilingual. None moves to English.
  • en-US: 0 of 16,521 utterances (all splits, accents kept and stripped) change label or route.

End to end, scenario as an 18-way choice with real weights (MPS): each utterance is scored on both checkpoints, then each routing's picks are replayed (matches Router.predict on 40/40 sampled items). Question format follows research/scripts/bench_local.py; the issue's wording was not published, so absolute numbers differ from its 0.330 / 0.535.

de-DE acc de-DE to English en-US acc en-US to English
main 0.395 1,508 0.623 2,974
this PR 0.415 736 0.623 2,974
forced english 0.370 0.623
forced multilingual 0.455 0.532

German +2.0 points, a third of the routing loss recovered; first 200 rows, the issue's slice: 0.365 -> 0.420. The 736 German states still sent to English: 352 undecided (the follow-up), 303 under four words (skipped by the heuristic on purpose), 81 still en (English song titles, will).

Reproduce the routing shares
# share.py -- python share.py 1.1/data/de-DE.jsonl
import json, sys, unicodedata
from laya.lang import analyse
strip = lambda t: "".join(c for c in unicodedata.normalize("NFKD", t) if not unicodedata.combining(c))
rows = [json.loads(line) for line in open(sys.argv[1], encoding="utf-8")]
utts = [r["utt"] for r in rows if r["partition"] == "test"]
for tag, us in [("kept", utts), ("stripped", [strip(u) for u in utts])]:
    print("accents %s: %.3f to multilingual" % (tag, sum(not analyse(u)["is_english"] for u in us) / len(us)))

Data: https://amazon-massive-nlu-dataset.s3.amazonaws.com/amazon-massive-dataset-1.1.tar.gz

Verification

  • tests/test_router.py: 399 passed, 0 failed (main: 386). The 8 German checks fail on main; the 2 es/du checks fail with those words back in the list.
  • English controls: "turn off smart lamp in den" (the en-US MASSIVE utterance nearest to flipping, de=2 against en=1) and "im so sorry, am an hour late, stuck in traffic". Both fail if in/was become German-only; the second fails if any one of am, an, so joins the German list.
  • Every suite in CI's test list, ruff (CI rule set) and compileall: two clean runs, then a rerun from the committed HEAD in a fresh checkout (tests, routes, both tables). Each commit passes on its own.
  • Not run: tests/test_local_e2e.py, which needs the typed-decisions checkpoint in a local layout; no model code changed.
  • macOS, Python 3.11, torch 2.14.

Commits

  1. fix(lang): German stopwords, with the German routing tests and the test_lang_guess update.
  2. test(lang): pin es and du out of the German list.
  3. test(lang): pin English that shares words with the German list.

Copilot AI lite review requested due to automatic review settings September 22, 2026 05:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The change is narrowly scoped to stopword data with targeted regression tests, and the only requested edits are minor comment wording clarifications.

Review effort: Lite
Findings: 2 Low severity

Open (2)
What changed in this PR

Improves the Latin-script stopword heuristic so plain-ASCII German is identified as de (instead of being mislabeled en) and therefore routed to the multilingual checkpoint, addressing the “German labeled English” portion of #54 without retraining.

Changes:

  • Expanded the German stopword set in laya/lang.py with high-frequency German function words and added in/was to prevent them being English-only evidence.
  • Added regression tests that pin the previously misrouted ASCII-German examples and an English control case.
File Description
laya/​lang.py Extends German stopwords to strengthen guess_latin_language() on plain-ASCII German.
tests/​test_router.py Adds regression tests for ASCII-German language guessing and routing behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread laya/lang.py
"meinem", "meiner", "diese", "dieser", "diesen", "dieses", "einen", "einem", "einer",
"wie", "wo", "wann", "welche", "im", "zum", "zur", "aus", "bei", "nach", "noch", "bitte",
"heute", "jetzt", "kann", "kannst", "habe", "gibt", "wird",
# shared with English on purpose: counted for English alone, they outvoted short German
Comment thread tests/test_router.py
Comment on lines +229 to +230
# No umlaut for the diacritic rate to catch, and `in`/`was` counted for English alone, so these were
# labelled English and handed to the checkpoint that cannot read them.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, this is a big win, and I measured it on real data. On MASSIVE German, the share routed to the multilingual checkpoint goes from 46.9% to 76.1% with accents and from 23.3% to 66.4% with accents stripped, with only one English text moving in 20,000.

One regression to fix first. Spanish drops from 68.1% to 66.1% (45.2% to 40.4% stripped), and French dips too. The cause is es and du in the new German words: they are also Spanish and French function words, so they become shared and stop counting as evidence for those languages. I tried the PR with just those two removed. German still gets nearly all the gain (74.6% / 64.2%), Spanish and French return exactly to main, and the routing tests pass. Could you drop es and du? Then I'll merge it.

@Rish-it

Rish-it commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

@NandhaKishorM On it

@Rish-it
Rish-it force-pushed the fix/lang-ascii-german-routing branch from 41c469f to f239521 Compare September 23, 2026 17:26
Short German requests without umlauts were labelled English and routed to
the English checkpoint (NandhaKishorM#54). The `de` stopword list held only 23 words and
missed most of the function words assistant traffic is made of (ich, wie,
mir, bitte, meinen, im, heute), while `in` and `was` counted for English
alone. German needs a two-hit margin over English, so a sentence like
"wie lautet die temperatur in fulda in hessen" scored de=1 against en=2 and
fell through to "en".

Add 38 high-frequency German function words that no other list claims, and
list `in` and `was` for German too so they no longer count as evidence for
English only.

test_lang_guess pinned "Mein Konto wurde zweimal belastet, bitte erstatten
Sie" to the English checkpoint; it now routes to multilingual.

MASSIVE 1.1 test split, share routed to the English checkpoint:
  de-DE 0.507 -> 0.247 (accents stripped: 0.753 -> 0.364)
  en-US 1.000 -> 1.000; 0 of 16,521 en-US utterances change label or route
  every other locale unchanged, except af-ZA (-0.013) and one stripped
  vi-VN state, which move to multilingual

German with no identified language and no umlauts still takes the router's
`default` (English unless configured); that case is a separate follow-up
agreed on NandhaKishorM#54.
`es` and `du` are German function words, but Spanish and French claim them
too, and a word two lists share names neither language. With them in the
German list, "que hora es en australia" and "baisse le volume du
haut-parleur" are left undecided and routed to the English checkpoint, and
the MASSIVE es-ES share routed to multilingual falls from 0.711 to 0.692
(0.488 to 0.447 with accents stripped; fr-FR 0.822 to 0.816).

Pin both so a later German addition cannot bring them back.
@Rish-it
Rish-it force-pushed the fix/lang-ascii-german-routing branch from f239521 to b22698c Compare September 23, 2026 17:30
@Rish-it

Rish-it commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

@NandhaKishorM Done: dropped es and du and rebased on main. On MASSIVE, es-ES and fr-FR now route exactly as on main, utterance by utterance, and de-DE goes from 0.493 to 0.753 to multilingual (0.247 to 0.636 stripped), with no en-US change. Two tests pin es/du out of the German list.

Two things the rebase changed:

Updated numbers are in the description.

`in` and `was` now count for German as well as English, and `im` and `den`
are English tokens too. Pin two English states that must keep routing to
English: "turn off smart lamp in den", the en-US MASSIVE utterance nearest
to flipping (de=2 against en=1, three needed), and a chat line with `im`
that flips if any one of `am`, `an` or `so` joins the German list. Making
`in`/`was` German-only flips both.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for dropping es and du. I re-ran the 20,000 English texts: only one moves, an English sentence about a German diplomat with a German name, so German gets the gain you measured without costing Spanish, French or English. Merging.

@NandhaKishorM
NandhaKishorM merged commit 1bee2d0 into NandhaKishorM:main Sep 23, 2026
@Rish-it

Rish-it commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Cross-checked #130 (at 5be6e11) against the 12 open PRs that touch laya/lang.py, laya/router.py, tests/test_router.py or tests/test_lang_guess.py:

with textual both suites together routing, PR + #130 vs the PR alone (MASSIVE test splits)
#270, #258, #207, #166, #138, #120 merge cleanly pass: test_router (399 to 424 checks), test_lang_guess, and each PR's own suites only de-DE moves, 0.493 -> 0.753 (0.247 -> 0.636 stripped); es-ES, fr-FR, en-US: 0 utterances move
#206, #204, #196, #127, #91 already conflict with main; #130 changes none of those conflict regions not run
#194 already conflicts with main in tests/test_router.py, a section added after the same line; #130's block sits in that region, so the resolution keeps one more section not run

None of the lang.py PRs (#207, #206, #194, #91) adds any of #130's 40 German words to another list, so none of them turns a German word into a shared one.

@Rish-it

Rish-it commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @NandhaKishorM! I'd love to pick up some more tasks from your Slack. Let me know! Feel free to assign me

baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
NandhaKishorM#130)

* fix(lang): identify plain-ASCII German instead of labelling it English

Short German requests without umlauts were labelled English and routed to
the English checkpoint (NandhaKishorM#54). The `de` stopword list held only 23 words and
missed most of the function words assistant traffic is made of (ich, wie,
mir, bitte, meinen, im, heute), while `in` and `was` counted for English
alone. German needs a two-hit margin over English, so a sentence like
"wie lautet die temperatur in fulda in hessen" scored de=1 against en=2 and
fell through to "en".

Add 38 high-frequency German function words that no other list claims, and
list `in` and `was` for German too so they no longer count as evidence for
English only.

test_lang_guess pinned "Mein Konto wurde zweimal belastet, bitte erstatten
Sie" to the English checkpoint; it now routes to multilingual.

MASSIVE 1.1 test split, share routed to the English checkpoint:
  de-DE 0.507 -> 0.247 (accents stripped: 0.753 -> 0.364)
  en-US 1.000 -> 1.000; 0 of 16,521 en-US utterances change label or route
  every other locale unchanged, except af-ZA (-0.013) and one stripped
  vi-VN state, which move to multilingual

German with no identified language and no umlauts still takes the router's
`default` (English unless configured); that case is a separate follow-up
agreed on NandhaKishorM#54.

* test(lang): keep es and du out of the German stopwords

`es` and `du` are German function words, but Spanish and French claim them
too, and a word two lists share names neither language. With them in the
German list, "que hora es en australia" and "baisse le volume du
haut-parleur" are left undecided and routed to the English checkpoint, and
the MASSIVE es-ES share routed to multilingual falls from 0.711 to 0.692
(0.488 to 0.447 with accents stripped; fr-FR 0.822 to 0.816).

Pin both so a later German addition cannot bring them back.

* test(lang): pin English that shares words with the German list

`in` and `was` now count for German as well as English, and `im` and `den`
are English tokens too. Pin two English states that must keep routing to
English: "turn off smart lamp in den", the en-US MASSIVE utterance nearest
to flipping (de=2 against en=1, three needed), and a chat line with `im`
that flips if any one of `am`, `an` or `so` joins the German list. Making
`in`/`was` German-only flips both.
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants