Skip to content

fix(lang): send undecided Latin text with evidence of another language to multilingual - #286

Closed
Rish-it wants to merge 9 commits into
NandhaKishorM:mainfrom
Rish-it:fix/lang-undecided-four-words
Closed

Rish-it wants to merge 9 commits into
NandhaKishorM:mainfrom
Rish-it:fix/lang-undecided-four-words

Conversation

@Rish-it

@Rish-it Rish-it commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #130 for path 1 of #54. From four words up, undecided Latin text goes to the multilingual checkpoint only with positive evidence of another language. That evidence is a non-English letter, two distinct words from other languages' lists, or one list word that English text never uses. Anything else still follows default. Rebased on current main (fa9a2a7, 0.3.24, after #728).

This is the fourth version after review:

This version keeps the en-US count at zero and gets about 80% of #600's non-English gain.

Change

  • laya/lang.py:
    • latin_profile() returns other_evidence. analyse() reports undecided text with that evidence as not English, so it routes to multilingual whatever default is.
    • New function-word lists for Indonesian, Malay, Javanese, Swahili, Tagalog, Afrikaans, Welsh, Danish, Norwegian, Finnish and Slovenian. The Dutch and Romanian lists gain common pronouns, prepositions and ASCII spellings (mijn, cum, cand). Plain ASCII in these languages left no evidence to read, which is where most of fix(lang): route undecided plain-ASCII text of 4+ words to multilingual #600's lead came from. Only function words are added, no MASSIVE-domain words such as "play" or "lamp". Three kinds of word are left out:
      • a word another list already holds, so no other language's score changes;
      • a word found in more than 1 in 2,000 English documents;
      • for Danish and Norwegian, any Swedish word, so Swedish keeps its name. Romanian keeps pe and ca out, as its list's comment asks.
    • _FOREIGN_WORDS: the 841 list words found in none of 212,592 English documents (the train splits of CLINC150, BANKING77, SST-2 and AG News). One of them is evidence on its own. Any other list word needs a second, different one (do, van, per).
  • scripts/build_foreign_words.py: rebuilds _FOREIGN_WORDS and laya-ts/src/foreign-words.ts from the lists and those four corpora at pinned revisions.
    • --check exits 1 when either file is stale.
    • A word added to a list is not in the set until the script runs again, so it starts out needing a second word, the safe side for English.
    • The set lives in lang.py, not in a module of its own, so that research/evals/massive_route_comparison.py can still load lang.py from any commit.
  • laya/router.py: that case gets its own reason (… but function words of other languages, or … a non-English letter) instead of 0% non-English letters.
  • laya-ts: the same rule, lists (word for word, same order) and reason in lang.ts and router.ts.
  • README.md: the default paragraph describes the rule.
'streiche alle meine geplanten termine'   de  `alle` + `meine`    main -> english   now -> multilingual
'batalkan alarm saya pukul tujuh pagi'    id  `saya`              main -> english   now -> multilingual
'tafadhali washa plagi mahiri'            sw  `tafadhali`         main -> english   now -> multilingual
'slett alarmen jeg nettopp satte'         nb  `jeg`               main -> english   now -> multilingual
'how do my health benefits work'          en  `do` only           main -> english   now -> english
'kung fu panda three'                     en  `kung` (tl) only    main -> english   now -> english
'verificar qualquer email da amazon'      pt  `da` only           main -> english   now -> english

English cost

Routing only, Router().route(text, ...)["model"], no checkpoint loaded:

corpus n moved to multilingual
MASSIVE en-US test (SetFit/amazon_massive_intent_en-US) 2,974 0 (main: 0)
MASSIVE en-US, all splits 16,521 1 (play do re mi)
CLINC150 test 5,500 0
BANKING77 test 3,076 0
SST-2 validation 872 0
AG News test 7,600 0
support tickets, en (Tobi-Bueck/customer-support-tickets) 28,261 0

The one row that moves is the same one 50d0763 moved. No utterance moves to English in any of the 51 MASSIVE test locales.

The four train splits that build _FOREIGN_WORDS come from the same corpora as four of these test sets, so I also ran ten English test sets that share nothing with them, against #600 on the same main:

independent English test sets n this PR #600
MTOP en, HWU64, SNIPS, ATIS, TREC (assistant commands and questions) 8,255 1 255
tweet_eval sentiment, tweet_sentiment_extraction, toxic_conversations_50k, emotion, amazon_counterfactual en 68,037 25 1,686

The one assistant command that moves is List Sulek Mina mum upcoming birthdays: Finnish mina is also a name. About half of the 25 tweets and comments are not English: Latin, Polish, German, French, Romanian, Javanese, Afrikaans and Tagalog.

Per-locale shares

Share of the MASSIVE test split (mteb/amazon_massive_intent @ 940fd47, 2,974 per locale) routed to the multilingual checkpoint. Every locale that moves:

locale main this PR utterances moved
ms 0.004 0.747 2,211
id 0.007 0.707 2,082
jv 0.010 0.667 1,954
tl 0.115 0.747 1,881
sw 0.137 0.679 1,614
cy 0.133 0.556 1,259
af 0.168 0.518 1,041
nl 0.283 0.603 950
ro 0.533 0.755 658
sl 0.574 0.789 640
nb 0.496 0.688 569
da 0.534 0.679 431
fi 0.668 0.767 294
es 0.711 0.805 279
pt 0.765 0.832 197
it 0.635 0.693 172
de 0.752 0.806 161
fr 0.822 0.867 134
pl 0.813 0.847 101
sv 0.870 0.889 57
is 0.909 0.924 43
tr 0.905 0.919 41
hu 0.936 0.943 21
sq 0.915 0.921 19
lv 0.934 0.940 18
vi 0.956 0.959 8
az 0.973 0.976 7
hi 0.994 0.994 1

en-US and the other 22 locales do not move.

research/evals/massive_route_comparison.py --before-ref fa9a2a7 --after-ref 3610a87 reports:

  • Overall checkpoint route accuracy goes from 77.43% to 88.53%, with no locale decreasing.
  • Named-language accuracy goes from 8.84% to 14.92%. sv loses 2 names, with no route change.

German support tickets (de, 33,504): 101 more reach multilingual.

End to end

Both checkpoints were scored on every moved row, 16,843 in all, with no first-200 sample.

  • Cases come from build_suite in research/eval/laya_eval.py: intent with 20 options, scenario with all 18 MASSIVE scenarios, fp32 on MPS.
  • Net = (multilingual − english correct) / 2,974, in points on the full split.
  • Accuracy is from seed 13; nets are seed 13 / seed 29.

Locales with 50 or more moved rows:

locale moved intent: english / multilingual ckpt intent net scenario: english / multilingual ckpt scenario net
ms 2,211 0.257 / 0.403 +10.89 / +12.41 0.239 / 0.377 +10.22 / +11.43
id 2,082 0.356 / 0.494 +9.68 / +10.36 0.304 / 0.450 +10.22 / +10.32
jv 1,954 0.180 / 0.267 +5.72 / +4.71 0.139 / 0.263 +8.10 / +7.97
tl 1,881 0.297 / 0.387 +5.68 / +3.33 0.304 / 0.401 +6.19 / +5.78
sw 1,614 0.099 / 0.196 +5.25 / +4.61 0.066 / 0.198 +7.16 / +6.96
cy 1,259 0.123 / 0.156 +1.41 / +1.41 0.079 / 0.198 +5.01 / +5.04
af 1,041 0.306 / 0.336 +1.04 / +1.55 0.265 / 0.376 +3.87 / +3.60
nl 950 0.408 / 0.467 +1.88 / +1.65 0.375 / 0.424 +1.58 / +1.38
ro 658 0.353 / 0.409 +1.24 / +1.04 0.302 / 0.409 +2.35 / +2.56
sl 640 0.184 / 0.289 +2.25 / +2.12 0.161 / 0.328 +3.60 / +3.46
nb 569 0.413 / 0.471 +1.11 / +0.84 0.285 / 0.397 +2.15 / +2.05
da 431 0.459 / 0.494 +0.50 / +0.07 0.355 / 0.515 +2.32 / +2.15
fi 294 0.241 / 0.330 +0.87 / +0.67 0.173 / 0.248 +0.74 / +0.91
es 279 0.484 / 0.534 +0.47 / +0.44 0.348 / 0.401 +0.50 / +0.20
pt 197 0.371 / 0.442 +0.47 / +0.37 0.406 / 0.416 +0.07 / +0.40
it 172 0.430 / 0.465 +0.20 / +0.24 0.407 / 0.390 −0.10 / +0.10
de 161 0.484 / 0.534 +0.27 / +0.24 0.398 / 0.540 +0.77 / +0.91
fr 134 0.597 / 0.478 −0.54 / −0.64 0.485 / 0.440 −0.20 / −0.27
pl 101 0.297 / 0.337 +0.13 / +0.20 0.287 / 0.356 +0.24 / +0.30
sv 57 0.439 / 0.491 +0.10 / +0.13 0.316 / 0.509 +0.37 / +0.40
en-US 0 – 0 – 0

Pooled over the 50 non-English locales, the net is +0.99 / +0.92 points on intent and +1.32 / +1.32 on scenario. French is the one locale that loses: its moved rows already scored well on the English checkpoint. That was true of 50d0763 and of #600 as well.

Against #600

Same main, same harness, both seeds:

main 50d0763 this PR #600
MASSIVE en-US test rows moved (of 2,974) 0 0 0 102
English texts moved (of 61,830) – 1 1 688
independent English texts moved (of 76,292) – – 26 1,941
en-US net, intent – 0 0 −0.57 / −0.34
en-US net, scenario – 0 0 −0.20 / −0.03
non-English net, intent – +0.08 / +0.07 +0.99 / +0.92 +1.18 / +1.19
non-English net, scenario – +0.15 / +0.14 +1.32 / +1.32 +1.64 / +1.67
MASSIVE checkpoint route accuracy 77.43% 79.22% 88.53% 91.77%
committed word set – – 841 words, generated 9,890 words, generated

Verification

  • tests/test_router.py: 742 passed, 0 failed (main: 703).
    • New checks: MASSIVE utterances for each new or extended list, two en-US controls whose only list word English uses too (dim, kung), a check that every word of _FOREIGN_WORDS belongs to a list, and one that da and nb hold no Swedish word.
    • Mutations: dropping the single-word rule turns 9 checks red, letting every list word count alone turns 6 red, and removing the new lists turns 2 red.
  • laya-ts: 628 passed (main: 597), tsc --noEmit clean. tests/undecided-evidence.test.ts runs the same cases.
  • scripts/test_suites.py (all 77 CI suites, with the mcp/serve/langchain extras), ruff (CI rule set) and compileall pass at 3610a87.
  • Each commit passes test_router and the TS suite on its own.
  • scripts/build_foreign_words.py --check passes at 3610a87, and exits 1 when a word is removed from the set.
  • Routes from the committed code match the measured run on all 120 corpora: MASSIVE validation and test, the English sets and both ticket sets. The lists were chosen on MASSIVE validation; test was read only for the numbers above.
  • The scoring harness reproduces @phant0um's published fix(lang): route undecided plain-ASCII text of 4+ words to multilingual #600 en-US numbers exactly (0.804 / 0.637 intent, 0.559 / 0.500 scenario).
  • macOS (M4 Pro), Python 3.11, torch 2.14, Node 26.

Commits

  1. fix(lang): the evidence rule in latin_profile() and analyse(), with its routing tests.
  2. fix(router): the route reason names the evidence.
  3. docs(readme): the default paragraph.
  4. feat(lang): lists for seven plain-ASCII languages, _FOREIGN_WORDS and its generator, with tests.
  5. feat(ts): the rule in laya-ts.
  6. docs(readme): single-word evidence.
  7. feat(lang): lists for Danish, Norwegian, Finnish and Slovenian, and more function words for the others.
  8. feat(ts): the same lists in laya-ts.

🤖 Generated with Claude Code

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for the careful follow-up to #130. I ran it on the 20,000 English texts I use for every routing change: 281 of them (1.4%) move to the multilingual checkpoint, against 0 to 5 for the routing PRs merged so far. Most are short English commands and questions with no function word, for example "tell me my current savings account's interest rate", "play recently added music" and "how many days before my credit card arrives", and the English checkpoint is the better one for those. So I can't take the four-word rule as it is. A rule that also needs positive evidence of another language (a stopword hit or a non-English letter) would keep the gain for undecided non-English text without moving plain English.

@Rish-it
Rish-it force-pushed the fix/lang-undecided-four-words branch from d3f8946 to 1d250c4 Compare September 24, 2026 05:30
Copilot AI lite review requested due to automatic review settings September 24, 2026 05:30
@Rish-it Rish-it changed the title fix(lang): send undecided Latin text of four or more words to multilingual fix(lang): send undecided Latin text with evidence of another language to multilingual Sep 24, 2026
@Rish-it

Rish-it commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for running it on the 20,000. I reworked it the way you suggested: undecided four-word text now leaves default only when it has a word from another language's list or a non-English letter. It's rebased on current main, and the description is updated.

English moved to multilingual, first version against now:

corpus n first version now
MASSIVE en-US, all splits 16,521 1,286 31
CLINC150 test 5,500 203 12
BANKING77 test 3,076 96 10
SST-2 validation 872 24 0
AG News test 7,600 4 2

Your three examples stay English, and they are now pinned in the tests.

End to end on the MASSIVE test splits:

  • de-DE goes from 0.415 to 0.433, keeping +1.8 of the +2.1.
  • pt-PT goes from 0.423 to 0.429.
  • en-US goes from 0.6234 to 0.6227; 8 of 2,974 utterances move.

The cost falls on languages with no word list here. Indonesian reaches 0.318 instead of 0.409, because nothing in plain-ASCII Indonesian tells it apart from English.

Of the English that still moves, 17 of the 31 en-US states carry only do, which is Portuguese. Leaving do out of the evidence cuts the 31 to 14, but do is also Polish and Slovenian. It would cost Polish 3.8 points of stripped share, so I left it in. I can take it out if your 20,000 say otherwise.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The parallel TypeScript implementation and tests do not yet include this routing fix.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Low severity

Open (1)
What changed in this PR

This PR improves Latin-script routing by sending undecided text with non-English evidence to the multilingual checkpoint.

Changes:

  • Adds word and diacritic evidence tracking.
  • Updates routing reasons and regression tests.
  • Documents the default-routing exception.
File Summary
tests/​test_router.py Adds routing and control coverage.
README.md Documents the updated routing behavior.
laya/​router.py Adds evidence-specific routing reasons.
laya/​lang.py Detects additional non-English evidence.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread README.md Outdated
```

Very short Latin-script text often carries nothing that identifies its language (`"Quero cancelar"`, `"Esqueci minha senha"`). Such text goes to `default`, which is `"english"` unless you change it. If most of your traffic is not English, set:
Very short Latin-script text often carries nothing that identifies its language (`"Quero cancelar"`, `"Esqueci minha senha"`). Such text goes to `default`, which is `"english"` unless you change it. From four words up, text that names no language but has no English function word and does have one of another language, or a non-English letter, goes to the multilingual checkpoint instead (`"verificar qualquer email da amazon"`). If most of your traffic is not English, set:
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks for reworking this so carefully; the positive-evidence version is much closer. I ran it on about 23,000 English texts from public test sets and 143 still move to multilingual, mostly short questions whose only 'foreign' word is a common one like 'do', 'todo', 'solo' or 'van' ('how do my health benefits work', 'las vegas weather today'). Could you require at least two distinct words from other languages' lists, or a non-English letter, before undecided text leaves the default? In my local run that brought English moves down to 5 while keeping a good part of the German, Spanish, Romanian and Welsh gain. It will mean relaxing two of your new pins; please report the per-locale shares with the English count, and rebase on #207, which has now landed.

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rule is right. The rebase is the whole problem, and it is worth about two orders of magnitude, so it is worth being precise about.

Routing here is a pure function — roughly 30 µs per call, 20,000 classifications in well under a second, no weights involved. So the measurement you were asked for is not expensive, and I ran it rather than asking you to.

The stale base is costing you almost all of the English win

On the PR's current base, English text that happens to contain two or more of the eight words main itself now flags as English/foreign collisions routes to multilingual 99.945% of the time (19,989 / 20,000). Resolve the single laya/lang.py hunk the way the surrounding code now wants and the same corpus comes to 0.635% (127 / 20,000). That is a 157× difference caused entirely by the missing rebase.

It is visible on ordinary README text, on the two trees:

'- [Per-call hooks](#per-call-hooks)'
    #286 head  -> english=False
    main       -> english=True

Why, mechanically

main has since introduced _EN_COLLISION_WORDS = {"come", "son", "do", "care", "todo", "im", "per", "plus"} and uses it in the scoring at lang.py:418. Your branch's rule is the older _OTHER_WORDS check at :429:

other = diac > 0 or len(_OTHER_WORDS.intersection(words)) >= 2

_OTHER_WORDS never contains those eight words, so the new-rule path cannot see them. Whichever side of that one hunk you take decides whether the rule fires on collision vocabulary or not — and --theirs silently reinstates the regression with no test signal. Whichever resolution you choose should be written down in the PR, because the two answers are not close.

The fix is to keep both sets: rebase onto 6d942c92081fbc139e736bbd9ac0023223c29b7f, resolve that hunk so both _OTHER_WORDS and _EN_COLLISION_WORDS are honoured, re-run the English count, and drop solo from the README sentence. The rule as specified, the precedence, and the interaction with the explicit lang / lang_guess / override options all check out; seven mutations of the new logic each turn a test red, so the tests bite. Nothing about the design needs to change.

One caveat on my numbers, since they are not the same measurement as yours. Your corpus is private, and I did not download MASSIVE/CLINC150/BANKING77/SST-2/AG-News, so I could not reproduce your absolute 1.4%. My synthetic English is deliberately English-only vocabulary, which structurally under-samples the collision-word channel — that is why my no-accent row is 0/40,000 while your corpus clearly contains real collision-word English. So treat my figures as bounding the two branches, not estimating your denominator. The one cheap thing worth running on your side is a count of your ~23,000 texts with ≥2 distinct _EN_COLLISION_WORDS members, zero English function words, and diacritic rate < 2% — that is the population at risk and it needs no GPU.

Also worth knowing: there is no CI run at any of this branch's three commits — the fork has no workflows, so pull_request runs nowhere. Neither green nor red, and nothing is attributable to this PR.

Two smaller notes: the branch is a long way behind main, and five open PRs (#91, #130, #196, #204, #206) have histories disjoint from main, which makes conflict analysis against them impossible right now. #206 is worth a look specifically — excluding URLs and e-mails when counting words edits the very words list your rule intersects.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Needs a rebase before I can act on it, and I have a measurement for you to aim at.

The positive-evidence design is what I asked for and the description now says the right rule: from four words up, undecided Latin text leaves default only with two distinct words from other languages' lists or a non-English letter. I wanted to measure the current head rather than discuss it, and I could not, because it no longer merges: laya/lang.py conflicts against main, which has moved 67 pull requests since your last push. #728 (Swedish detection, in 0.3.23) rewrote parts of the same function.

Here is the baseline to measure against, so we are reading the same number. On main at e525557, routing only, no checkpoint loaded, MASSIVE en-US test split via SetFit/amazon_massive_intent_en-US, 2,974 rows:

en-US -> english: 2974   multilingual: 0

So English currently stays put entirely on this corpus, and any movement your rule causes shows up against zero rather than against noise. That is the number I will apply when you resubmit: how many of those 2,974 move, and what happens to accuracy on the ones that do.

Two things worth knowing before you spend the time:

This overlaps #600. #600 is the other attempt at path 1 of #54 and takes the opposite approach: route undecided plain-ASCII of four or more words to multilingual by default, with an English-lexicon exception. The two conflict in README.md and laya/lang.py, so at most one of them lands. #600 currently carries a measured en-US cost of roughly 0.6 to 0.9 points on the moved utterances, which is why I am still holding it rather than merging it.

That makes your framing the stronger one, if the numbers hold. Requiring positive evidence means English text with no evidence of another language does not move at all, which is the property #600 has to buy back with a 9,890-word lexicon. If your version moves far fewer en-US rows than #600 does and picks up the same non-English wins, it is the better change and I would take it over #600.

So: rebase onto current main, report the en-US movement against the 2,974 above, and report what the rule buys on a few non-English locales from the same corpus. If the trade is better than #600's I will say so on both PRs.

Thanks for two rounds of reworking this already. The move from "four words" to "four words plus evidence" to "four words plus two distinct words of evidence" is the right direction and each step came with numbers.

@Rish-it

Rish-it commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto main at fa9a2a7; the head is 50d0763 and merges cleanly. The one conflict was laya/lang.py against #460 and #728, and I kept both sides: _EN_COLLISION_WORDS still drives the stopword scores, _OTHER_WORDS drives the evidence check, and the short-Swedish early return carries other_evidence: False. I also took solo out of the README and the code comment, since it is in no list (thanks @Bruce-Yii).

Measured the way you set it out: routing only, no checkpoint loaded, MASSIVE en-US test via SetFit/amazon_massive_intent_en-US, 2,974 rows.

main  fa9a2a7   en-US -> english: 2974   multilingual: 0
#286  50d0763   en-US -> english: 2974   multilingual: 0

No en-US row moves, so there is no accuracy change to report there. Across all the English I have (MASSIVE en-US all splits, CLINC150, BANKING77, SST-2, AG News and 28,261 English support tickets, 61,830 texts), one moves: play do re mi.

For what it buys, I scored both checkpoints on every moved row rather than the first 200. I used build_suite from research/eval/laya_eval.py (20-option intent, 18-option scenario, seed 13, fp32), and net is the change on the full split in points. Since #600 is the alternative, I measured it on the same main with the same harness. The harness reproduces @phant0um's en-US numbers for #600 exactly (0.804 / 0.637 intent).

split #286 moved intent scenario #600 moved intent scenario
en-US 0 0 0 102 −0.6 −0.2
de-DE 59 +0.2 +0.3 343 +0.3 +1.6
es-ES 213 +0.4 +0.1 518 +1.0 +0.8
fr-FR 110 −0.4 −0.2 342 −0.5 −0.1
it-IT 98 +0.1 −0.2 560 +0.7 +0.5
nl-NL 124 +0.3 −0.4 1127 +2.2 +2.7
pt-PT 103 +0.5 0 393 +0.4 +0.2
ro-RO 378 +0.5 +1.1 883 +1.4 +2.6
cy-GB 271 −0.1 +1.1 1482 +2.0 +5.4

Over all 50 non-English splits, the net is +0.08 intent / +0.15 scenario for this PR and +1.18 / +1.64 for #600. A second seed (29) gives the same picture: +0.07 / +0.14 against +1.19 / +1.67, with #600's en-US cost at −0.3 intent and 0.0 scenario.

So the first half of your condition holds and the second does not. This PR moves no en-US rows, but it picks up about a tenth of #600's non-English gain. Most of #600's lead is in languages with no stopword list here: id, ms, jv, sw and tl gain 3 to 14 points each, and plain ASCII in them carries no evidence this rule can read. #600 also leads on nl, it and es. On these numbers I would not argue for this PR over #600. The choice is whether 0.3 to 0.6 en-US intent points is worth that gain, and that is yours to make.

One part of this PR complements #600 rather than competing with it. #600 only routes plain ASCII, so it leaves alone the 522 rows this PR moves on a non-English letter too rare for the diacritic rate. On those rows the multilingual checkpoint is 11 points more accurate on scenario in both seeds (+59 and +56 correct), and level on intent within noise. If you take #600, I can send that rule alone on top of it.

The description has the per-locale shares and both seeds for every split that moves.

@Rish-it

Rish-it commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Pushed ab74437. It keeps en-US at zero moved rows and closes most of the gap to #600.

My last comment showed where #600's lead came from: id, ms, jv, sw and tl. Plain ASCII in those languages carried no evidence this rule could read. There are two changes on top of 50d0763:

  1. Function-word lists for Indonesian, Malay, Javanese, Swahili, Tagalog, Afrikaans and Welsh. A word another list already holds is left out, and so is one found in more than 1 in 2,000 English documents.
  2. One list word is evidence on its own when it occurs in none of 212,592 English documents (the train splits of CLINC150, BANKING77, SST-2 and AG News). That covers 568 of the 833 list words. They are committed as _FOREIGN_WORDS in lang.py, with scripts/build_foreign_words.py to rebuild them (pinned revisions, --check). The other 265 words, such as do, van and per, still need a second word.

I used the same protocol and harness as before, with both seeds. Nets are in points on the full split:

                                   main     50d0763         ab74437         #600
en-US test rows moved (of 2,974)   0        0               0               102
English texts moved (of 61,830)    -        1               1               688
en-US net, intent                  -        0               0               -0.57 / -0.34
non-English net, intent            -        +0.08 / +0.07   +0.80 / +0.80   +1.18 / +1.19
non-English net, scenario          -        +0.15 / +0.14   +1.09 / +1.10   +1.64 / +1.67
MASSIVE route accuracy             77.43%   79.22%          86.55%          91.77%

This gets about two thirds of #600's non-English gain with no en-US cost. On intent, id gains 9.3 points, ms 9.9, jv 5.3, tl 5.7 and sw 3.6. No locale's route accuracy goes down, and no row moves to English. #600 still gains more: about 0.4 intent and 0.55 scenario points pooled over the 50 non-English locales. Against that, it costs 0.3 to 0.6 en-US intent points.

There is one caveat on the English count. The four train splits that build _FOREIGN_WORDS come from the same corpora as four of the English test sets. MASSIVE en-US and the 28,261 support tickets share nothing with them, and 1 of their 44,782 texts moves.

The rule and the lists are also ported to laya-ts, with the same cases as tests. The description has the per-locale tables.

@Rish-it

Rish-it commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Two follow-ups to my last comment. The head is now 3610a87.

English cost on sets the word filter never saw. That comment had one caveat: _FOREIGN_WORDS is built from the train splits of four corpora whose test splits are also in my English count. So I ran ten English test sets that share nothing with them, against #600 on the same main:

                                                         n        this PR   #600
MTOP en, HWU64, SNIPS, ATIS, TREC                        8,255    1         255
tweets and comments (tweet_eval, tweet_sentiment_
  extraction, toxic_conversations_50k, emotion,
  amazon_counterfactual en)                              68,037   25        1,686

The one assistant command that moves is List Sulek Mina mum upcoming birthdays, because Finnish mina is also a name. About half of the 25 tweets are not English: Latin, Polish, German, French, Romanian, Javanese, Afrikaans and Tagalog.

More of #600's gain. Most of what #600 still caught was plain-ASCII text whose function words no list held, so 3610a87 adds lists for Danish, Norwegian, Finnish and Slovenian. It also extends the Dutch, Romanian, Swahili, Malay, Indonesian, Javanese and Welsh lists (mijn, cum, kuhusu, kepada).

  • Only function words were added, no MASSIVE-domain words such as "play" or "lamp".
  • The filters are the same as before. Danish and Norwegian also leave out every Swedish word, so Swedish keeps its name.
  • I chose the words on MASSIVE validation and read test only for the numbers.
                                   main     ab74437         3610a87         #600
en-US test rows moved (of 2,974)   0        0               0               102
English texts moved (of 61,830)    -        1               1               688
en-US net, intent                  -        0               0               -0.57 / -0.34
non-English net, intent            -        +0.80 / +0.80   +0.99 / +0.92   +1.18 / +1.19
non-English net, scenario          -        +1.09 / +1.10   +1.32 / +1.32   +1.64 / +1.67
MASSIVE route accuracy             77.43%   86.55%          88.53%          91.77%

That is about 80% of #600's non-English gain with no en-US cost. Across 138,122 English texts, 27 move, against 2,629 for #600. No locale's route accuracy goes down, and no row moves to English. The description has the per-locale tables.

…e to multilingual

Latin-script text that names no language and has no non-English letters
took the router's `default`, English unless configured (NandhaKishorM#54, path 1).
From four words up latin_profile() scores the text, and a single English
function word names it `en`, so text still undecided there has none.
When it carries two different words from other languages' lists, or a
non-English letter too rare for the diacritic rate, that is evidence it
is not English: "streiche alle meine geplanten termine" (`alle`,
`meine`).

One such word is not evidence: English uses `do`, `las`, `solo` and
`van` too ("how do my health benefits work"), and a word seen twice
("apaga la luz de la cocina") is still one word.

latin_profile() returns the evidence as `other_evidence`, and analyse()
reports such text as not English, so it routes to multilingual whatever
`default` is. Without evidence nothing changes: English with no function
word ("play recently added music"), text with one list word, and
languages with no list here keep `default`.

MASSIVE 1.1 scenario, test split, 2,974 utterances per locale, end to end:
  ro-RO 0.343 -> 0.359   cy-GB 0.114 -> 0.128   de-DE 0.415 -> 0.418
  pt-PT 0.423 -> 0.424   es-ES 0.420 -> 0.420   nl-NL 0.396 -> 0.393
  en-US 0.623 -> 0.623
English moved to multilingual: MASSIVE en-US 1 of 16,521 (all splits);
CLINC150, BANKING77, AG News, SST-2 and a synthetic support-ticket set
none. No utterance of any locale moves to English.
Undecided Latin text that goes to multilingual on other-language
evidence has a diacritic rate under NON_EN_DIACRITIC_RATE, so the
reason read "language not identified but 0% non-English letters",
which says the opposite of why it moved. It now names the evidence:
two function words of other languages, or a non-English letter.
Rish-it and others added 7 commits October 12, 2026 20:28
The `default` paragraph now says that from four words up, text with no
English function word but two different words from other languages'
lists, or a non-English letter, goes to the multilingual checkpoint,
and why one such word is not enough.
… another language

Undecided Latin text of four or more words left the default only with
two distinct list words or a non-English letter. Indonesian, Malay,
Javanese, Swahili, Tagalog, Afrikaans and Welsh had no list, so their
plain ASCII carried no evidence at all and went to the English
checkpoint.

- Add function-word lists for those seven languages. A word another list
  already holds is left out, so no other language's score changes, and so
  is a word in more than 1 in 2,000 English train documents.
- One list word is now evidence on its own when it occurs in none of
  212,592 English documents (train splits of CLINC150, BANKING77, SST-2
  and AG News). Those 568 words are `_FOREIGN_WORDS` in laya/lang.py,
  generated by scripts/build_foreign_words.py (pinned revisions,
  --check), which also writes laya-ts/src/foreign-words.ts. lang.py stays
  self-contained, so research/evals/massive_route_comparison.py can still
  load it from any commit. A word English uses too (`do`, `van`, `per`)
  still needs a second one.
- The route reason says "function words of other languages", since one
  can now be enough.

MASSIVE en-US test: 0 of 2,974 rows move (main: 0). Across 61,830
English texts one moves, the same one as before (`play do re mi`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
latinProfile() returns otherEvidence (a non-English letter, two distinct
list words, or one word from FOREIGN_WORDS), analyseText() treats
undecided text with it as not English, and the router reason names the
evidence, as in laya/lang.py and laya/router.py. The seven new lists are
copied word for word.

A Danish sentence pinned as undecided is now named `af` (`kan`, `en`).
It was not English before and is not now; only the name changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ore plain-ASCII function words

Most of what NandhaKishorM#600 still caught and this rule missed was plain-ASCII text
whose function words no list held: Danish, Norwegian, Finnish and
Slovenian had no list, and the Dutch, Romanian, Swahili, Malay,
Indonesian, Javanese and Welsh lists lacked common pronouns,
prepositions and question words (`mijn`, `cum`, `kuhusu`, `kepada`).

- Add `da`, `nb`, `fi` and `sl` lists, and extend `nl`, `ro`, `id`,
  `ms`, `jv`, `sw` and `cy`. Same filters as before: no word another
  list holds, none in more than 1 in 2,000 English documents. `da` and
  `nb` also leave out every Swedish word, so Swedish keeps its name.
  Romanian keeps `pe` and `ca` out, as its list's comment asks.
- Only function words were added, no MASSIVE-domain words such as
  "play" or "lamp". The words were chosen on MASSIVE validation, and
  test was read only for the final numbers.
- Regenerated `_FOREIGN_WORDS`: 841 of 1,162 list words.

MASSIVE test: en-US 0 of 2,974 rows move, no row moves to English, and
checkpoint route accuracy goes from 86.55% to 88.54%. Across 61,830
English texts one moves (the same `play do re mi`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Same words, in the same order, as laya/lang.py. "vertel me wat er
gebeurt op instagram" is now named `nl` rather than undecided, so the
undecided pin moves to the two cases that stay undecided.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Closing this as superseded, and by two changes rather than one.

The foreign-word evidence idea landed through #460 (6b8be28a, "fix/latin-distinct-evidence"), and then 3cf26cbc ("feat(router)!: default to the multilingual checkpoint for undecided text") made the question largely moot: every state whose language the detector cannot place now routes to multilingual by default, so Latin text with evidence of another language reaches the multilingual checkpoint whether or not the evidence is detected.

laya/lang.py has five commits since your branch point and the branch is 354 behind main, which is the shortest gap of the open language PRs but still past the point where the generated foreign-words.ts and laya/english_lexicon.py tables would rebase cleanly against what is there now.

Worth saying what did not get taken up, in case you want to return to it: the detector still reports only script for a short English imperative, so the distinction between "confidently English" and "undecided, defaulted to multilingual" is not visible to a caller reading analyse(). That is a real reporting gap and a smaller change than this PR. If that interests you, it would be worth its own issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants