Repository navigation
Conversation
|
Thank you for the careful follow-up to #130. I ran it on the 20,000 English texts I use for every routing change: 281 of them (1.4%) move to the multilingual checkpoint, against 0 to 5 for the routing PRs merged so far. Most are short English commands and questions with no function word, for example "tell me my current savings account's interest rate", "play recently added music" and "how many days before my credit card arrives", and the English checkpoint is the better one for those. So I can't take the four-word rule as it is. A rule that also needs positive evidence of another language (a stopword hit or a non-English letter) would keep the gain for undecided non-English text without moving plain English. |
d3f8946 to
1d250c4
Compare
|
Thanks for running it on the 20,000. I reworked it the way you suggested: undecided four-word text now leaves English moved to multilingual, first version against now:
Your three examples stay English, and they are now pinned in the tests. End to end on the MASSIVE test splits:
The cost falls on languages with no word list here. Indonesian reaches 0.318 instead of 0.409, because nothing in plain-ASCII Indonesian tells it apart from English. Of the English that still moves, 17 of the 31 en-US states carry only |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The parallel TypeScript implementation and tests do not yet include this routing fix.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
This PR improves Latin-script routing by sending undecided text with non-English evidence to the multilingual checkpoint.
Changes:
- Adds word and diacritic evidence tracking.
- Updates routing reasons and regression tests.
- Documents the default-routing exception.
| File | Summary |
|---|---|
tests/test_router.py |
Adds routing and control coverage. |
README.md |
Documents the updated routing behavior. |
laya/router.py |
Adds evidence-specific routing reasons. |
laya/lang.py |
Detects additional non-English evidence. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ``` | ||
|
|
||
| Very short Latin-script text often carries nothing that identifies its language (`"Quero cancelar"`, `"Esqueci minha senha"`). Such text goes to `default`, which is `"english"` unless you change it. If most of your traffic is not English, set: | ||
| Very short Latin-script text often carries nothing that identifies its language (`"Quero cancelar"`, `"Esqueci minha senha"`). Such text goes to `default`, which is `"english"` unless you change it. From four words up, text that names no language but has no English function word and does have one of another language, or a non-English letter, goes to the multilingual checkpoint instead (`"verificar qualquer email da amazon"`). If most of your traffic is not English, set: |
1d250c4 to
4da353e
Compare
|
Thanks for reworking this so carefully; the positive-evidence version is much closer. I ran it on about 23,000 English texts from public test sets and 143 still move to multilingual, mostly short questions whose only 'foreign' word is a common one like 'do', 'todo', 'solo' or 'van' ('how do my health benefits work', 'las vegas weather today'). Could you require at least two distinct words from other languages' lists, or a non-English letter, before undecided text leaves the default? In my local run that brought English moves down to 5 while keeping a good part of the German, Spanish, Romanian and Welsh gain. It will mean relaxing two of your new pins; please report the per-locale shares with the English count, and rebase on #207, which has now landed. |
7c236b5 to
ba60706
Compare
Bruce-Yii
left a comment
There was a problem hiding this comment.
The rule is right. The rebase is the whole problem, and it is worth about two orders of magnitude, so it is worth being precise about.
Routing here is a pure function — roughly 30 µs per call, 20,000 classifications in well under a second, no weights involved. So the measurement you were asked for is not expensive, and I ran it rather than asking you to.
The stale base is costing you almost all of the English win
On the PR's current base, English text that happens to contain two or more of the eight words main itself now flags as English/foreign collisions routes to multilingual 99.945% of the time (19,989 / 20,000). Resolve the single laya/lang.py hunk the way the surrounding code now wants and the same corpus comes to 0.635% (127 / 20,000). That is a 157× difference caused entirely by the missing rebase.
It is visible on ordinary README text, on the two trees:
'- [Per-call hooks](#per-call-hooks)'
#286 head -> english=False
main -> english=True
Why, mechanically
main has since introduced _EN_COLLISION_WORDS = {"come", "son", "do", "care", "todo", "im", "per", "plus"} and uses it in the scoring at lang.py:418. Your branch's rule is the older _OTHER_WORDS check at :429:
other = diac > 0 or len(_OTHER_WORDS.intersection(words)) >= 2_OTHER_WORDS never contains those eight words, so the new-rule path cannot see them. Whichever side of that one hunk you take decides whether the rule fires on collision vocabulary or not — and --theirs silently reinstates the regression with no test signal. Whichever resolution you choose should be written down in the PR, because the two answers are not close.
The fix is to keep both sets: rebase onto 6d942c92081fbc139e736bbd9ac0023223c29b7f, resolve that hunk so both _OTHER_WORDS and _EN_COLLISION_WORDS are honoured, re-run the English count, and drop solo from the README sentence. The rule as specified, the precedence, and the interaction with the explicit lang / lang_guess / override options all check out; seven mutations of the new logic each turn a test red, so the tests bite. Nothing about the design needs to change.
One caveat on my numbers, since they are not the same measurement as yours. Your corpus is private, and I did not download MASSIVE/CLINC150/BANKING77/SST-2/AG-News, so I could not reproduce your absolute 1.4%. My synthetic English is deliberately English-only vocabulary, which structurally under-samples the collision-word channel — that is why my no-accent row is 0/40,000 while your corpus clearly contains real collision-word English. So treat my figures as bounding the two branches, not estimating your denominator. The one cheap thing worth running on your side is a count of your ~23,000 texts with ≥2 distinct _EN_COLLISION_WORDS members, zero English function words, and diacritic rate < 2% — that is the population at risk and it needs no GPU.
Also worth knowing: there is no CI run at any of this branch's three commits — the fork has no workflows, so pull_request runs nowhere. Neither green nor red, and nothing is attributable to this PR.
Two smaller notes: the branch is a long way behind main, and five open PRs (#91, #130, #196, #204, #206) have histories disjoint from main, which makes conflict analysis against them impossible right now. #206 is worth a look specifically — excluding URLs and e-mails when counting words edits the very words list your rule intersects.
|
Needs a rebase before I can act on it, and I have a measurement for you to aim at. The positive-evidence design is what I asked for and the description now says the right rule: from four words up, undecided Latin text leaves Here is the baseline to measure against, so we are reading the same number. On So English currently stays put entirely on this corpus, and any movement your rule causes shows up against zero rather than against noise. That is the number I will apply when you resubmit: how many of those 2,974 move, and what happens to accuracy on the ones that do. Two things worth knowing before you spend the time: This overlaps #600. #600 is the other attempt at path 1 of #54 and takes the opposite approach: route undecided plain-ASCII of four or more words to multilingual by default, with an English-lexicon exception. The two conflict in That makes your framing the stronger one, if the numbers hold. Requiring positive evidence means English text with no evidence of another language does not move at all, which is the property #600 has to buy back with a 9,890-word lexicon. If your version moves far fewer en-US rows than #600 does and picks up the same non-English wins, it is the better change and I would take it over #600. So: rebase onto current Thanks for two rounds of reworking this already. The move from "four words" to "four words plus evidence" to "four words plus two distinct words of evidence" is the right direction and each step came with numbers. |
496b720 to
50d0763
Compare
|
Rebased onto Measured the way you set it out: routing only, no checkpoint loaded, MASSIVE No en-US row moves, so there is no accuracy change to report there. Across all the English I have (MASSIVE en-US all splits, CLINC150, BANKING77, SST-2, AG News and 28,261 English support tickets, 61,830 texts), one moves: For what it buys, I scored both checkpoints on every moved row rather than the first 200. I used
Over all 50 non-English splits, the net is +0.08 intent / +0.15 scenario for this PR and +1.18 / +1.64 for #600. A second seed (29) gives the same picture: +0.07 / +0.14 against +1.19 / +1.67, with #600's en-US cost at −0.3 intent and 0.0 scenario. So the first half of your condition holds and the second does not. This PR moves no en-US rows, but it picks up about a tenth of #600's non-English gain. Most of #600's lead is in languages with no stopword list here: id, ms, jv, sw and tl gain 3 to 14 points each, and plain ASCII in them carries no evidence this rule can read. #600 also leads on nl, it and es. On these numbers I would not argue for this PR over #600. The choice is whether 0.3 to 0.6 en-US intent points is worth that gain, and that is yours to make. One part of this PR complements #600 rather than competing with it. #600 only routes plain ASCII, so it leaves alone the 522 rows this PR moves on a non-English letter too rare for the diacritic rate. On those rows the multilingual checkpoint is 11 points more accurate on scenario in both seeds (+59 and +56 correct), and level on intent within noise. If you take #600, I can send that rule alone on top of it. The description has the per-locale shares and both seeds for every split that moves. |
|
Pushed My last comment showed where #600's lead came from: id, ms, jv, sw and tl. Plain ASCII in those languages carried no evidence this rule could read. There are two changes on top of
I used the same protocol and harness as before, with both seeds. Nets are in points on the full split: This gets about two thirds of #600's non-English gain with no en-US cost. On intent, id gains 9.3 points, ms 9.9, jv 5.3, tl 5.7 and sw 3.6. No locale's route accuracy goes down, and no row moves to English. #600 still gains more: about 0.4 intent and 0.55 scenario points pooled over the 50 non-English locales. Against that, it costs 0.3 to 0.6 en-US intent points. There is one caveat on the English count. The four train splits that build The rule and the lists are also ported to |
|
Two follow-ups to my last comment. The head is now English cost on sets the word filter never saw. That comment had one caveat: The one assistant command that moves is More of #600's gain. Most of what #600 still caught was plain-ASCII text whose function words no list held, so
That is about 80% of #600's non-English gain with no en-US cost. Across 138,122 English texts, 27 move, against 2,629 for #600. No locale's route accuracy goes down, and no row moves to English. The description has the per-locale tables. |
…e to multilingual Latin-script text that names no language and has no non-English letters took the router's `default`, English unless configured (NandhaKishorM#54, path 1). From four words up latin_profile() scores the text, and a single English function word names it `en`, so text still undecided there has none. When it carries two different words from other languages' lists, or a non-English letter too rare for the diacritic rate, that is evidence it is not English: "streiche alle meine geplanten termine" (`alle`, `meine`). One such word is not evidence: English uses `do`, `las`, `solo` and `van` too ("how do my health benefits work"), and a word seen twice ("apaga la luz de la cocina") is still one word. latin_profile() returns the evidence as `other_evidence`, and analyse() reports such text as not English, so it routes to multilingual whatever `default` is. Without evidence nothing changes: English with no function word ("play recently added music"), text with one list word, and languages with no list here keep `default`. MASSIVE 1.1 scenario, test split, 2,974 utterances per locale, end to end: ro-RO 0.343 -> 0.359 cy-GB 0.114 -> 0.128 de-DE 0.415 -> 0.418 pt-PT 0.423 -> 0.424 es-ES 0.420 -> 0.420 nl-NL 0.396 -> 0.393 en-US 0.623 -> 0.623 English moved to multilingual: MASSIVE en-US 1 of 16,521 (all splits); CLINC150, BANKING77, AG News, SST-2 and a synthetic support-ticket set none. No utterance of any locale moves to English.
Undecided Latin text that goes to multilingual on other-language evidence has a diacritic rate under NON_EN_DIACRITIC_RATE, so the reason read "language not identified but 0% non-English letters", which says the opposite of why it moved. It now names the evidence: two function words of other languages, or a non-English letter.
The `default` paragraph now says that from four words up, text with no English function word but two different words from other languages' lists, or a non-English letter, goes to the multilingual checkpoint, and why one such word is not enough.
… another language Undecided Latin text of four or more words left the default only with two distinct list words or a non-English letter. Indonesian, Malay, Javanese, Swahili, Tagalog, Afrikaans and Welsh had no list, so their plain ASCII carried no evidence at all and went to the English checkpoint. - Add function-word lists for those seven languages. A word another list already holds is left out, so no other language's score changes, and so is a word in more than 1 in 2,000 English train documents. - One list word is now evidence on its own when it occurs in none of 212,592 English documents (train splits of CLINC150, BANKING77, SST-2 and AG News). Those 568 words are `_FOREIGN_WORDS` in laya/lang.py, generated by scripts/build_foreign_words.py (pinned revisions, --check), which also writes laya-ts/src/foreign-words.ts. lang.py stays self-contained, so research/evals/massive_route_comparison.py can still load it from any commit. A word English uses too (`do`, `van`, `per`) still needs a second one. - The route reason says "function words of other languages", since one can now be enough. MASSIVE en-US test: 0 of 2,974 rows move (main: 0). Across 61,830 English texts one moves, the same one as before (`play do re mi`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
latinProfile() returns otherEvidence (a non-English letter, two distinct list words, or one word from FOREIGN_WORDS), analyseText() treats undecided text with it as not English, and the router reason names the evidence, as in laya/lang.py and laya/router.py. The seven new lists are copied word for word. A Danish sentence pinned as undecided is now named `af` (`kan`, `en`). It was not English before and is not now; only the name changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ore plain-ASCII function words Most of what NandhaKishorM#600 still caught and this rule missed was plain-ASCII text whose function words no list held: Danish, Norwegian, Finnish and Slovenian had no list, and the Dutch, Romanian, Swahili, Malay, Indonesian, Javanese and Welsh lists lacked common pronouns, prepositions and question words (`mijn`, `cum`, `kuhusu`, `kepada`). - Add `da`, `nb`, `fi` and `sl` lists, and extend `nl`, `ro`, `id`, `ms`, `jv`, `sw` and `cy`. Same filters as before: no word another list holds, none in more than 1 in 2,000 English documents. `da` and `nb` also leave out every Swedish word, so Swedish keeps its name. Romanian keeps `pe` and `ca` out, as its list's comment asks. - Only function words were added, no MASSIVE-domain words such as "play" or "lamp". The words were chosen on MASSIVE validation, and test was read only for the final numbers. - Regenerated `_FOREIGN_WORDS`: 841 of 1,162 list words. MASSIVE test: en-US 0 of 2,974 rows move, no row moves to English, and checkpoint route accuracy goes from 86.55% to 88.54%. Across 61,830 English texts one moves (the same `play do re mi`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Same words, in the same order, as laya/lang.py. "vertel me wat er gebeurt op instagram" is now named `nl` rather than undecided, so the undecided pin moves to the two cases that stay undecided. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3610a87 to
c73908d
Compare
|
Closing this as superseded, and by two changes rather than one. The foreign-word evidence idea landed through #460 (
Worth saying what did not get taken up, in case you want to return to it: the detector still reports only |

Follow-up to #130 for path 1 of #54. From four words up, undecided Latin text goes to the multilingual checkpoint only with positive evidence of another language. That evidence is a non-English letter, two distinct words from other languages' lists, or one list word that English text never uses. Anything else still follows
default. Rebased on currentmain(fa9a2a7, 0.3.24, after #728).This is the fourth version after review:
50d0763) moved no en-US row, but bought about a tenth of fix(lang): route undecided plain-ASCII text of 4+ words to multilingual #600's non-English gain.This version keeps the en-US count at zero and gets about 80% of #600's non-English gain.
Change
laya/lang.py:latin_profile()returnsother_evidence.analyse()reports undecided text with that evidence as not English, so it routes to multilingual whateverdefaultis.mijn,cum,cand). Plain ASCII in these languages left no evidence to read, which is where most of fix(lang): route undecided plain-ASCII text of 4+ words to multilingual #600's lead came from. Only function words are added, no MASSIVE-domain words such as "play" or "lamp". Three kinds of word are left out:peandcaout, as its list's comment asks._FOREIGN_WORDS: the 841 list words found in none of 212,592 English documents (the train splits of CLINC150, BANKING77, SST-2 and AG News). One of them is evidence on its own. Any other list word needs a second, different one (do,van,per).scripts/build_foreign_words.py: rebuilds_FOREIGN_WORDSandlaya-ts/src/foreign-words.tsfrom the lists and those four corpora at pinned revisions.--checkexits 1 when either file is stale.lang.py, not in a module of its own, so thatresearch/evals/massive_route_comparison.pycan still loadlang.pyfrom any commit.laya/router.py: that case gets its own reason (… but function words of other languages, or… a non-English letter) instead of0% non-English letters.laya-ts: the same rule, lists (word for word, same order) and reason inlang.tsandrouter.ts.README.md: thedefaultparagraph describes the rule.English cost
Routing only,
Router().route(text, ...)["model"], no checkpoint loaded:SetFit/amazon_massive_intent_en-US)play do re mi)en(Tobi-Bueck/customer-support-tickets)The one row that moves is the same one
50d0763moved. No utterance moves to English in any of the 51 MASSIVE test locales.The four train splits that build
_FOREIGN_WORDScome from the same corpora as four of these test sets, so I also ran ten English test sets that share nothing with them, against #600 on the samemain:en, HWU64, SNIPS, ATIS, TREC (assistant commands and questions)enThe one assistant command that moves is
List Sulek Mina mum upcoming birthdays: Finnishminais also a name. About half of the 25 tweets and comments are not English: Latin, Polish, German, French, Romanian, Javanese, Afrikaans and Tagalog.Per-locale shares
Share of the MASSIVE test split (
mteb/amazon_massive_intent@940fd47, 2,974 per locale) routed to the multilingual checkpoint. Every locale that moves:en-US and the other 22 locales do not move.
research/evals/massive_route_comparison.py --before-ref fa9a2a7 --after-ref 3610a87reports:German support tickets (
de, 33,504): 101 more reach multilingual.End to end
Both checkpoints were scored on every moved row, 16,843 in all, with no first-200 sample.
build_suiteinresearch/eval/laya_eval.py: intent with 20 options, scenario with all 18 MASSIVE scenarios, fp32 on MPS.Locales with 50 or more moved rows:
Pooled over the 50 non-English locales, the net is +0.99 / +0.92 points on intent and +1.32 / +1.32 on scenario. French is the one locale that loses: its moved rows already scored well on the English checkpoint. That was true of
50d0763and of #600 as well.Against #600
Same
main, same harness, both seeds:50d0763Verification
tests/test_router.py: 742 passed, 0 failed (main: 703).dim,kung), a check that every word of_FOREIGN_WORDSbelongs to a list, and one thatdaandnbhold no Swedish word.laya-ts: 628 passed (main: 597),tsc --noEmitclean.tests/undecided-evidence.test.tsruns the same cases.scripts/test_suites.py(all 77 CI suites, with themcp/serve/langchainextras),ruff(CI rule set) andcompileallpass at3610a87.test_routerand the TS suite on its own.scripts/build_foreign_words.py --checkpasses at3610a87, and exits 1 when a word is removed from the set.Commits
fix(lang): the evidence rule inlatin_profile()andanalyse(), with its routing tests.fix(router): the route reason names the evidence.docs(readme): thedefaultparagraph.feat(lang): lists for seven plain-ASCII languages,_FOREIGN_WORDSand its generator, with tests.feat(ts): the rule inlaya-ts.docs(readme): single-word evidence.feat(lang): lists for Danish, Norwegian, Finnish and Slovenian, and more function words for the others.feat(ts): the same lists inlaya-ts.🤖 Generated with Claude Code