Repository navigation
fix(lang): do not name all-caps acronym and address lines as foreign prose - #1013
Conversation
…prose
An all-caps segment is exempt from the acronym blanking in `_named_prose_language`, because a
customer shouting in Portuguese is still Portuguese. A line of bare acronyms has no lowercase
either, so it took the same exemption, kept its "words" and was named as prose. Three separate
holes, all now closed.
1. The evidence bar was on the line, not on the tokens that scored
Requiring one word of five letters anywhere is no evidence for the named language. `ANGELES`,
`VEGAS` and `SEGUNDO` are long; what named Spanish was `LOS`, `LAS` and `EL`. So the all-caps
US address block -- the commonest real instance of this shape -- walked straight through:
COM DOS LAN WAN VPN ROUTER -> 'pt'
SERVER LOG COM DOS LAN WAN VPN TIMEOUT -> 'pt'
STORES LOS ANGELES LAS VEGAS EL PASO CLOSED -> 'es'
The bar is now on the matched stopwords: an all-caps line must hold a matched stopword of
_SHOUTED_MIN_STOPWORD letters, or non-English diacritics, as well as one word of
_SHOUTED_MIN_WORD letters. All three return None.
2. The fix was in the wrong place to close the reported defect
`_named_prose_language` is reached only from the mixed-segment scan, and `analyse` runs that
scan only once the state already reads English overall. Below three English lines the acronym
line tipped `latin_profile` on the joined window and the scan never ran, so a unit assertion
on `_named_prose_language` passed while `Router.route` on the same string still said
multilingual:
"MON DES EST LA" -> multilingual
"Refund please.\n" + that line -> multilingual
"Refund please.\n"*2 + that line -> multilingual
"Refund please.\n"*3 + that line -> english
The root cause is that the whole-state verdict never applied the module's own rule that an
acronym is not a word: the acronyms voted in it. `_analyse_text` now re-takes the verdict with
the all-caps runs blanked for mixed-case text, and applies the shouted bar for all-caps text,
leaving the text undecided so `looks_non_english` still decides it. All four route english.
3. A vetoed line was dropped instead of falling through
In `_leaf_non_english` a named language that `_named_prose_language` disbelieved hit a bare
`continue`, which skipped the diacritic branch that exists for text carrying non-English letters
that no stopword list can name. A veto is not a verdict of English. It now falls through, and
`language_undecided` is no longer required there, because a language named and then disbelieved
is in the same evidential position as one never named. On the MASSIVE test split as written this
recovers 114 rows as a field -- `vertel my van my vergaderings van more oggend` and the like --
and newly drops 0.
`_shouted` also now requires an uppercase character, not merely the absence of a lowercase one.
A caseless script has no lowercase either, and `_WORD` splits at every combining mark, so the
tokens are short by construction and the bar would be unclearable. Nothing routes on that path
today (`_STOP["bn"]` is romanised), so this changes no behaviour; it keeps the bar on cased text.
Measurement
research/evals/massive_route_comparison.py, a4a8921 against this commit, pinned MASSIVE test
split (mteb/amazon_massive_intent @ 940fd47), 151,674 examples over 51 locales:
route accuracy English 1.000000 non-English 0.769765 overall 0.774279
named-language overall 0.088367 (English 0.801950, non-English 0.074095)
per-locale route and named deltas: 0.000000 at every one of the 51 locales
Unchanged, as before: not one of MASSIVE's 17,844 Latin-locale segments is all-caps, so the
split never reaches this branch. Three corpora that do were used to choose the rule -- 10,960
MASSIVE test lines upper-cased that the baseline names correctly (genuine shouted prose, must
survive), 500,000 lines of 4-8 acronyms drawn from 5,670 real ones, and 50,000 all-caps address
and signage lines built from 18,718 real US city names:
rule prose kept acronym FP address FP
none (before any guard) 10,960 100.00% 410 84
one word >= 5 anywhere (28fceac) 10,854 99.03% 212 83
matched stopword >= 3 10,625 96.94% 159 9
matched stopword >= 4 8,841 80.67% 14 2
matched stopword >= 4 AND >= 5 8,762 79.95% 6 2 <- chosen
matched stopword >= 5 6,902 62.97% 0 1
4 is the lowest bar that rejects LOS/LAS/EL; 3 leaves the reported case standing. 5 costs 1,860
more real lines to remove 6 more acronym runs in 500,000. The word bar, swept with the stopword
bar at 4: none 8,841 prose / 14 acronym, >= 5 8,762 / 6, >= 6 8,418 / 2, >= 7 7,459 / 0; 5 buys
8 of the 14 for 79 real lines, about 10:1, where 6 would cost 344 more for 4, about 86:1. The
diacritic disjunct is free on both false-positive corpora -- 6 and 2 either way -- and lifts
prose kept from 61.58% to 79.95%.
Cost, stated plainly: 2,198 of the 10,960 shouted-prose lines are given up, 82.62% of them six
tokens or longer, the longest a 34-token Dutch sentence. These are sentences, not fragments.
At routing level over the 77,324 non-English Latin MASSIVE lines, multilingual share:
plain, bare state 0.5625 -> 0.5625 plain, as a field 0.5049 -> 0.5056
upper, bare state 0.5557 -> 0.5072 upper, as a field 0.4971 -> 0.4594
So the whole cost falls on fully upper-cased input, which is the corpus by construction and a
rarity in real traffic; on MASSIVE as written routing is unchanged or slightly better.
No regression: tests/test_lang_guess.py 111 -> 146 passed, test_lang_stats.py 38,
test_blank_lang_routing.py 24, test_router.py 771, test_cli_lang_guess.py 17,
test_onnx_lang_parity.py 20 -- 0 failed, and every count outside test_lang_guess.py identical
before and after. No existing test encoded the old behaviour.
Mutants, each reverting one fix and nothing else: the shouted bar in `_named_prose_language`
fails 17 of the added assertions; the `_analyse_text` veto fails 3, all of them the bare and
diluted routing cases that no test covered before; the `_leaf_non_english` fall-through fails 3.
… veto Review of 9e8c196 found the mixed-case half of that guard unbarred, a second hole where a caseless script carries the state, and several figures in its own evidence that do not reproduce. Two behaviour fixes, one new committed corpus generator, and every number below re-measured on this commit. 1. Emphasis capitals deleted the evidence that named the language `_analyse_text` re-took the whole-state verdict with every all-caps run blanked. The all-caps branch beside it applies an evidence bar (`_shouted_evidence`, with a diacritic escape); this branch applied none, so when the shouted words were the ones that named the language the evidence was simply gone: sag mir das heutige datum multilingual 'de' -> SHOUT the last two: english quiero cancelar mi pedido por favor multilingual 'es' -> SHOUT the last two: english The diacritic escape does not catch these -- diacritic-free Spanish, Portuguese, Dutch and transliterated German all have `looks_non_english == False`. Monotonicity was broken too: the fully upper-cased form is held to the shouted bar and clears it, but prefixing one lowercase English token moved the text into this branch, which had no bar at all. Over the 77,324 rows of MASSIVE's 26 non-English Latin locales, 4,285 rows routed multilingual upper-cased and english with `ok ` in front of the same upper-case text, against 279 at a4a8921 and 316 here. A run is now blanked only while every all-caps run is short enough to be an acronym, by `_shouted_evidence`'s own first test: a run of `_SHOUTED_MIN_WORD` letters is a word, and a word is not blanked out of its own sentence. The claim in 9e8c196 that "the whole cost falls on fully upper-cased input ... a rarity in real traffic" was wrong: emphasis capitals in mixed-case prose are ordinary, and most of the cost was landing there. Two narrower and two wider rules were measured and rejected. Holding the caps runs to the full `_shouted_evidence` bar re-breaks both cases above (`HEUTIGE` and `DATUM` match no German stopword). Accepting the blanked verdict only when it names something reopens the reported defect at one line of English dilution -- `latin_profile("Refund please.")` is None -- and the existing test catches it, 145 passed / 1 failed. 2. A caseless script carrying the state went to the English checkpoint 'ПРИВЕТ MON DES EST LA' a4a8921 multilingual 'fr' -> 9e8c196 english 'Привет MON DES EST LA' a4a8921 multilingual 'fr' -> 9e8c196 english Both caps bars answer by taking evidence away from the Latin letters, and `looks_non_english`, which decides what is left, reads Latin diacritics only. A state that is 35% Cyrillic letters had nothing left once `MON DES EST LA` was disbelieved. Nothing else catches that fall either: the script promotion in `_analyse_text` needs `_non_latin_words`, which drops any run starting with a capital, so a shouted mixed-script line has no promotable word by construction. The first case is the all-caps branch and the second the mixed-case one, so both were affected. Neither bar now applies when the non-Latin letters clear the module's own dilution floor. The floor was already written twice in `_analyse_text`; it is now `_non_latin_carries`, called from both sites with no change to the existing one. A wrong language name costs nothing there, because the checkpoint is chosen on `is_english`. Measurement Routing over the MASSIVE test split @ 940fd47a, Router(preload=False), a4a8921 / 9e8c196 / this commit. 26 non-English Latin locales -- af az cy da de es fi fr hu id is it lv ms nb nl pl pt ro sl sq sv sw tl tr vi, every row of each test file, no filtering, n=77,324. Rows moved multilingual -> english (a regression) against a4a8921, and multilingual share: shape regressions 9e8c196 -> here share a4a8921 / 9e8c196 / here plain, bare 0 -> 0 0.5981 0.5981 0.5981 last two words SHOUTED 2,211 -> 447 0.5967 0.5681 0.5909 first half SHOUTED 4,786 -> 856 0.5968 0.5349 0.5858 'ok ' + fully upper-cased 7,744 -> 117 0.5882 0.4880 0.5866 fully upper-cased, bare 3,727 ->3,727 0.5912 0.5430 0.5430 Nothing improves on the two emphasis shapes and nothing else moves: 0 rows go english -> multilingual on any of these five shapes, and plain unmodified text is untouched at 0 route changes. On the eight locales the review used (de fr es it pt nl sv ro), the same two shapes are 1,753 -> 293 and 3,835 -> 752. The fully-upper row is unchanged by this commit, by design: that is the all-caps branch, which has had its bar since 9e8c196. Per-locale, the residual emphasis cost is concentrated where detection is strongest -- de 273, nl 100, fr 95, es 92 on the first-half shape -- and the residual rows are ones where every emphasised word is four letters or shorter, so length cannot tell them from an acronym run: `quelle heure EST IL`, `dime la fecha DE HOY`. 9e8c196 filed its sweep under "No regression". That sweep could not have shown one: not one of MASSIVE's 151,674 rows holds a single uppercase character, so `_shouted` is never true and the blanking is a no-op, and the harness passes a plain `str`, which `analyse` returns before `_leaf_non_english` is reached. All three fixes in that commit were invisible to it. The table above is over shapes that do reach the code. Two of them are synthetic -- MASSIVE has no shouting to find -- and that is stated rather than hidden. The two figures 9e8c196 quoted for fix 3 do not reproduce. As a field it recovers 41 rows, not 114, and 0 are newly dropped. The number depends entirely on the carrier, which was never stated; with the one its own new test uses, {"subject": "Please take a look at this ticket for the customer today", "note": <row>}, it is 41 over all 51 locales and over the 26 above, spread sl 8, vi 5, cy 4, sv 4, af 3, da 3, nb 3, tr 3, pt 2, de/es/fr/is/pl/ro 1 each. Share, as a field: 0.5381 -> 0.5386 -> 0.5386 on the 26 locales; upper-cased as a field, 0.5303 -> 0.4924 -> 0.4926. The four routing-share rows in 9e8c196 do not reproduce in absolute terms either -- it gives 0.5625 for plain bare where the row set above gives 0.5981, a consistent offset of about 0.035 across all four -- so its row set or filtering differed in a way it never described. The deltas it reported are right: -0.0485 against -0.0482 measured here for upper bare. The set and the filtering are stated above. research/evals/shouted_bar_corpora.py The acronym and address corpora behind `_SHOUTED_MIN_STOPWORD` and `_SHOUTED_MIN_WORD` were never committed -- no builder, no pinned list, no result JSON -- so every threshold decision in the file header was unfalsifiable and the trade ratios could not be rechecked. This script rebuilds all three corpora from pinned sources (MASSIVE @ 940fd47a, `random.Random(0)`, pools committed in the script) and sweeps the bar over them. It applies the bar from its own swept parameters rather than importing it, so the table stays meaningful after the constants move. It reproduces the prose column of the header table exactly, every row, including the two the review re-measured and found mislabelled: the rows for stopword >= 3 and >= 5 silently carried the >= 5 word bar as well, where the >= 4 row did not, so the six rows were not comparable. The table now applies one named bar per row, and has corrected values: stopword >= 3 alone keeps 10,729 (97.89%), not 10,625 (96.94%), and >= 5 alone keeps 6,963 (63.53%), not 6,902 (62.97%). Its false-positive columns are its own, and they change one conclusion. The new pools are smaller than the uncommitted originals (581 acronyms, 311 place names against 5,670 and 18,718), and on them a stopword bar of 4 already closes both columns to 0. So the 79 real prose lines that `_SHOUTED_MIN_WORD = 5` costs on top of it buy nothing these corpora can see, and the "10:1" and "86:1" ratios 9e8c196 quoted for that bar are not reproducible. The constant is kept, because a smaller corpus failing to find a false positive is not evidence there is none, and the header now says exactly that and says what a reader would have to do to move it. Also corrected in the header: `WIE SPAET IST ES IN KOELN` was given as an example of text kept "with its umlauts". It has none. The umlaut spelling `WIE SPÄT IST ES IN KÖLN` is the one the diacritic branch keeps; the ASCII spelling has no long matched German stopword (`wie`, three letters) and is vetoed -- genuine German on the English checkpoint, one of the 3,727 upper-cased casualties and not an example of a fix. Neither gate added here recovers it, and widening the stopword bar to recover it would reopen the acronym runs it exists to reject. And "4 is the lowest bar that rejects LOS/LAS/EL" was near-tautological at three, three and two letters; what the table decides is that 3 is ruled out, which is now what it says. Tests tests/test_lang_guess.py 146 -> 176 passed, add-only (68 lines added, 0 removed). No test covered the mixed-case branch for genuine foreign prose: the two existing MIXED_CASE_STILL_NONE cases assert `_named_prose_language` only and pass with the guard reverted. Added: ten emphasis-capitalised foreign lines, eight of them real MASSIVE test rows shouted where a person would shout them and two hand-written, asserted on both route and named language; the monotonicity pair; three acronym lines that must stay english, including `ok MON DES EST LA`; and the two mixed-script states, on route and on `is_english`. test_lang_stats.py 38, test_blank_lang_routing.py 24, test_router.py 771, test_cli_lang_guess.py 17, test_onnx_lang_parity.py 20 -- 0 failed, identical at a4a8921, 9e8c196 and here (test_lang_guess.py: 111, 146, 176). Mutants, each reverting one fix in a scratch copy and nothing else, from a green baseline, all killed by assertions and none by an error: blank every all-caps run again 21 of test_lang_guess fail apply the caps bars to caseless-carried states 4 drop the shouted bar in `_named_prose_language` 17 (17 before) drop the whole-state caps veto 6 ( 3 before) drop the `_leaf_non_english` fall-through 3 ( 3 before) None of the three mutants from 9e8c196 survives. The whole-state veto mutant now fails 6 rather than 3, because the new emphasis tests cover it from the acronym side as well. Cost, re-derived on this commit: 2,198 of the 10,960 shouted-prose lines are given up, 1,816 of them (82.62%) six tokens or longer, the longest a 34-token Dutch sentence. The word bar costs 79 lines (8,841 -> 8,762); >= 6 would cost 344 more; the stopword bar at 5 would cost 1,860 more.
#1013 fixed `_named_prose_language` naming a line of all-caps English acronyms as foreign prose, and added `_SHOUTED_MIN_STOPWORD` / `_SHOUTED_MIN_WORD`. The Java port asserts against fixtures recorded from the Python package, so #938's copies went stale the moment both landed. Regenerated with `laya-java/scripts/gen_fixtures.py`. Recorded on CPython 3.12 (Unicode 15.0.0), the version java.yml pins. Regenerating on 3.11 instead rewrites every Unicode table digest and `unicode_version` to 14.0.0, which is the drift this fixture set exists to catch rather than absorb. The behavioural delta is one case: a segment that read `fr` from `MON LA EST COM DES` now reads `pt` from the Portuguese sentence actually in the state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
44 pull requests from 8 contributors. Headline fixes: predict_long no longer splits a state that fits (#966), a line of English acronyms no longer routes to the multilingual checkpoint (#1013), and a mis-spelled min_confidence map key is refused instead of silently gating nothing (#1002). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged in 0.3.29. This is the Reproduced on real weights before and after: The reason this needed care rather than a quick patch is the third row. The all-caps guard is deliberate and its comment says why, so the obvious fix of stripping short all-caps tokens would have taken The second cause is the better find. One thing landed with this that you could not have known about. #938's Java port records its fixtures from Verified on the merged tree: all 83 suites pass, 76 real-weights decisions unchanged, and the fix is live in 0.3.29. |
…arrying NandhaKishorM#938 was merged at 561c496 -- the `Tokenizer.decode` commit -- so everything after it is still open: `hooks/`, `Decisions.java` and `fixtures/typed_decisions.json` are absent upstream. upstream/main has since moved 158 commits, which is what made this conflict. Of the 51 commits on this branch, 27 have a same-subject twin upstream: they landed with that merge under different SHAs. 24 are genuinely new. 13 files conflicted. Resolved per file rather than by preference. For ten of them the test is mechanical -- upstream has not touched the file since my work merged, so this side is upstream's content plus the new work, and `--ours` loses nothing. Three needed a decision: `decode/Decoder.java` takes UPSTREAM. My own comment in that file deferred the question: an empty action block padded to two and published {0.5, 0.5}, and the comment said changing it "belongs in its own change against that test rather than buried in this one; raised with the maintainer separately". The maintainer decided -- it throws now -- and taking this side would have reverted their decision on a question I handed them. `fixtures/lang_detect.json` takes UPSTREAM. It was re-recorded by 7a2f1bb after the Python lang fix landed as NandhaKishorM#1013, so upstream's copy pins the fixed behaviour and this branch's copy pins the bug. `.github/workflows/java.yml` takes OURS, verified rather than assumed. Upstream added one `gradle/actions/wrapper-validation` step in dc72133; this side has it in all four jobs that run `./gradlew`, each before that job's first `./gradlew`. Parsed the YAML to confirm, because dropping a security step is the worst thing a merge can do quietly. Two tests then failed, and BOTH fail on pristine upstream/main at e08843b with nothing from this branch in the tree. They are inherited, not caused here. `DecoderTest` is fixed here, because its assertion had stopped matching the code it guards. `Decoder.actionProbabilities` throws on a zero-width block while the test asserted {0.5f, 0.5f}. The implementation is the half that is right: 0.5 is a number no model produced, published into the field callers gate escalation on, and the reference raises there too -- `laya/onnx_agent.py` indexes `act[offset + r, 0]`, so zero columns cannot return. The test now asserts the refusal and its message, and a mutant that disables the refusal fails it. `LanguageDetectionParityTest` is NOT fixed in this commit, and the reason is worth stating. `english-note-with-acronym-line-and-foreign-line` expects `pt` and the port answers `fr`: the fixture was re-recorded from the fixed `laya/lang.py`, but `lang/LanguageDetection.java` was never updated to match, so the two have diverged in exactly the way the parity gate exists to prevent -- and in exactly the way NandhaKishorM#1013's own PR body warned they would if the fixture moved without the port. Porting that fix is a real change against 116 recorded cases and belongs in its own commit, not inside a merge. Verified: `gradle build -x test` clean under -Werror with javadoc; 924 tests complete; the only failure left is the LanguageDetectionParityTest case above.
…pected The Python side merged the caps bars (NandhaKishorM#1013 and its follow-up); the Java port never got them, so LanguageDetectionParityTest has been failing on main for the one recorded case where an all-caps line of English abbreviations outvoted a genuinely Portuguese line. MON, LA, EST, COM and DES are hockey teams, states, time zones and radio bands that collide with French stopwords. Before this, the bare-acronym line was named fr and won the whole-state verdict. Ported from laya/lang.py: - SHOUTED_MIN_STOPWORD and SHOUTED_MIN_WORD, with the reference's sweep recorded on each. - shouted(): a cased segment written entirely in capitals. The uppercase half is not redundant -- a caseless script has no lowercase either, and would otherwise be held to a bar its tokens cannot clear. Both halves go through the recorded case tables, not the JDK's isUpperCase/isLowerCase, which disagree with Python on 240 code points. - shoutedEvidence(): a word of SHOUTED_MIN_WORD letters, and then either non-English diacritics or a matched stopword of SHOUTED_MIN_STOPWORD letters. - nonLatinCarries(), named rather than left inline, because the caps bars ask the same question to decide what to keep out. - the bar applied in namedProseLanguage, and again in analyseText so the whole-state verdict obeys the module's own rule that an acronym is not a word. Vetoing there leaves the text undecided, so looksNonEnglish still decides it. - only acronym-shaped runs are blanked, via upperLetterRuns(). Blanking every all-caps run deletes emphasis capitals, which are ordinary in prose and usually sit on the words that carry the sentence. blankUpperRuns() itself is unchanged: it is a recorded helper the fixture pins, so analyseText filters before calling it. - leafNonEnglish(): a veto is not a verdict of English. A field whose named language the prose scan disbelieves now falls through to the diacritic branch instead of being dropped. The fixture is not regenerated. lang_detect.json was re-recorded upstream from the fixed reference and is the target here. ShoutedAcronymBarTest asserts the behaviour directly, because the fixture cannot: of the seven mutants used to check it, only one is caught by the parity suite at all. Deleting the analyseText block, blanking every all-caps run, reverting the leaf fall-through, dropping the non-Latin exemption and loosening either bar by one letter all leave LanguageDetectionParityTest green. Suite: 934 tests 1 failed -> 947 tests 0 failed. Parity 116/116.
BREAKING: Router(default=...) changes from "english" to "multilingual". Only text whose language the detector cannot place is affected; identified English still routes to the English checkpoint and detected non-Latin script already routed to multilingual. Evidence, from research/results/cpu_51_language_sweep_refreshed.json (laya 0.3.20): the multilingual checkpoint leads on 50 of 51 languages, the exception being English itself (0.820 against 0.710), and by 0.180 macro accuracy excluding English. The break-even is about 62% English traffic. The asymmetry that settles it for undecided text is the failure mode rather than the mean: off English the English checkpoint collapses while staying confident (Khmer 0.000 accuracy at 0.952 mean confidence), so no min_confidence gate downstream can catch it. #54 measured the cost directly: 128 of 200 German utterances reached the undecided branch and lost 20 accuracy points against model="multilingual" on the same rows. #384 is the same fallback reached through dict-state input. Restore the old behaviour with Router(default="english") or LAYA_DEFAULT_MODEL=english. Tests that pinned the old constant now pin an explicit default where they are about something else (hooks, budgets, batch grouping, the #1013 acronym fix), and assert both arms where they are about the default itself. README, docs/routing.md, docs/docker.md and docs/cli-mcp.md updated; the serve tripwire that exists to catch exactly that is updated with them. Also: the README keeps only the current release's notes, with older ones left to the GitHub releases page. The batch-hooks bullet moved into the batch-prediction section, since it documents which kwargs each method takes rather than release history. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One line of ordinary English abbreviations inside an otherwise English state makes
analysename the whole state French, andRouter.routethen sends it to themultilingual checkpoint. Reproduces on
mainata4a8921.MON,DES,ESTandLAare Monday, a part number, Eastern Standard Time and LosAngeles. They collide with the French stopwords des, est and la. A wrong route is not
a slightly worse answer — the English checkpoint collapses off English and the multilingual
one gives up about 6 points on English suites, so this is a confidently wrong answer on
ordinary support text.
Why it happens
laya/lang.py, in_named_prose_language:The acronym blanking is conditional on the segment containing a lowercase letter, and that
condition is deliberate — the comment above it says why: "A segment written entirely in
capitals keeps its words -- a customer shouting in Portuguese is still Portuguese."
A line of bare acronyms has no lowercase letter either, so it takes the same branch: the
acronyms survive as "words", clear the four-token bar, and
latin_profilenames them. The>= 2 distinct stopwordsbar does not help, because des, est and la are threedifferent French stopwords.
The guard is right in both directions it was designed for. Only the third case is wrong:
mainThe MON DES EST LA codes were sent todayNoneNone— correctESTA MENSAGEM E CONFIDENCIAL E NAO DEVE SER COMPARTILHADAptpt— correctMON DES EST LAfrNoneA second, deeper cause showed up while fixing the first:
_analyse_textnever applied themodule's own rule that an acronym is not a word, so acronyms voted in the whole-state
verdict as well. That is why
The MON DES EST LA codes were sent todayroutesmultilingualonmaineven though_named_prose_languagecorrectly returnsNoneforit — the helper was not the only path.
Why the obvious fix is wrong
Stripping all-caps tokens up to acronym length in an all-caps segment breaks the case the
guard exists for. Measured against
_STOP:ESTA MENSAGEM E CONFIDENCIAL E NAO DEVE SER COMPARTILHADAwould loseESTA,NAO,DEVEandSER, drop below the two-distinct-stopword bar and returnNone. That tradesthis defect for a worse one.
The second obvious fix is also wrong, and this one is worth recording because it looked
right and passed every test in the repo. Blanking the all-caps runs in mixed-case text and
re-voting has no evidence bar at all, so when the words that named the language are the
shouted ones, the evidence simply disappears:
Over real MASSIVE rows (8 Latin locales, n=20,708): 1,753 rows regressed with the last
two words capitalised and 3,835 with the first half capitalised, and nothing improved.
Monotonicity broke outright — 4,285 rows where the fully upper-cased form routed
multilingualbut the same text with"ok "prefixed routedenglish, i.e. adding onelowercase English token made foreign detection strictly worse.
What this does instead
Three changes, each measured:
matched stopword of
_SHOUTED_MIN_STOPWORDletters, or non-English diacritics,and one word of
_SHOUTED_MIN_WORDletters. Four is the lowest stopword bar thatrejects
LOS/LAS/EL; three leaves the reported string standing._analyse_textapplies the module's own acronym rule. It re-takes the whole-stateverdict with all-caps runs blanked for mixed-case text and applies the shouted bar for
all-caps text, leaving the text undecided so
looks_non_englishstill decides — thediacritic safety net is measured on the original text.
run reaches
_SHOUTED_MIN_WORD. This is what keeps emphasis capitals: it cuts the tworegression figures above from 1,753 and 3,835 to 447 and 856, and the
"ok "-prefixedshape from 7,744 to 117, with monotonicity violations back to 316 against a baseline
of 279.
Plus:
_shoutednow requires an uppercase character, so caseless scripts are not held to abar they cannot clear; a veto falls through rather than being read as a verdict of English
(114 rows recovered, 0 newly dropped, verified line by line rather than as a net); and
_non_latin_carrieskeeps a state that is mostly non-Latin out of the caps veto entirely.End-to-end, on real weights
Both checkpoints loaded from disk — ModernBERT-large and mmBERT-base — and
router.predictrun end to end, so the change is observed through actual inference rather than through a
routing-key check that never loads a model.
lang.py's sha256 is recorded per run anddiffers between the two, which is what proves the fix was in the tree under test.
maina4a8921multilingual/frenglish/enMON DES EST LAbaremultilingual/frenglishok MON DES EST LAmultilingual/frenglishCOM DOS LAN WAN VPN ROUTERmultilingual/ptenglishSTORES LOS ANGELES LAS VEGAS EL PASO CLOSEDmultilingual/esenglishThe MON DES EST LA codes were sent todaymultilingual/frenglish/ensag mir das HEUTIGE DATUMmultilingual/demultilingual/dequiero cancelar mi PEDIDO POR FAVORmultilingual/esmultilingual/esWELCHE WECKER habe ich gestelltmultilingual/demultilingual/deESTA MENSAGEM E CONFIDENCIAL ...multilingual/ptmultilingual/ptQUERO MEU DINHEIRO DE VOLTA AGORAmultilingual/ptmultilingual/ptПРИВЕТ MON DES EST LAmultilingualmultilingualenglish/multilingualenglish/multilingualSix cases fixed, including one —
The MON DES EST LA codes were sent today— that theoriginal report listed as already correct, because
_named_prose_languagereturnsNonefor it while the whole-state path still routed it
multilingual. Every shouted-foreign andemphasis-capital case holds. The two sentinels must land on different checkpoints or the
comparison is vacuous; they do.
One deliberate casualty, stated rather than hidden:
WIE SPAET IST ES IN KOELN— theASCII spelling, no umlauts — goes
multilingualtoenglish. It carries three matchedGerman stopwords, longest word 3 letters, diacritic rate 0.0, so nothing distinguishes it
from a run of codes. The umlaut spelling
WIE SPÄT IST ES IN KÖLNstaysmultilingualonboth. A
>= 4 distinct matched stopwordsdisjunct would recover 393 real prose lines atzero false positives on the rebuilt corpora and rejects all five reported acronym strings —
left out of this PR as a separate change with its own evidence needs.
Also run: the rig's 39-case real-model routing harness, A/B. 0 route differences, the
only differing field being wall-clock
ms, with its own route sentinels passing. Thatcorpus targets the per-leaf scan rather than capitalisation, so it is a regression result,
not evidence the fix works — the table above is that.
Tests
tests/test_lang_guess.py111 → 176, add-only: the test diff removes exactly one lineand it is an import. 96 lines added, 1 removed — no existing assertion was dropped or
loosened, because none of them encoded the old behaviour.
Run the way CI runs them (
python tests/<name>.py, perci.yml), with both checkpoints andthe exported multilingual graph on disk:
Every count except
test_lang_guessis byte-identical to the same file ata4a8921, sonothing outside the changed module moved. One honest note on that last line:
test_onnx_lang_paritypasses identically with no checkpoint and no graph — it drives_inferwith a stub session and a fake tokenizer by design, as its own docstring says. It islisted here as a suite count, not as real-weights evidence; that evidence is the table above.
New assertions are behavioural —
Router.route(...)["model"], not just the private helper —covering ten emphasis-capitalised foreign lines (eight real MASSIVE rows), the monotonicity
pair, three acronym lines including
ok MON DES EST LA, and two mixed-script states.Five mutants, each reverting one rule, all killed by assertions rather than errors: the
shouted bar (21 failures), the caseless-carry guard (4), the whole-state caps veto (6), the
_leaf_non_englishfall-through (3), and the acronym-shaped blanking gate (21). An earlierversion of one test was vacuous — it passed under its mutant because its diacritics
cleared the bar directly so the fall-through never ran; it is replaced with three real
MASSIVE rows that are genuinely vetoed (one distinct stopword where two are required) and
carry diacritics above the rate.
How this was validated
The fix went through two rounds of adversarial review before this PR, and both rounds found
real defects. Recording that here because the second round is the reason the current
implementation is narrow rather than broad, and a reviewer should not have to rediscover it.
Round one, against the first attempt. Four findings. The headline one: the commit message
asserted the defect was fixed, and the defect still reproduced —
analyse("MON DES EST LA")was fixed in isolation while the diluted form (the acronym line inside real Englishprose, which is the actual reported bug) still routed
multilingualat 0, 1 and 2 lines ofEnglish context. The verification behind that claim had tested only the single 4-line document
the commit itself used. The other three: the threshold table's numbers did not reproduce, a
veto was being read as a verdict of English, and the shouted bar was measured on the wrong
token set.
Round two, against the rework. Nine findings, two of them serious enough to change the
design:
"Why the obvious fix is wrong" above — 1,753 and 3,835 rows over real MASSIVE data, nothing
improved, monotonicity broken. This is what produced change 3 in "What this does instead".
0.000000across 51locales, from a corpus with zero uppercase characters. Documented below.
Plus: a mixed cased-and-caseless state routed to the English checkpoint; the "114 rows
recovered" figure did not reproduce as written (it is carrier-dependent, and measures 41
with the carrier the test itself uses); two rows of the threshold table silently included a
second bar so the six rows were not on a common footing; the four routing-share figures
reproduced in delta but not in absolute level; and the worked example in the code comment
was wrong —
WIE SPAET IST ES IN KOELNhas no umlauts and is a casualty of this change, nota case it fixes.
What was done about the measurement discipline, not just the code. Three habits carried
through this change and each of them caught something:
was found vacuous — it passed under its own mutant because its diacritics cleared the
bar directly, so the code path it claimed to pin never ran. It was replaced with three real
MASSIVE rows that are genuinely vetoed (one distinct stopword where two are required) and
carry diacritics above the rate.
rather than assumed. Counting is what exposed the blind sweep.
lang.py's sha256 per run and the two differ; the harness also carries route sentinels thatmust land on different checkpoints, so a run where everything agrees cannot be mistaken for
a working comparison. Without those two guards, "no differences" is equally consistent with
having tested the same code twice.
On the evidence, because one piece of it was worthless
research/evals/massive_route_comparison.pyover the pinned MASSIVE split reports aper-locale routing delta of exactly
0.000000at all 51 locales for this change. Thatnumber is not evidence of anything: not one of its 151,674 rows contains a single
uppercase character, so
_shouted()is always false and theisupper()blanking is ano-op — the changed code is unreachable from that corpus. The per-leaf path is unreachable
too, because the harness passes a plain string and
_leaf_non_englishreturns early forone.
So the cost is measured on corpora that can see the change instead: upper-cased,
emphasis-capitalised and field-wrapped rows derived from the same pinned split. Of 10,960
upper-cased MASSIVE lines the baseline names, this keeps 8,762 and gives up 2,198 —
82.62% of them six tokens or longer, the longest a 34-token Dutch sentence. These are
sentences, not fragments, and that is the real price of the change.
research/evals/shouted_bar_corpora.pyis added so the threshold choice is reproducible.The previous justification rested on 500,000 synthetic acronym lines and 50,000 address
lines that nothing in the repository could rebuild, which made every constant in the file
header unfalsifiable. The generator is deterministic, pinned to MASSIVE
940fd47aandrandom.Random(0), applies the bar from swept parameters so it survives the constantsmoving, and runs offline. Stated plainly: on the rebuilt corpora a stopword bar of 4 already
closes both false-positive columns to zero, so
_SHOUTED_MIN_WORD = 5buys nothingmeasurable there and the old "10:1" and "86:1" trade ratios do not reproduce. The constant
is kept — a smaller corpus failing to find a false positive is not evidence there is none —
and the header now says exactly that and what it would take to move it.
Scanning
CodeQL (python,
security-extended+security-and-quality) and Semgrep (p/python,p/security-audit) both run over this branch. No finding in any of the three files itchanges. The pre-existing findings elsewhere are unchanged, including the
dynamic-urllib-use-detectedonlaya/mcp/remote.py:96thatsecurity.ymlalreadydocuments as the false positive keeping Python out of the blocking Semgrep gate.
Note for the JVM port
laya-javapins the CURRENT behaviour infixtures/lang_detect.json(
english-note-with-acronym-line-and-foreign-linerecordslanguage: "fr",is_english: false). That is deliberate — the port follows the reference including thisdefect — so when this lands, that fixture is re-recorded in the same change and the two move
together rather than silently diverging.