Skip to content
Closed
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -578,7 +578,7 @@ router.route({"body": "Der Kunde wurde zweimal belastet"}, questions).reason
# "Latin script but language looks like 'de', not English"
```

Very short Latin-script text often carries nothing that identifies its language (`"Quero cancelar"`, `"Esqueci minha senha"`). Such text goes to `default`, which is `"multilingual"` since 0.4.0. Text the detector *can* place is unaffected: identified English still goes to the English checkpoint.
Very short Latin-script text often carries nothing that identifies its language (`"Quero cancelar"`, `"Esqueci minha senha"`). Such text goes to `default`, which is `"multilingual"` since 0.4.0. Text the detector *can* place is unaffected: identified English still goes to the English checkpoint. From four words up, text that names no language and has no English function word, but does contain evidence of another language, goes to the multilingual checkpoint regardless of `default`: a non-English letter, two different words from other languages' lists (`"streiche alle meine geplanten termine"`), or one list word that English text never uses (`"batalkan alarm saya pukul tujuh pagi"`). Those words are `_FOREIGN_WORDS` in `laya/lang.py`, generated by `scripts/build_foreign_words.py`. One word that English uses too is not enough: `do`, `todo` and `van`.

```python
router = Router() # default="multilingual"
Expand Down
61 changes: 61 additions & 0 deletions laya-ts/src/foreign-words.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
/** Function words of other languages that English text does not use.
*
* Generated by `scripts/build_foreign_words.py`; do not edit by hand. Same words as `_FOREIGN_WORDS`
* in `laya/lang.py`: the words of the non-English lists found in none of 212,592 English documents.
* 841 words.
*/
export const FOREIGN_WORDS: ReadonlySet<string> = new Set(`
abar abbiamo aceasta această acest acesta acolo acum adakah adesso agora aici ainda ajke akan
akhon ako aku algo alguem alguém alla alle alors alsjeblieft alstublieft amader amake amma amra
ancaq anche ancora apakah apnake apnar apnara apni appen aqui aquí arep artiq artıq asbe
asseblief astazi atau aterbetala aterbetalning até auch auf aunque avete awak azi baada babagan
bado bagaimana bakal behover behöver belum berapa berapakah besok bhai bhalo bien bij bila
bilakah bir bisa bisakah bitte ble blivit boleh bolte bolun bukan bukas bydd cadê cand cati
chaina chdi ciao cine como consigo cuando daardie dadi dagli daha dalam dalla dallo dapat
dapatkah dari daripada dat ddim debiterade deg degli dello dengan denne depois dere deres despre
dessa detta dette deu deux deve devo devono deyil deze dhateng dheweke dhonnobad dich diese
diesen dieser dieses dilam dimana dimanakah diolch dir disse dite diye diyechi dois doit donc
donde dorkar duibar dupa după durung də efo eger ei eich ein eine einen einer eisiau eita ekbar
ekhon ekhono ekta elles endi entao então esok esta estamos estava esto estou está etta etter
että faktura fara fel ferot ficou fiz foarte foi fois fost fra från fue fueron fyddai får før
für fără gaan gallu gan ganger gani geche geen gelem gese gibt già gli gostaria graag gyda gyfer
gång gånger gör göra görə habe haben hadde haluaisin haluan hanen hanya hapa hapana harus havde
heb hebt heddiw heeft hefyd heidän hemos hendak hende hendes heute hierdie hii hij hingga hittar
hivi hiyo hizi hjalp hjalpa hjälp hjälpa hobe hocche hoeveel hoise hoje hoye hoyeche hoyni hulle
huomenna huu hva hvad hvala hvem hvilke hvilken hvilket hvor hvordan hvorfor hvornaar hvornår
hwn hyn hän hänen här həm hər iawn ich ieri ik ikaw iki ikke iku ile ili ilə imorgen ingin ini
installningarna installningen inställningarna inställningen inte isio isso isto isə iti iyan
iyon jabe jaz jeg jer jeres jetzt jij jika jodi joku jonno jotain jours jouw juga julle jutri jy
já kabeh kabla kaikki kailan kaj kako kalau kalian kaliyan kalke kama kami kamu kanggo kann
kannst kanthi kapan karena karo katera kateri katero katika kdaj kdo kenapa kenen keno kepada
kepiye kerana kesho ketika keu kharap khub kibhabe kichu kiitos kila kintu kita kivabe kje
kokhon koliko kommer korbo korchi kore koreche korechi korlam korsi korte korun kothay koto kowe
kraschar kuhusu kuinka kuka kula kulo kuna kunne kutoka kuwi kvittot kwa kwamba kwenye lagbe
lagi lakin lakini lebih lini losenord lui luwih lösenord maar mahu mais mana manakah maneh masih
mau mayroon mboten meget meidän meine meinem meinen meiner mellem mellom menawa mengapa menika
menyang mereka mes meu meus mihin mij mijn miksi mikä milloin mimi mina minha minhas minua
minulle minun minä missa missä mista mistä mita miten mitä moddhe mois moj moja moje molto muito
mung mutta mye myos myös más måste môre mən naar nach nai nama naman namin nang nanging nani nao
naomba nar nataka natin nawr naš ndani ndiyo necesito negli nei nekaj nel nella neng nenhum
nenhuma neu ngapi ngayon nggih nhw nicht nie niet nila ninataka ning ninguem ninguém nini niya
niye nje njegov njen noch noen nog nogen noget noite nossa nosso nous nya nyinyi não nämä när
nästa nå når nə obrigada obrigado och också od oder oggi ogsaa også oi olan oleh oli olmasa olub
olá ond onde onek oni onko onlar ontem onze oor opo oppnar ora ovat paano pabo pada paini paket
paki pakiusap paling paljonko pamoja panjenengan parben parbo parchi parchina pela pelo pengin
pentru perche peux peyechi pira piye più plis poate pob poco pode podem por porque poti pourquoi
preciso prosim pryd puede pueden punika punya pwy până quand quando quero questa questo quiero
rhaid rhywbeth roedd saan saat sahaja saja saka saking samo sampai sampeyan sampun sangat sapa
sathe saya scusa sebelum sedaj sedang sekali sekarang semasa sempre semua semula sesudah sesuk
setelah shathe shob shomossa siapa siapakah sich siella siellä sila silakan sina sini sinua
sinulle sinun sinä sisi siya siz skal skicka skulle smo sobre soko somossa sono sonra sparningen
spårningen stato ste sua sudah sul sulla sulle sunt sut sydd são säger să sən taalla tader
tafadhali tahole takk tama taman tambem também tanaan tapi tarde tayo teidän telah tena tengo
tenho tentang tetapi theke tidak tiene tienen tolong tomar tomra tous toute trebuie très três
tudi tudo tulung tumi tuo tva tvoj tvoja två tá tämä tämän tänään täällä ucun uga ugi uit unde
unrhyw untuk upp uppdaterats utawa utawi valo vandag var vas vaš ved venligst vennligst veut
veux vill vipi voces você vocês voglio voisitko voitko voor vores vorrei vous vrea vreau vse vya
våre vă və waar waarom wae wakati wala wangu wann wanneer wao wapi wedi welche werden wewe wij
wir wird worden wordt wrth wurde yake yako yalniz yalnız yangu ydy yeye yfory yma yn yna yote
yox yoxdur yw zaidi zakaj zal zdaj zij zijn zum zur är återbetala återbetalning çox è é être în
öppnar üçün če și ți əgər
`.split(/\s+/).filter((w) => w.length > 0));
95 changes: 89 additions & 6 deletions laya-ts/src/lang.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import { FOREIGN_WORDS } from "./foreign-words.js";

type ScriptRanges = Array<[string, Array<[number, number]>]>;

const SCRIPT_RANGES: ScriptRanges = [
Expand Down Expand Up @@ -76,7 +78,10 @@ const STOP: Record<string, Set<string>> = {
"nel", "nell", "negli", "sul", "sulla", "sulle", "dal", "dalla", "dallo", "dagli", "dei",
"delle", "dello", "degli", "agli", "alle", "col"]),
nl: new Set(["het", "een", "van", "is", "op", "te", "dat", "niet", "met", "voor", "zijn", "aan",
"door", "maar", "ook", "worden", "deze", "naar", "wordt"]),
"door", "maar", "ook", "worden", "deze", "naar", "wordt",
"mijn", "ik", "mij", "jij", "jouw", "uw", "zij", "wij", "hij", "hem", "haar", "onze",
"hun", "alsjeblieft", "alstublieft", "graag", "heb", "hebt", "heeft", "kun", "zal", "geen", "bij",
"tot", "welke", "waarom", "morgen", "er"]),
// Keep Swedish routing in parity with Python. Distinctive words help ASCII-normalised ticket text;
// shared Nordic words are tracked separately so Danish and Norwegian are not named Swedish.
sv: new Set(["jag", "är", "och", "inte", "att", "från", "till", "behöver", "får", "skulle", "ska",
Expand All @@ -89,7 +94,10 @@ const STOP: Record<string, Set<string>> = {
"blivit", "debiterade", "appen"]),
ro: new Set(["și", "să", "este", "sunt", "care", "pentru", "din", "dar", "după", "până", "fără",
"ale", "lui", "în", "fost", "acum", "vreau", "trebuie", "foarte", "acest", "această",
"acesta", "aceasta", "mi", "ți", "vă", "nu"]),
"acesta", "aceasta", "mi", "ți", "vă", "nu",
"despre", "cum", "rog", "cu", "mea", "mei", "azi", "astazi", "cand", "poti", "unde",
"cine", "cate", "cati", "imi", "iti", "vrea", "poate", "pot", "fara", "dupa", "pana", "maine", "aici",
"acolo"]),
// Romanized Bangla ("Banglish"): how Bangla is typed in chats, tickets and email when no Bengali
// keyboard is at hand. It has no diacritics, so without a list it read as undecided-but-English
// and went to the English checkpoint, which scores 0.08 on Bangla at 0.94 confidence.
Expand All @@ -108,6 +116,67 @@ const STOP: Record<string, Set<string>> = {
"var", "yox", "yoxdur", "mən", "sən", "biz", "siz", "onlar", "daha", "çox", "cox",
"hər", "nə", "kimi", "görə", "sonra", "əgər", "eger", "deyil", "lakin", "amma",
"ancaq", "artıq", "artiq", "də", "isə", "həm", "yalnız", "yalniz"]),
// Plain-ASCII languages that had no list (#54); laya/lang.py says what was left out and why.
id: new Set(["yang", "untuk", "dengan", "dari", "ke", "ini", "itu", "saya", "aku", "kamu", "anda",
"dia", "kami", "kita", "mereka", "apa", "apakah", "bagaimana", "kapan", "dimana", "mana", "siapa",
"berapa", "tidak", "bukan", "akan", "sudah", "belum", "sedang", "bisa", "dapat", "harus", "mau",
"ingin", "tolong", "juga", "atau", "tetapi", "tapi", "pada", "dalam", "oleh", "karena", "jika",
"kalau", "ada", "semua", "lagi", "saja", "nya", "besok", "sekarang", "tentang", "kepada", "daripada",
"mengapa", "kenapa", "silakan", "bisakah", "dapatkah", "sesudah", "sebelum", "setelah", "hingga",
"sampai", "sangat", "lebih", "paling", "masih", "hanya", "punya", "kalian", "saat", "ketika",
"sini"]),
ms: new Set(["yang", "untuk", "dengan", "dari", "ke", "ini", "itu", "saya", "aku", "awak", "anda",
"dia", "kami", "kita", "mereka", "apa", "adakah", "bagaimana", "bila", "mana", "siapa", "berapa",
"tidak", "tak", "bukan", "akan", "sudah", "telah", "boleh", "mahu", "hendak", "tolong", "juga",
"atau", "tetapi", "pada", "dalam", "oleh", "kerana", "jika", "kalau", "ada", "semua", "lagi",
"sahaja", "esok", "sekarang", "berapakah", "kepada", "daripada", "bilakah", "semula", "kenapa",
"mengapa", "siapakah", "dimanakah", "manakah", "sekali", "pun", "sangat", "lebih", "paling", "masih",
"hanya", "punya", "kalian", "semasa", "ketika", "sini"]),
jv: new Set(["aku", "kowe", "sampeyan", "dheweke", "iki", "kuwi", "iku", "lan", "karo", "ing",
"menyang", "saka", "kanggo", "apa", "piye", "kepiye", "kapan", "endi", "sapa", "pira", "ora", "wis",
"durung", "arep", "bakal", "isa", "iso", "bisa", "gelem", "tulung", "uga", "utawa", "nanging", "ana",
"kabeh", "maneh", "wae", "mung", "kang", "sing", "ning", "neng", "nang", "sesuk", "saiki", "opo",
"ku", "kula", "soko", "babagan", "dadi", "kanthi", "menawa", "pengin", "sampun", "mboten", "nggih",
"kulo", "panjenengan", "niki", "menika", "punika", "saking", "dhateng", "kaliyan", "ugi", "utawi",
"luwih"]),
sw: new Set(["wa", "za", "kwa", "katika", "hii", "hiyo", "huu", "hizi", "mimi", "wewe", "yeye", "sisi",
"nyinyi", "wao", "nini", "vipi", "lini", "wapi", "nani", "ngapi", "sana", "tafadhali", "kwamba",
"lakini", "kama", "bado", "tena", "pia", "yote", "kila", "leo", "kesho", "yangu", "yako", "yake",
"wangu", "wako", "kwenye", "cha", "vya", "gani", "kutoka", "kuhusu", "zaidi", "hivi", "kuna",
"nataka", "ninataka", "naomba", "sasa", "pamoja", "ndiyo", "hapana", "asante", "ndani", "nje", "hapa",
"pale", "wakati", "kabla", "baada", "ili"]),
tl: new Set(["ang", "ng", "mga", "ay", "ako", "ikaw", "ka", "siya", "kami", "tayo", "sila", "ko",
"niya", "namin", "natin", "nila", "ito", "iyan", "iyon", "ano", "paano", "kailan", "saan", "sino",
"ilan", "hindi", "oo", "po", "naman", "lang", "rin", "ngayon", "bukas", "kung", "mayroon", "wala",
"gusto", "paki", "pakiusap"]),
af: new Set(["en", "nie", "ek", "jy", "hy", "sy", "ons", "julle", "hulle", "vir", "uit", "oor",
"wat", "hoe", "wanneer", "waar", "hoeveel", "asseblief", "kan", "sal", "moet", "wil", "gaan",
"dit", "hierdie", "daardie", "nog", "vandag", "môre"]),
cy: new Set(["yr", "yn", "gyda", "wrth", "oes", "ydy", "yw", "roedd", "bydd", "fy", "dy", "eich",
"beth", "sut", "pryd", "ble", "pwy", "faint", "dim", "ddim", "ond", "neu", "hefyd", "nawr", "heddiw",
"yfory", "chi", "ti", "nhw", "fe", "gan", "efo", "yma", "gyfer", "sydd", "dwi", "unrhyw", "wedi",
"gallu", "eisiau", "isio", "fyddai", "rhaid", "chdi", "fo", "hwn", "hon", "hyn", "yna", "rhywbeth",
"pob", "mwy", "iawn", "diolch", "plis"]),
da: new Set(["jeg", "hun", "ham", "hende", "jer", "dine", "hendes", "vores", "jeres", "deres", "og",
"af", "til", "ved", "hos", "før", "mellem", "hvad", "hvem", "hvor", "hvornår", "hvornaar", "hvordan",
"hvorfor", "hvilken", "hvilket", "hvilke", "er", "havde", "kunne", "skal", "ikke", "venligst",
"imorgen", "denne", "dette", "disse", "noget", "nogen", "også", "ogsaa", "meget", "mere", "kun",
"paa"]),
nb: new Set(["jeg", "hun", "dere", "meg", "deg", "ham", "dine", "våre", "deres", "og", "til", "ved",
"hos", "etter", "før", "mellom", "hva", "hvem", "hvor", "når", "hvordan", "hvorfor", "hvilken",
"hvilket", "hvilke", "er", "hadde", "kunne", "skal", "ikke", "nei", "takk", "vennligst", "nå",
"imorgen", "denne", "dette", "disse", "noe", "noen", "også", "mye", "kun"]),
fi: new Set(["minä", "mina", "sinä", "sina", "hän", "minun", "sinun", "hänen", "hanen", "meidän",
"teidän", "heidän", "minulle", "sinulle", "minua", "sinua", "mitä", "mita", "mikä", "mika", "missä",
"missa", "mistä", "mista", "mihin", "milloin", "miten", "kuinka", "kuka", "kenen", "miksi",
"paljonko", "onko", "voitko", "voisitko", "haluan", "haluaisin", "ole", "oli", "ovat", "tai", "mutta",
"että", "etta", "jos", "kun", "myös", "myos", "nyt", "tänään", "tanaan", "huomenna", "kiitos",
"kaikki", "jotain", "joku", "tämä", "tama", "tämän", "taman", "nämä", "nama", "siellä", "siella",
"täällä", "taalla"]),
sl: new Set(["jaz", "ona", "oni", "moj", "moja", "moje", "tvoj", "tvoja", "njegov", "njen", "naš",
"vaš", "vas", "kaj", "kdo", "kje", "kdaj", "kako", "zakaj", "koliko", "kateri", "katera", "katero",
"sem", "smo", "ste", "bo", "bom", "bi", "ker", "če", "z", "iz", "od", "pri", "prosim", "hvala",
"danes", "jutri", "zdaj", "sedaj", "tudi", "samo", "vse", "nekaj"]),
};

const NON_EN_DIACRITICS = new Set(
Expand Down Expand Up @@ -159,6 +228,9 @@ const EN_COLLISION_WORDS = new Set(
["come", "son", "do", "care", "todo", "im", "per", "plus"].filter((w) =>
Object.entries(STOP).some(([lg, sw]) => lg !== "en" && sw.has(w))),
);
// Every word a non-English list holds. In undecided text of four or more words, two different ones are
// evidence of another language, and so is one in FOREIGN_WORDS, which English text never uses (#54).
const OTHER_WORDS = new Set(Object.entries(STOP).filter(([lg]) => lg !== "en").flatMap(([, ws]) => [...ws]));

// Short support fragments need a narrower vocabulary than the general four-word language guess.
// Generic words such as "fel" and "hjälp" are deliberately omitted from this subset.
Expand Down Expand Up @@ -314,6 +386,7 @@ export interface LatinProfile {
englishHits: number;
diacriticRate: number;
looksNonEnglish: boolean;
otherEvidence: boolean;
}

export function latinProfile(text: string): LatinProfile {
Expand All @@ -330,10 +403,13 @@ export function latinProfile(text: string): LatinProfile {
const nordicOverlap = words.some((w) => NORDIC_OVERLAP_WORDS.has(w)) &&
!words.some((w) => EN_ONLY_WORDS.has(w));
if (words.length > 1 && words.length < 4 && words.some((w) => SHORT_SWEDISH_WORDS.has(w))) {
return { language: "sv", englishHits: 0, diacriticRate: diacRate, looksNonEnglish: nonEnglish };
return { language: "sv", englishHits: 0, diacriticRate: diacRate, looksNonEnglish: nonEnglish, otherEvidence: false };
}
if (words.length < 4) {
return { language: null, englishHits: 0, diacriticRate: diacRate, looksNonEnglish: nonEnglish || nordicOverlap };
return {
language: null, englishHits: 0, diacriticRate: diacRate, looksNonEnglish: nonEnglish || nordicOverlap,
otherEvidence: false,
};
}
// A collision word counts once however often it repeats; every other word counts its hits.
const counts = new Map<string, number>();
Expand Down Expand Up @@ -366,7 +442,13 @@ export function latinProfile(text: string): LatinProfile {
} else if (en && (!nonEnglish || englishRescuedByWords(wordSet, diacRate))) {
lang = "en";
}
return { language: lang, englishHits: en, diacriticRate: diacRate, looksNonEnglish: nonEnglish || (lang === null && nordicOverlap) };
// a non-English letter too rare for the rate counts as evidence too
const otherWords = [...wordSet].filter((w) => OTHER_WORDS.has(w));
const otherEvidence = diac > 0 || otherWords.length >= 2 || otherWords.some((w) => FOREIGN_WORDS.has(w));
return {
language: lang, englishHits: en, diacriticRate: diacRate,
looksNonEnglish: nonEnglish || (lang === null && nordicOverlap), otherEvidence,
};
}

export function guessLatinLanguage(text: string): string | null {
Expand Down Expand Up @@ -490,7 +572,8 @@ function analyseText(text: string): AnalyseResult {
const profLat = latinProfile(text);
const lang = profLat.language;
const undecided = lang === null;
const english = lang === "en" || (undecided && !profLat.looksNonEnglish);
// Undecided text with evidence of another language is not English either (#54), see laya/lang.py.
const english = lang === "en" || (undecided && !profLat.looksNonEnglish && !profLat.otherEvidence);
return {
script: "latin", scriptProfile: prof, language: lang,
isEnglish: english, languageUndecided: undecided,
Expand Down
9 changes: 8 additions & 1 deletion laya-ts/src/router.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { analyse, type AnalyseResult } from "./lang.js";
import { NON_EN_DIACRITIC_RATE, analyse, type AnalyseResult } from "./lang.js";
import type { PredictOptions, QuestionDef, SystemOneResult } from "./agent.js";
import { checkMinConfidence, flagLowConfidence } from "./common.js";
import { decide, type DecideOptions, type DecisionResult } from "./structured.js";
Expand Down Expand Up @@ -557,6 +557,13 @@ export class Router extends HookRegistry {
`(${JSON.stringify(det.mixedSegment.slice(0, 60))}); the English checkpoint cannot read it`;
} else if (det.language) {
reason = `Latin script but language looks like ${JSON.stringify(det.language)}, not English`;
} else if (det.diacriticRate < NON_EN_DIACRITIC_RATE) {
// Too few non-English letters to count alone: `analyse` routed it on four or more words with no
// English function word and some evidence of another language (#54).
const evidence = det.diacriticRate ? "a non-English letter" : "function words of other languages";
reason =
`Latin script, language not identified; no English function word, but ${evidence}; ` +
"not safe for the English checkpoint";
} else {
reason =
`Latin script, language not identified but ${Math.round(100 * det.diacriticRate)}% ` +
Expand Down
3 changes: 2 additions & 1 deletion laya-ts/tests/lang.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,8 @@ describe("lang", () => {
it.each<[string, string | null]>([
["Grüße aus Köln, wir melden uns wegen der Rechnung", "de"],
["sluk lyset i soveværelset", null],
["kan jeg få en refundering for det dobbelte beløb", null],
// Danish has no list; `kan` and `en` are Afrikaans too, which names the wrong language but not English
["kan jeg få en refundering for det dobbelte beløb", "af"],
["stäng av ljuset i sovrummet", null],
["jag vill ha en återbetalning för den dubbla avgiften", "sv"],
["The naïve façade needs a fresh coat of paint", null],
Expand Down
Loading
Loading