Repository navigation
fix(router): read lang="en_US" as English - #204
paulocarinhena wants to merge 1 commit into
Conversation
`route(lang=...)` took the primary subtag by splitting on "-" only, which fits BCP 47 (`en-US`) but not the POSIX/Django/Babel spelling (`en_US`, `en_US.UTF-8`) that a locale read from the environment or a user profile usually has. `lang="en_US"` was therefore treated as a non-English language and routed to the multilingual checkpoint, the opposite of what the caller asked for, as was `lang=" en "`. Split on `-`, `_`, `.` and `@`, after stripping whitespace.
01ba951 to
4c7b576
Compare
|
Thanks for this. #211 landed today and its |
|
Closing as superseded: #211 landed with Your framing of why it mattered is the part worth keeping. Splitting on Verified still correct on today's Thanks for the careful locale handling, and for the run of Portuguese routing work around it. |
The problem
Router.route(lang=...)extracts the primary language by splitting on-only:That works for BCP 47 tags (
en-US). It does not work for the POSIX / Django / Babel spelling with_, which is what a locale looks like when it comes from theLANGenvironment variable,locale.getlocale(), Babel'sstr(Locale), Django'sto_locale()/ locale directory names, or a stored user profile:lang=is the explicit override, and the README recommends passing it when the caller already knows the language. When the caller says "this is English", the router sends it to the checkpoint that is weaker on English (MASSIVE intent 0.657 vs 0.783) and thereasonreportsexplicit lang='en_US', so nothing looks wrong.What this PR changes
One line in
Router.route. After stripping whitespace, it takes the primary subtag by splitting on-,_,.or@:Nothing else changes. Non-English locales in either spelling (
pt_BR,pt-BR) still go tomultilingual, and a code that merely starts withen(enm, Middle English) is still not treated as English.Tests
tests/test_router.pyadds cases to the existing explicit-routing table:en_US,EN_GB,en_US.UTF-8and" en "go toenglish, even on Hindi text, which proves the override wins over detection.pt_BR,pt-BRandenmgo tomultilingual.