Skip to content

fix(router): read lang="en_US" as English - #204

Closed
paulocarinhena wants to merge 1 commit into
NandhaKishorM:mainfrom
paulocarinhena:fix/router-lang-underscore
Closed

paulocarinhena wants to merge 1 commit into
NandhaKishorM:mainfrom
paulocarinhena:fix/router-lang-underscore

Conversation

@paulocarinhena

@paulocarinhena paulocarinhena commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

The problem

Router.route(lang=...) extracts the primary language by splitting on - only:

key = "english" if str(lang).lower().split("-")[0] in ("en", "eng", "english") else "multilingual"

That works for BCP 47 tags (en-US). It does not work for the POSIX / Django / Babel spelling with _, which is what a locale looks like when it comes from the LANG environment variable, locale.getlocale(), Babel's str(Locale), Django's to_locale() / locale directory names, or a stored user profile:

router.route(state, lang="en-US").model        # -> english       ✓
router.route(state, lang="en_US").model        # -> multilingual  ✗
router.route(state, lang="EN_GB").model        # -> multilingual  ✗
router.route(state, lang="en_US.UTF-8").model  # -> multilingual  ✗
router.route(state, lang=" en ").model         # -> multilingual  ✗

lang= is the explicit override, and the README recommends passing it when the caller already knows the language. When the caller says "this is English", the router sends it to the checkpoint that is weaker on English (MASSIVE intent 0.657 vs 0.783) and the reason reports explicit lang='en_US', so nothing looks wrong.

What this PR changes

One line in Router.route. After stripping whitespace, it takes the primary subtag by splitting on -, _, . or @:

primary = re.split(r"[-_.@]", str(lang).strip().lower())[0]

Nothing else changes. Non-English locales in either spelling (pt_BR, pt-BR) still go to multilingual, and a code that merely starts with en (enm, Middle English) is still not treated as English.

Tests

tests/test_router.py adds cases to the existing explicit-routing table:

  • English: en_US, EN_GB, en_US.UTF-8 and " en " go to english, even on Hindi text, which proves the override wins over detection.
  • Not English: pt_BR, pt-BR and enm go to multilingual.
branch:  208 passed, 0 failed
main:    the 4 English-locale cases fail; the 3 non-English cases pass on both

`route(lang=...)` took the primary subtag by splitting on "-" only, which fits BCP 47 (`en-US`) but
not the POSIX/Django/Babel spelling (`en_US`, `en_US.UTF-8`) that a locale read from the
environment or a user profile usually has. `lang="en_US"` was therefore treated as a non-English
language and routed to the multilingual checkpoint, the opposite of what the caller asked for, as
was `lang=" en "`.

Split on `-`, `_`, `.` and `@`, after stripping whitespace.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks for this. #211 landed today and its _english_from_code parses en_US, en_US.UTF-8 and en-US the same way, so this is covered. Thank you for the careful locale handling.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Closing as superseded: #211 landed with _english_from_code, which parses en_US, en_US.UTF-8 and en-US through the same path.

Your framing of why it mattered is the part worth keeping. Splitting on - only works for BCP 47, and the _ spelling is what a locale actually looks like when it arrives from LANG, locale.getlocale(), Babel's str(Locale) or Django's to_locale(). Those are the ordinary sources for this argument in a real application, so the parameter worked for the hand-typed case and failed for the programmatic one.

Verified still correct on today's main: Router().route("hello", lang="en_US") picks the english checkpoint.

Thanks for the careful locale handling, and for the run of Portuguese routing work around it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants