Repository navigation
fix(email): the name class excludes lowercase in every script - #503
Conversation
`[^\W\d_a-zß-öø-ÿ]` excludes the lowercase letters only where Latin-1 spells them, so a
name starting with ł, ż, д or α counted as capitalised-in-any-script, the line was cut as
a sign-off, and whatever followed it went with the cut:
$ python -c "from laya.email import clean_email_body as c; print(c('Hi,\n\nPlease refund invoice 4411.\nThanks, żaneta\nCould you confirm by Friday?'))"
Hi,
Please refund invoice 4411.
The TS port asks the case question per character instead (`[\p{Lu}\p{Lt}\p{Lo}]`), which is
the rule NandhaKishorM#245's comment describes — "a name is capitalised in any script" — so Python moves
to it: the tail is matched structurally, and each token's first character is judged by
Unicode category (Lu/Lt/Lo). A combining mark rides along with the letter before it
(`Jose\u0301` spells `José`), which `\w` alone does not cover.
$ python tests/test_email.py # before the fix, with the new cases
105 passed, 5 failed
$ python tests/test_email.py # after
110 passed, 0 failed
Deliberate choices tested:
- **Categories, not a predicate.** `token[0].isalpha() and not token[0].islower()` accepts
modifier letters (`Thanks, ʻŌlelo`), which the port rejects; the category check agrees
with the port on the whole 58-case corpus, the predicate does not.
- **A callable marker, not a wider class.** `re` cannot say "not a lowercase letter in any
script"; listing ranges is what failed. The English sign-off joins `_SIGNATURE_MARKERS` as
a callable (the port's list is callables too), and `_marker_matches` dispatches it.
- **Marks via `\u0300-\u036f`.** The combining diacriticals are the block a decomposed
Latin-script name actually carries; other mark blocks are not covered, matching `\w`.
Known limitation: a mark from another block (a Devanagari matra, say) still ends the token
in Python where the port's `\p{M}` carries it. The port needs no code change, only its
comment, which quoted the old class. Related: NandhaKishorM#422 touches other hunks of
`laya-ts/src/email.ts`; no line overlap.
|
Short answer: the per-character framing is the right fix, and it makes the divergence detectable instead of implicit. One parity gap remains, measured on this branch: the continuation class Running a name through the tail matcher: So Since Not blocking the approach — flagging it because parity is the whole point of the callable move. |
|
Thanks @PerryLink. Merged, thanks for keeping the Python and TS sign-off rules on the same definition. It ships in 0.3.21. |
A closing line is the closing word plus at most a name, and the name's first letter must not be lowercase — capitalised in any script (
Regards, Łukasz) or caseless (Regards, 山田). The class that excludes the lowercase letters stops at Latin-1, though: a name starting with ł, ż, д or α is lowercase, yet it read as capitalised, the line counted as a sign-off, and the cut dropped the body after it:Root cause
laya/email.py:60matched name tokens with[^\W\d_a-zß-öø-ÿ][\w'-]*— "excludes the lowercase letters", as #245's comment puts it, but only wherea-z,ß-öandø-ÿreach; every other script's lowercase passes the class. The port (laya-ts/src/email.ts:25,[\p{Lu}\p{Lt}\p{Lo}]) asks the case question per character, and the port is the side that matches #245's stated rule ("a name is capitalised in any script"), so Python moves to it.What changes
The sign-off marker is now the callable
_is_english_signoffin_SIGNATURE_MARKERS: the tail is matched structurally, and each token's first character is judged by Unicode category (Lu/Lt/Lo, stdlibunicodedata, no new dependency) — the same question the port's\p{Lu}/\p{Lt}/\p{Lo}asks. Combining marks ride along with the letter before them (Jose\u0301spellsJosé).Fifteen lines change meaning; all in the safe direction — kept where they were cut (the dropped-request class #132 fixed, one script over), and cut where a decomposed name now counts:
Thanks, łukaszThanks, žanetaBest regards, дмитрийThanks, αλέξανδροςThanks, ʻŌleloThanks, ⅧThanks, ıbrahimThanks, ярославThanks, ο ΚώσταςThanks, Jose+ U+0301Thanks, Jose+ U+0301 +aThanks, A+ U+0301Thanks, œThanks, 𐐨Thanks, āna"cut" = the line is read as the start of a signature and everything after it is dropped.
ʻŌlelostarts with a modifier letter andⅧis a numeral — neither isLu/Lt/Lo;ı,𐐨(Deseret) are lowercase outside ASCII. On a 58-case corpus (every script and edge class tried) the two sides now disagree on nothing; these 15 are the rows where main differed from the port. I ran the shipped function and the port's regexes over the same list to check this.Test
tests/test_email.pygains the cases (four kept, one cut), each failing on main with its own name:ruff check(E9,F63,F7,F82,F401,F811) andcompileallclean; the adjacent suites (serve, router, presets, truncation, audit) pass unchanged.Deliberate choices tested
token[0].isalpha() and not token[0].islower()accepts modifier letters (Thanks, ʻŌlelo), which the port rejects; the category check agrees with the port on the whole corpus, the predicate does not._SIGNATURE_MARKERSas a callable (the port's marker list is callables too), and_marker_matchesdispatches regex and callable for the one consumer (clean_email_body).Known limitation
A mark from another block (a Devanagari matra, say) still ends the token in Python where the port's
\p{M}carries it —\u0300-\u036fcovers the Latin decompositions. The port needs no code change; its comment no longer quotes the old class. Related: #422 touches other hunks oflaya-ts/src/email.ts; no line overlap.