Skip to content

fix(email): the name class excludes lowercase in every script - #503

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/email-signoff-name-case
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/email-signoff-name-case

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

A closing line is the closing word plus at most a name, and the name's first letter must not be lowercase — capitalised in any script (Regards, Łukasz) or caseless (Regards, 山田). The class that excludes the lowercase letters stops at Latin-1, though: a name starting with ł, ż, д or α is lowercase, yet it read as capitalised, the line counted as a sign-off, and the cut dropped the body after it:

$ python -c "from laya.email import clean_email_body as c; print(c('Hi,\n\nPlease refund invoice 4411.\nThanks, żaneta\nCould you confirm by Friday?'))"
Hi,

Please refund invoice 4411.

Root cause

laya/email.py:60 matched name tokens with [^\W\d_a-zß-öø-ÿ][\w'-]* — "excludes the lowercase letters", as #245's comment puts it, but only where a-z, ß-ö and ø-ÿ reach; every other script's lowercase passes the class. The port (laya-ts/src/email.ts:25, [\p{Lu}\p{Lt}\p{Lo}]) asks the case question per character, and the port is the side that matches #245's stated rule ("a name is capitalised in any script"), so Python moves to it.

What changes

The sign-off marker is now the callable _is_english_signoff in _SIGNATURE_MARKERS: the tail is matched structurally, and each token's first character is judged by Unicode category (Lu/Lt/Lo, stdlib unicodedata, no new dependency) — the same question the port's \p{Lu}/\p{Lt}/\p{Lo} asks. Combining marks ride along with the letter before them (Jose\u0301 spells José).

Fifteen lines change meaning; all in the safe direction — kept where they were cut (the dropped-request class #132 fixed, one script over), and cut where a decomposed name now counts:

line on main now (= port)
Thanks, łukasz cut kept
Thanks, žaneta cut kept
Best regards, дмитрий cut kept
Thanks, αλέξανδρος cut kept
Thanks, ʻŌlelo cut kept
Thanks, Ⅷ cut kept
Thanks, ıbrahim cut kept
Thanks, ярослав cut kept
Thanks, ο Κώστας cut kept
Thanks, Jose + U+0301 kept cut
Thanks, Jose + U+0301 + a kept cut
Thanks, A + U+0301 kept cut
Thanks, œ cut kept
Thanks, 𐐨 cut kept
Thanks, āna cut kept

"cut" = the line is read as the start of a signature and everything after it is dropped. ʻŌlelo starts with a modifier letter and Ⅷ is a numeral — neither is Lu/Lt/Lo; ı, 𐐨 (Deseret) are lowercase outside ASCII. On a 58-case corpus (every script and edge class tried) the two sides now disagree on nothing; these 15 are the rows where main differed from the port. I ran the shipped function and the port's regexes over the same list to check this.

Test

tests/test_email.py gains the cases (four kept, one cut), each failing on main with its own name:

$ python tests/test_email.py      # with the new cases, before the fix
105 passed, 5 failed
  FAIL signoff kept/latin-extended lowercase name: got 'Hi,\n\nPlease refund invoice 4411.' want '...\nThanks, żaneta'
$ python tests/test_email.py      # after
110 passed, 0 failed

ruff check (E9,F63,F7,F82,F401,F811) and compileall clean; the adjacent suites (serve, router, presets, truncation, audit) pass unchanged.

Deliberate choices tested

  • Categories, not a predicate. token[0].isalpha() and not token[0].islower() accepts modifier letters (Thanks, ʻŌlelo), which the port rejects; the category check agrees with the port on the whole corpus, the predicate does not.
  • A callable marker, not a wider class. The alternative — extend the excluded ranges to more scripts — is the same closed-ended approach that produced this bug. The English sign-off joins _SIGNATURE_MARKERS as a callable (the port's marker list is callables too), and _marker_matches dispatches regex and callable for the one consumer (clean_email_body).
  • The port is not changed to match Python. The port's per-character question is the intent both comments state; making it mirror the class would keep the bug and touch a side that already behaves correctly.

Known limitation

A mark from another block (a Devanagari matra, say) still ends the token in Python where the port's \p{M} carries it — \u0300-\u036f covers the Latin decompositions. The port needs no code change; its comment no longer quotes the old class. Related: #422 touches other hunks of laya-ts/src/email.ts; no line overlap.

`[^\W\d_a-zß-öø-ÿ]` excludes the lowercase letters only where Latin-1 spells them, so a
name starting with ł, ż, д or α counted as capitalised-in-any-script, the line was cut as
a sign-off, and whatever followed it went with the cut:

$ python -c "from laya.email import clean_email_body as c; print(c('Hi,\n\nPlease refund invoice 4411.\nThanks, żaneta\nCould you confirm by Friday?'))"
Hi,

Please refund invoice 4411.

The TS port asks the case question per character instead (`[\p{Lu}\p{Lt}\p{Lo}]`), which is
the rule NandhaKishorM#245's comment describes — "a name is capitalised in any script" — so Python moves
to it: the tail is matched structurally, and each token's first character is judged by
Unicode category (Lu/Lt/Lo). A combining mark rides along with the letter before it
(`Jose\u0301` spells `José`), which `\w` alone does not cover.

$ python tests/test_email.py      # before the fix, with the new cases
105 passed, 5 failed
$ python tests/test_email.py      # after
110 passed, 0 failed

Deliberate choices tested:

- **Categories, not a predicate.** `token[0].isalpha() and not token[0].islower()` accepts
  modifier letters (`Thanks, ʻŌlelo`), which the port rejects; the category check agrees
  with the port on the whole 58-case corpus, the predicate does not.
- **A callable marker, not a wider class.** `re` cannot say "not a lowercase letter in any
  script"; listing ranges is what failed. The English sign-off joins `_SIGNATURE_MARKERS` as
  a callable (the port's list is callables too), and `_marker_matches` dispatches it.
- **Marks via `\u0300-\u036f`.** The combining diacriticals are the block a decomposed
  Latin-script name actually carries; other mark blocks are not covered, matching `\w`.

Known limitation: a mark from another block (a Devanagari matra, say) still ends the token
in Python where the port's `\p{M}` carries it. The port needs no code change, only its
comment, which quoted the old class. Related: NandhaKishorM#422 touches other hunks of
`laya-ts/src/email.ts`; no line overlap.
@aashish254

aashish254 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Short answer: the per-character framing is the right fix, and it makes the divergence detectable instead of implicit. One parity gap remains, measured on this branch: the continuation class [\w\u0300-\u036f'-] covers only the Latin combining range, while the port's [\p{L}\p{M}\p{N}_'-] covers every mark. Names written with non-Latin combining marks (nearly every Indic script) still split the two implementations. The comment says "as \p{M} allows in the port", but U+0300–U+036F is about 5% of \p{M}.

Running a name through the tail matcher: Jose\u0301 matches (the tested case); मुकेश doesn't, and முகமது doesn't either — TS says True for both, because the virama and matras are Mn outside U+036F.

So Regards, मुकेश cuts the body in laya-ts but keeps it in Python — the same class of bug as the żaneta one this PR fixes, just relocated from the first letter to the continuation letters. The first letters pass fine (मु is Lo); it's the marks mid-token that break the match.

Since _is_english_signoff already imports unicodedata, the small fix is to judge continuation characters by category too rather than by range — a helper like unicodedata.category(ch)[0] in "LMN" or ch in "_'-" used by a structural token scan, or at minimum extend the class toward full \p{M}. Either way a Devanagari/Tamil name in the parity table (check("signoff kept/…") on both sides) would pin it, since the current tests only reach decomposed Latin.

Not blocking the approach — flagging it because parity is the whole point of the callable move.

@NandhaKishorM
NandhaKishorM merged commit b7a2d11 into NandhaKishorM:main Sep 27, 2026
22 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @PerryLink. Merged, thanks for keeping the Python and TS sign-off rules on the same definition. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants