Repository navigation
fix(email): From: starting a sentence deleted the request - #371
Merged
NandhaKishorM merged 1 commit intoSep 24, 2026
Merged
Conversation
The English quote-header marker was `^\s*From:\s.+$`, which matches any line that starts `From: `. Ordinary prose opens that way, so the request was deleted and the model answered an empty body — the same silent failure this module has been fixed for twice already (NandhaKishorM#94, NandhaKishorM#227): ```python from laya import email body = ("Hi team,\n" "From: my side the integration works, but please refund the duplicate charge today.\n" "Thanks") print(email.clean_email_body(body)) # before # 'Hi team,' # -> "refund" in cleaned is False; the model never sees the request # after # 'Hi team,\nFrom: my side the integration works, but please refund the duplicate charge today.\nThanks' ``` The `De:` marker two lines below already had this exact fix, for the same reason and with a comment saying so — `De:` also opens ordinary Portuguese/Spanish lines. `From:` opens ordinary English lines, and a reply header always carries the sender, so it is now gated on an address the same way: ```python re.compile(r"^\s*From:\s.*[@<]", re.I), # was r"^\s*From:\s.+$" ``` Deliberate choices tested: - **Gate on an address rather than on the shape of the text.** A word-count or capitalisation rule would be guessing at what a sentence looks like; "a reply header names its sender" is a property of the header, and it is the rule `De:` already uses, so the two now read the same way instead of one being stricter for no stated reason. - **A bare `From: Name` header is still cut, by its neighbours.** This is the part I got wrong first: I assumed `_SIGNATURE_MARKERS` covered `Sent:`, but `_HEADER_NEXT` only listed the Portuguese/Spanish `Enviado:`/`Data:`/`Fecha:` forms, so dropping the address from the marker leaked every English quote header written without one. `_HEADER_FROM_NAME`/`_HEADER_NEXT` now accept the English pair too — `From: <name>` followed by `Sent:` or a dated `Date:` — which is the translation of the rule that was already there for `De:`. - **`To:` alone is still not enough**, matching the existing note that "`De: 10/09 / Para: 15/09` is how a leave request reads". `From: Alice Smith` followed by `To: Bob` and a request is kept, because a `To:` line is not evidence of a quote block. - **When it cannot be told apart, the request wins.** A bare `From: Name` with no header neighbour is kept. A leaked quote is noise the model can read past; a deleted request is a wrong answer, which is the asymmetry NandhaKishorM#94 and NandhaKishorM#227 were both about. The suite had **no `From:` case at all** before this, which is why it stayed green on the bug — `python tests/test_email.py` reported `90 passed, 0 failed` on `main` with the bug present. The new block covers five prose forms, the four quoted-header forms that must still cut, and the two ambiguous forms that must be kept: ``` # with the marker reverted to r"^\s*From:\s.+$" 98 passed, 7 failed FAIL en/`From:` prose keeps the request: the reported case Hi team, FAIL en/`From:` prose keeps the request: lowercase Hi, FAIL en/`From:` prose keeps the request: uppercase Hi, FAIL en/`From:` prose keeps the request: extra spacing Hi, FAIL en/`From:` prose keeps the request: mid-body Hello, FAIL en/bare `From:` with no header neighbour is kept: From: Name then To: Hi, FAIL en/bare `From:` with no header neighbour is kept: From: Name alone Hi, # restored 105 passed, 0 failed ``` The four quoted-header positive controls stay green in both states, so the fix is not simply "stop cutting on `From:`". ``` $ python tests/test_email.py 105 passed, 0 failed (90 before) $ python tests/test_lang_guess.py 52 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_criteria.py 128 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` Known limitation: `From:` followed by an address but no other header line is treated as a quote header and cut, because that is what an address on a `From:` line overwhelmingly is in an email body. A caller who writes prose that literally begins `From: <someone@example.com>` loses that line; I could not construct a realistic case where that is a request, and the quoted-header case is the common one.
PerryLink
force-pushed
the
fix/email-from-quote-marker
branch
from
September 24, 2026 10:35
d54e5ed to
6ffd496
Compare
This was referenced Sep 24, 2026
Owner
|
Thanks for such a thorough fix. A request that starts with 'From:' no longer vanishes, and bare Outlook-style English headers are still cut because the English Sent: and Date: neighbours were added alongside the existing De: rule. The positive and negative cases in the tests make both directions clear. Merging, and I will close #346 in its favour. |
PerryLink
added a commit
to PerryLink/laya
that referenced
this pull request
Sep 24, 2026
The TypeScript port kept the pre-NandhaKishorM#338 form of all three header patterns, so it deleted the rest of a request whose body contained a line starting `From: `. - `QUOTE_HEADERS` matched `/^\s*From:\s.+$/i` where Python requires an address (`/^\s*From:\s.*[@<]/i`), so a prose line — "From: my side the integration works, but the downstream job still times out." — cut the message there. - `HEADER_FROM_NAME` was `/^\s*De:\s+\S/i` and `HEADER_NEXT` had no `Sent:` and no `Date:`, so the English half of the pair was missing: a bare "From: Maria Souza" header was never recognised even with its own `Sent:` line beside it. Both now mirror `laya/email.py:46-47`. `NandhaKishorM#371` changed exactly these three lines in `laya/email.py` and `tests/test_email.py` and merged on 2026-09-24; the port was not updated with it. `laya-ts/tests/` had no `From:` case at all, which is why this stayed green — the same reason the Python suite did, as NandhaKishorM#371's own description notes. Verified against Python rather than by eye. Every expectation in the new tests was produced by running `laya.clean_email_body` on the same input first: ``` Python result From prose -> the body, unchanged From addr -> 'Please refund my order' From+Sent -> 'Please refund order 123.' From bare -> the body, unchanged De+Date -> 'Please refund order 123.' ``` Before the change the same five cases give, in the copy at `laya-ts/`: ``` F1 keeps the request (Python: clean_email_body) AssertionError: expected 'Hi team, the export failed again this…' to be 'Hi team, the export failed again this…' - From: my side the integration works, but the downstream job still times out. - Could you take a look before Friday? F4 cuts 'De: Maria Souza' + 'Sent:' (Python does) + De: Maria Souza + Sent: Monday ``` After it, all five pass and the suite goes 253 → 258: ``` $ npx tsc -p tsconfig.json $ npx vitest run --reporter=basic Test Files 16 passed (16) Tests 258 passed (258) ``` Deliberate choices tested: - **The regexes are copied from Python, not re-derived.** These three lines have a history on the Python side — `NandhaKishorM#338`, `NandhaKishorM#346`, `NandhaKishorM#371` — and the port's job is to match the reference, so the fix is the reference's text plus a comment saying why each part is there. A re-derived equivalent would be a second place for the two sides to drift. - **The comments are ported too.** The Python source explains why `From:` needs an address and why a bare `From:` needs its neighbours; without that, the next person tightening the pattern has no way to know which half is load-bearing. - **One PR for both patterns.** They are the same rule seen from two sides — the marker that cuts on an address, and the neighbour rule that cuts on a header line — and `NandhaKishorM#371` changed them in one commit for the same reason. - **No other port gaps touched.** The audit that found this also found divergences in `lang.ts` and `common.ts`; they are separate PRs, so this diff is three lines of source and the tests for them. Known limitation: this is `laya-ts` only. The `SIGNOFF_TAIL` class in `email.ts:25` is deliberately not equivalent to Python's `[^\W\d_a-zß-öø-ÿ]` — Python's class admits lowercase letters outside Latin-1, so Python cuts a sign-off like `Regards, āā` and the port keeps it. The port matches the Python comment's stated intent, so that divergence is a bug in `laya/email.py`, not here, and is not addressed by this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(email):
From:starting a sentence deleted the requestThe English quote-header marker was
^\s*From:\s.+$, which matches any line that startsFrom:. Ordinary prose opens that way, so the request was deleted and the model answered anempty body — the same silent failure this module has been fixed for twice already (#94, #227):
The
De:marker two lines below already had this exact fix, for the same reason and with acomment saying so —
De:also opens ordinary Portuguese/Spanish lines.From:opens ordinaryEnglish lines, and a reply header always carries the sender, so it is now gated on an address
the same way:
Deliberate choices tested:
De:already uses, so the two now read the same way instead of one being stricter for no stated reason.From: Nameheader is still cut, by its neighbours. This is the part I got wrong first: I assumed_SIGNATURE_MARKERScoveredSent:, but_HEADER_NEXTonly listed the Portuguese/SpanishEnviado:/Data:/Fecha:forms, so dropping the address from the marker leaked every English quote header written without one._HEADER_FROM_NAME/_HEADER_NEXTnow accept the English pair too —From: <name>followed bySent:or a datedDate:— which is the translation of the rule that was already there forDe:.To:alone is still not enough, matching the existing note that "De: 10/09 / Para: 15/09is how a leave request reads".From: Alice Smithfollowed byTo: Boband a request is kept, because aTo:line is not evidence of a quote block.From: Namewith no header neighbour is kept. A leaked quote is noise the model can read past; a deleted request is a wrong answer, which is the asymmetry fix(email): keep the request when a disclaimer footer shares its paragraph #94 and fix(email): the word confidential is not a disclaimer #227 were both about.The suite had no
From:case at all before this, which is why it stayed green on the bug —python tests/test_email.pyreported90 passed, 0 failedonmainwith the bug present. Thenew block covers five prose forms, the four quoted-header forms that must still cut, and the two
ambiguous forms that must be kept:
The four quoted-header positive controls stay green in both states, so the fix is not simply
"stop cutting on
From:".Known limitation:
From:followed by an address but no other header line is treated as a quoteheader and cut, because that is what an address on a
From:line overwhelmingly is in an emailbody. A caller who writes prose that literally begins
From: <someone@example.com>loses thatline; I could not construct a realistic case where that is a request, and the quoted-header case
is the common one.
Relationship to #346, which opened first. That PR fixes the same issue with the same one-line regex change, and its diagnosis is the same as mine. Two differences, both reproduced by checking that branch out at
a48bab4and running it:check()ischeck(name, got, want)and the added case passes(body, expected, name), so the name lands inwantand the assertion compares the cleaned body against the string"en/From: prose is kept".python tests/test_email.pyon that branch reports90 passed, 1 failed. The production change does fix the reported case; the test pinning it does not pass. That PR also shows no CI runs, so nothing else catches it.From:header with no address in it. The marker is the only rule that used to cutFrom: Name, because_HEADER_FROM_NAME/_HEADER_NEXTonly know the Portuguese and Spanish forms. Gating the marker on an address therefore stops cutting every English quote header written without one:From:(the reported bug)From:header with an addressFrom: NamethenSent:From: NamethenDate:This PR is that one-line change plus the two
_HEADER_*edits that make the English bare header work the wayDe:already does, and the test cases for both directions. I have left a review on #346 with the reproductions rather than asking anyone to take my word for it, and I am happy to close this one if #346 is preferred — the change it needs is in the review. I opened it before I found #346, which is why both exist.