Skip to content

fix(email): From: starting a sentence deleted the request - #371

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/email-from-quote-marker
Sep 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/email-from-quote-marker

Conversation

@PerryLink

@PerryLink PerryLink commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

fix(email): From: starting a sentence deleted the request

The English quote-header marker was ^\s*From:\s.+$, which matches any line that starts
From: . Ordinary prose opens that way, so the request was deleted and the model answered an
empty body — the same silent failure this module has been fixed for twice already (#94, #227):

from laya import email

body = ("Hi team,\n"
        "From: my side the integration works, but please refund the duplicate charge today.\n"
        "Thanks")
print(email.clean_email_body(body))

# before
# 'Hi team,'
#   -> "refund" in cleaned is False; the model never sees the request

# after
# 'Hi team,\nFrom: my side the integration works, but please refund the duplicate charge today.\nThanks'

The De: marker two lines below already had this exact fix, for the same reason and with a
comment saying so — De: also opens ordinary Portuguese/Spanish lines. From: opens ordinary
English lines, and a reply header always carries the sender, so it is now gated on an address
the same way:

re.compile(r"^\s*From:\s.*[@<]", re.I),   # was r"^\s*From:\s.+$"

Deliberate choices tested:

  • Gate on an address rather than on the shape of the text. A word-count or capitalisation rule would be guessing at what a sentence looks like; "a reply header names its sender" is a property of the header, and it is the rule De: already uses, so the two now read the same way instead of one being stricter for no stated reason.
  • A bare From: Name header is still cut, by its neighbours. This is the part I got wrong first: I assumed _SIGNATURE_MARKERS covered Sent:, but _HEADER_NEXT only listed the Portuguese/Spanish Enviado:/Data:/Fecha: forms, so dropping the address from the marker leaked every English quote header written without one. _HEADER_FROM_NAME/_HEADER_NEXT now accept the English pair too — From: <name> followed by Sent: or a dated Date: — which is the translation of the rule that was already there for De:.
  • To: alone is still not enough, matching the existing note that "De: 10/09 / Para: 15/09 is how a leave request reads". From: Alice Smith followed by To: Bob and a request is kept, because a To: line is not evidence of a quote block.
  • When it cannot be told apart, the request wins. A bare From: Name with no header neighbour is kept. A leaked quote is noise the model can read past; a deleted request is a wrong answer, which is the asymmetry fix(email): keep the request when a disclaimer footer shares its paragraph #94 and fix(email): the word confidential is not a disclaimer #227 were both about.

The suite had no From: case at all before this, which is why it stayed green on the bug —
python tests/test_email.py reported 90 passed, 0 failed on main with the bug present. The
new block covers five prose forms, the four quoted-header forms that must still cut, and the two
ambiguous forms that must be kept:

# with the marker reverted to r"^\s*From:\s.+$"
98 passed, 7 failed
  FAIL en/`From:` prose keeps the request: the reported case Hi team,
  FAIL en/`From:` prose keeps the request: lowercase Hi,
  FAIL en/`From:` prose keeps the request: uppercase Hi,
  FAIL en/`From:` prose keeps the request: extra spacing Hi,
  FAIL en/`From:` prose keeps the request: mid-body Hello,
  FAIL en/bare `From:` with no header neighbour is kept: From: Name then To: Hi,
  FAIL en/bare `From:` with no header neighbour is kept: From: Name alone Hi,

# restored
105 passed, 0 failed

The four quoted-header positive controls stay green in both states, so the fix is not simply
"stop cutting on From:".

$ python tests/test_email.py                  105 passed, 0 failed   (90 before)
$ python tests/test_lang_guess.py             52 passed, 0 failed
$ python tests/test_router.py                 all routing tests passed
$ python tests/test_criteria.py               128 passed, 0 failed
$ python tests/test_packaging.py              all packaging tests passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/

Known limitation: From: followed by an address but no other header line is treated as a quote
header and cut, because that is what an address on a From: line overwhelmingly is in an email
body. A caller who writes prose that literally begins From: <someone@example.com> loses that
line; I could not construct a realistic case where that is a request, and the quoted-header case
is the common one.


Relationship to #346, which opened first. That PR fixes the same issue with the same one-line regex change, and its diagnosis is the same as mine. Two differences, both reproduced by checking that branch out at a48bab4 and running it:

  1. Its new test fails. check() is check(name, got, want) and the added case passes (body, expected, name), so the name lands in want and the assertion compares the cleaned body against the string "en/From: prose is kept". python tests/test_email.py on that branch reports 90 passed, 1 failed. The production change does fix the reported case; the test pinning it does not pass. That PR also shows no CI runs, so nothing else catches it.
  2. It does not cover a From: header with no address in it. The marker is the only rule that used to cut From: Name, because _HEADER_FROM_NAME/_HEADER_NEXT only know the Portuguese and Spanish forms. Gating the marker on an address therefore stops cutting every English quote header written without one:
case #346 this PR
prose starting From: (the reported bug) kept kept
From: header with an address cut cut
bare From: Name then Sent: kept — whole quoted block leaks cut
bare From: Name then Date: kept — whole quoted block leaks cut

This PR is that one-line change plus the two _HEADER_* edits that make the English bare header work the way De: already does, and the test cases for both directions. I have left a review on #346 with the reproductions rather than asking anyone to take my word for it, and I am happy to close this one if #346 is preferred — the change it needs is in the review. I opened it before I found #346, which is why both exist.

The English quote-header marker was `^\s*From:\s.+$`, which matches any line that starts
`From: `. Ordinary prose opens that way, so the request was deleted and the model answered an
empty body — the same silent failure this module has been fixed for twice already (NandhaKishorM#94, NandhaKishorM#227):

```python
from laya import email

body = ("Hi team,\n"
        "From: my side the integration works, but please refund the duplicate charge today.\n"
        "Thanks")
print(email.clean_email_body(body))

# before
# 'Hi team,'
#   -> "refund" in cleaned is False; the model never sees the request

# after
# 'Hi team,\nFrom: my side the integration works, but please refund the duplicate charge today.\nThanks'
```

The `De:` marker two lines below already had this exact fix, for the same reason and with a
comment saying so — `De:` also opens ordinary Portuguese/Spanish lines. `From:` opens ordinary
English lines, and a reply header always carries the sender, so it is now gated on an address
the same way:

```python
re.compile(r"^\s*From:\s.*[@<]", re.I),   # was r"^\s*From:\s.+$"
```

Deliberate choices tested:

- **Gate on an address rather than on the shape of the text.** A word-count or capitalisation rule would be guessing at what a sentence looks like; "a reply header names its sender" is a property of the header, and it is the rule `De:` already uses, so the two now read the same way instead of one being stricter for no stated reason.
- **A bare `From: Name` header is still cut, by its neighbours.** This is the part I got wrong first: I assumed `_SIGNATURE_MARKERS` covered `Sent:`, but `_HEADER_NEXT` only listed the Portuguese/Spanish `Enviado:`/`Data:`/`Fecha:` forms, so dropping the address from the marker leaked every English quote header written without one. `_HEADER_FROM_NAME`/`_HEADER_NEXT` now accept the English pair too — `From: <name>` followed by `Sent:` or a dated `Date:` — which is the translation of the rule that was already there for `De:`.
- **`To:` alone is still not enough**, matching the existing note that "`De: 10/09 / Para: 15/09` is how a leave request reads". `From: Alice Smith` followed by `To: Bob` and a request is kept, because a `To:` line is not evidence of a quote block.
- **When it cannot be told apart, the request wins.** A bare `From: Name` with no header neighbour is kept. A leaked quote is noise the model can read past; a deleted request is a wrong answer, which is the asymmetry NandhaKishorM#94 and NandhaKishorM#227 were both about.

The suite had **no `From:` case at all** before this, which is why it stayed green on the bug —
`python tests/test_email.py` reported `90 passed, 0 failed` on `main` with the bug present. The
new block covers five prose forms, the four quoted-header forms that must still cut, and the two
ambiguous forms that must be kept:

```
# with the marker reverted to r"^\s*From:\s.+$"
98 passed, 7 failed
  FAIL en/`From:` prose keeps the request: the reported case Hi team,
  FAIL en/`From:` prose keeps the request: lowercase Hi,
  FAIL en/`From:` prose keeps the request: uppercase Hi,
  FAIL en/`From:` prose keeps the request: extra spacing Hi,
  FAIL en/`From:` prose keeps the request: mid-body Hello,
  FAIL en/bare `From:` with no header neighbour is kept: From: Name then To: Hi,
  FAIL en/bare `From:` with no header neighbour is kept: From: Name alone Hi,

# restored
105 passed, 0 failed
```

The four quoted-header positive controls stay green in both states, so the fix is not simply
"stop cutting on `From:`".

```
$ python tests/test_email.py                  105 passed, 0 failed   (90 before)
$ python tests/test_lang_guess.py             52 passed, 0 failed
$ python tests/test_router.py                 all routing tests passed
$ python tests/test_criteria.py               128 passed, 0 failed
$ python tests/test_packaging.py              all packaging tests passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: `From:` followed by an address but no other header line is treated as a quote
header and cut, because that is what an address on a `From:` line overwhelmingly is in an email
body. A caller who writes prose that literally begins `From: <someone@example.com>` loses that
line; I could not construct a realistic case where that is a request, and the quoted-header case
is the common one.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks for such a thorough fix. A request that starts with 'From:' no longer vanishes, and bare Outlook-style English headers are still cut because the English Sent: and Date: neighbours were added alongside the existing De: rule. The positive and negative cases in the tests make both directions clear. Merging, and I will close #346 in its favour.

@NandhaKishorM
NandhaKishorM merged commit 83b45f3 into NandhaKishorM:main Sep 24, 2026
21 checks passed
PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 24, 2026
The TypeScript port kept the pre-NandhaKishorM#338 form of all three header patterns, so it deleted the rest
of a request whose body contained a line starting `From: `.

- `QUOTE_HEADERS` matched `/^\s*From:\s.+$/i` where Python requires an address
  (`/^\s*From:\s.*[@<]/i`), so a prose line — "From: my side the integration works, but the
  downstream job still times out." — cut the message there.
- `HEADER_FROM_NAME` was `/^\s*De:\s+\S/i` and `HEADER_NEXT` had no `Sent:` and no `Date:`, so
  the English half of the pair was missing: a bare "From: Maria Souza" header was never
  recognised even with its own `Sent:` line beside it. Both now mirror `laya/email.py:46-47`.

`NandhaKishorM#371` changed exactly these three lines in `laya/email.py` and `tests/test_email.py` and merged
on 2026-09-24; the port was not updated with it. `laya-ts/tests/` had no `From:` case at all,
which is why this stayed green — the same reason the Python suite did, as NandhaKishorM#371's own description
notes.

Verified against Python rather than by eye. Every expectation in the new tests was produced by
running `laya.clean_email_body` on the same input first:

```
                         Python result
From prose   -> the body, unchanged
From addr    -> 'Please refund my order'
From+Sent    -> 'Please refund order 123.'
From bare    -> the body, unchanged
De+Date      -> 'Please refund order 123.'
```

Before the change the same five cases give, in the copy at `laya-ts/`:

```
F1 keeps the request (Python: clean_email_body)
AssertionError: expected 'Hi team, the export failed again this…' to be 'Hi team, the export failed again this…'
  - From: my side the integration works, but the downstream job still times out.
  - Could you take a look before Friday?

F4 cuts 'De: Maria Souza' + 'Sent:' (Python does)
  + De: Maria Souza
  + Sent: Monday
```

After it, all five pass and the suite goes 253 → 258:

```
$ npx tsc -p tsconfig.json
$ npx vitest run --reporter=basic
 Test Files  16 passed (16)
      Tests  258 passed (258)
```

Deliberate choices tested:

- **The regexes are copied from Python, not re-derived.** These three lines have a history on the Python side — `NandhaKishorM#338`, `NandhaKishorM#346`, `NandhaKishorM#371` — and the port's job is to match the reference, so the fix is the reference's text plus a comment saying why each part is there. A re-derived equivalent would be a second place for the two sides to drift.
- **The comments are ported too.** The Python source explains why `From:` needs an address and why a bare `From:` needs its neighbours; without that, the next person tightening the pattern has no way to know which half is load-bearing.
- **One PR for both patterns.** They are the same rule seen from two sides — the marker that cuts on an address, and the neighbour rule that cuts on a header line — and `NandhaKishorM#371` changed them in one commit for the same reason.
- **No other port gaps touched.** The audit that found this also found divergences in `lang.ts` and `common.ts`; they are separate PRs, so this diff is three lines of source and the tests for them.

Known limitation: this is `laya-ts` only. The `SIGNOFF_TAIL` class in `email.ts:25` is deliberately
not equivalent to Python's `[^\W\d_a-zß-öø-ÿ]` — Python's class admits lowercase letters outside
Latin-1, so Python cuts a sign-off like `Regards, āā` and the port keeps it. The port matches the
Python comment's stated intent, so that divergence is a bug in `laya/email.py`, not here, and is
not addressed by this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants