Skip to content

Fix overly broad From header detection - #346

Closed
tyagivibhor111-hub wants to merge 1 commit into
NandhaKishorM:mainfrom
tyagivibhor111-hub:fix-from-header-marker
Closed

tyagivibhor111-hub wants to merge 1 commit into
NandhaKishorM:mainfrom
tyagivibhor111-hub:fix-from-header-marker

Conversation

@tyagivibhor111-hub

Copy link
Copy Markdown

What

Fixes Issue #338.

The English From: quote-header pattern was matching every line that started with From:, including normal prose such as:

From: my side the integration works, but please refund the duplicate charge today.

This caused clean_email_body() to treat the line as the start of a quoted email and remove the request from the cleaned body.

Why

A normal sentence beginning with From: should not be treated as a quoted email header.

This change makes the English From: detection require an email-address-like @ or < token, while keeping real From: headers working.

How it was verified

Ran:

python tests/test_router.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120

Also verified the regression case from #338 and confirmed that the cleaned output keeps the request, including the word refund.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

@PerryLink

Copy link
Copy Markdown
Contributor

I checked this out locally and ran it, because I had opened a PR for the same issue a few hours after you and wanted to compare rather than assume mine was better. Two things to fix before this can merge, both reproduced on your branch at a48bab4.

1. The new test fails, because its arguments are in the wrong order.

tests/test_email.py defines check(name, got, want), and the added case passes them as check(body, expected, name):

$ git checkout a48bab4 && python tests/test_email.py
90 passed, 1 failed
  FAIL Hi team
From: my side the integration works, but please refund the duplicate charge today.:
     got  'Hi team\nFrom: my side the integration works, but please refund the duplicate charge today.'
     want 'en/From: prose is kept'

The name lands in want, so the assertion compares the cleaned body against the string "en/From: prose is kept" and can never pass. The regression is real — the production change does fix the reported case — but the test that is supposed to pin it is red, and this PR has no CI runs on it, so nothing else will catch that. Swapping the arguments is enough:

check(
    "en/From: prose is kept",
    clean_email_body(...),
    "Hi team\nFrom: my side the integration works, but please refund the duplicate charge today.",
)

2. The one-line regex change loses every From: quote header that has no address in it.

^\s*From:\s.*[@<] is the right rule for the reported bug, and it is the same change I made first. But From: Name with no address is a normal Outlook header too, and the marker is now the only thing that used to cut it — the neighbouring-header rule, _HEADER_FROM_NAME/_HEADER_NEXT, only knows the Portuguese and Spanish forms (De: … Enviado:/Data:/Fecha:), so an English header with no address is no longer recognised at all. Reproduced on your branch:

prose (the reported bug)       OK    out='Hi team,\nFrom: my side the integration works, but please refun'
quoted header WITH address     OK    out='Hi,\nSee below.'
bare From: Name then Sent:     WRONG out='Hi,\nSee below.\n\nFrom: Alice Smith\nSent: Monday, 21 Sep 2026\n\nP'
bare From: Name then Date:     WRONG out='Hi,\nSee below.\n\nFrom: Alice Smith\nDate: 21/09/2026\n\nPlease ref'

The whole quoted block — From: Alice Smith, Sent:, and the quoted request under it — now stays in the body. For an email triage model that is the failure this module has been fixed for twice already (#94, #227): the model reads an old quoted request as part of the current one.

I hit exactly this when I made the same one-line change, and the fix is two small edits, which are the only difference between your branch and mine:

_HEADER_FROM_NAME = re.compile(r"^\s*(De|From):\s+\S", re.I)
_HEADER_NEXT = re.compile(r"^\s*(Enviad[oa]( em| el)?:\s|Sent:\s|(Data|Fecha|Date):\s.*\d{4})", re.I)

That is the English translation of the rule that was already there for De:, and it keeps the two ambiguous cases on the conservative side: From: Name followed by To: alone is still kept, because a To: line is not evidence of a quote block. With it, all four cases above pass. My branch has the change plus the test cases for it if that is easier than writing them again — take them, and I will close my PR. Yours was first and I would rather see one good fix land than two.

One note for whoever reviews: this PR shows no CI runs at all, so the failing test is not visible in the checks. That is worth knowing generally, since a red suite is the only signal these email changes get.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for jumping on #338 so quickly. Your diagnosis was right, and gating From: on an address is the core of the fix. #371 landed the same change together with the extra rule that still cuts bare 'From: Name' plus 'Sent:' headers, so I am closing this one in its favour. I really appreciate the contribution and would be glad to see more from you.

PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 24, 2026
The TypeScript port kept the pre-NandhaKishorM#338 form of all three header patterns, so it deleted the rest
of a request whose body contained a line starting `From: `.

- `QUOTE_HEADERS` matched `/^\s*From:\s.+$/i` where Python requires an address
  (`/^\s*From:\s.*[@<]/i`), so a prose line — "From: my side the integration works, but the
  downstream job still times out." — cut the message there.
- `HEADER_FROM_NAME` was `/^\s*De:\s+\S/i` and `HEADER_NEXT` had no `Sent:` and no `Date:`, so
  the English half of the pair was missing: a bare "From: Maria Souza" header was never
  recognised even with its own `Sent:` line beside it. Both now mirror `laya/email.py:46-47`.

`NandhaKishorM#371` changed exactly these three lines in `laya/email.py` and `tests/test_email.py` and merged
on 2026-09-24; the port was not updated with it. `laya-ts/tests/` had no `From:` case at all,
which is why this stayed green — the same reason the Python suite did, as NandhaKishorM#371's own description
notes.

Verified against Python rather than by eye. Every expectation in the new tests was produced by
running `laya.clean_email_body` on the same input first:

```
                         Python result
From prose   -> the body, unchanged
From addr    -> 'Please refund my order'
From+Sent    -> 'Please refund order 123.'
From bare    -> the body, unchanged
De+Date      -> 'Please refund order 123.'
```

Before the change the same five cases give, in the copy at `laya-ts/`:

```
F1 keeps the request (Python: clean_email_body)
AssertionError: expected 'Hi team, the export failed again this…' to be 'Hi team, the export failed again this…'
  - From: my side the integration works, but the downstream job still times out.
  - Could you take a look before Friday?

F4 cuts 'De: Maria Souza' + 'Sent:' (Python does)
  + De: Maria Souza
  + Sent: Monday
```

After it, all five pass and the suite goes 253 → 258:

```
$ npx tsc -p tsconfig.json
$ npx vitest run --reporter=basic
 Test Files  16 passed (16)
      Tests  258 passed (258)
```

Deliberate choices tested:

- **The regexes are copied from Python, not re-derived.** These three lines have a history on the Python side — `NandhaKishorM#338`, `NandhaKishorM#346`, `NandhaKishorM#371` — and the port's job is to match the reference, so the fix is the reference's text plus a comment saying why each part is there. A re-derived equivalent would be a second place for the two sides to drift.
- **The comments are ported too.** The Python source explains why `From:` needs an address and why a bare `From:` needs its neighbours; without that, the next person tightening the pattern has no way to know which half is load-bearing.
- **One PR for both patterns.** They are the same rule seen from two sides — the marker that cuts on an address, and the neighbour rule that cuts on a header line — and `NandhaKishorM#371` changed them in one commit for the same reason.
- **No other port gaps touched.** The audit that found this also found divergences in `lang.ts` and `common.ts`; they are separate PRs, so this diff is three lines of source and the tests for them.

Known limitation: this is `laya-ts` only. The `SIGNOFF_TAIL` class in `email.ts:25` is deliberately
not equivalent to Python's `[^\W\d_a-zß-öø-ÿ]` — Python's class admits lowercase letters outside
Latin-1, so Python cuts a sign-off like `Regards, āā` and the port keeps it. The port matches the
Python comment's stated intent, so that divergence is a bug in `laya/email.py`, not here, and is
not addressed by this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants