Skip to content

fix(email): keep the request when a disclaimer footer shares its paragraph - #94

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
PerryLink:fix/email-disclaimer-paragraph
Sep 21, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
PerryLink:fix/email-disclaimer-paragraph

Conversation

@PerryLink

@PerryLink PerryLink commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #93

clean_email_body can return an empty string for a legitimate email. _DISCLAIMER was applied
to whole paragraphs and matches on the single word confidential, so a footer that runs on without
a blank line sits in the same paragraph as the sender's request and both were deleted together:

from laya.email import clean_email_body

clean_email_body(
    "My account is locked.\n"
    "This email is confidential and intended solely for the named addressee.\n"
    "Please unlock it."
)
# before: ''                                      <- the whole body, request included, is gone
# after:  'My account is locked. Please unlock it.'

The blank line is the only thing that matters. Adding one before the footer was already enough to
avoid the bug, which is why this needed a wrapped footer to surface.

Agent.predict / Router.predict then receives a state with an empty body and answers, with
confidence, about an email it can no longer see. Nothing warns.

Change

The keep-or-drop decision now happens inside a paragraph instead of over the whole thing: a
paragraph is dropped only when every sentence in it is boilerplate, and otherwise just the
boilerplate sentences go. A paragraph with no disclaimer match is returned untouched, so the line
structure of ordinary mail is unchanged.

Two details are deliberate, and I tested the alternatives:

  • Splitting on sentences, not newlines. A footer broken across lines is one sentence and still
    disappears whole. Splitting on newlines as well left its first line behind -- that variant
    regressed the wrapped-footer case, so it is not what is here.
  • Erring towards keeping text. Dropping the sender's request is silent and severe; leaving one
    boilerplate line behind is neither. Where the two goals conflict, this change keeps the text.

Nothing else moved

Measured on 42626c3 and on this branch, same inputs:

input before after
pure request, no footer 'My account is locked. Please unlock it.' unchanged
request, blank line, then footer 'My account is locked.' unchanged
request and footer in one paragraph '' 'My account is locked.'
quoted history (On ... wrote:, >) removed removed
signature (-- block) removed removed
footer standing alone '' ''

Known limitation

Sentence splitting needs a sentence boundary. Prose with no terminal punctuation directly followed
by the footer in the same paragraph still loses its first fragment:

clean_email_body(
    "My account is locked\n"                        # no full stop before the newline
    "This email is confidential and intended solely for the named addressee.\n"
    "Please unlock it."
)
# before: ''                    <- everything lost
# after:  'Please unlock it.'   <- partial recovery, first fragment still attached to the footer

It recovers more than before rather than less, and I did not find a rule that fixes this case
without breaking the wrapped-footer case above. Flagging it rather than hiding it.

Verification

laya/email.py had no test file. This adds tests/test_email.py (11 cases) in the style of the
existing standalone suites, covering both directions: the request surviving the footer, and pure
footer paragraphs still being removed.

Before (main, 42626c3) -- 5 failures, all the same defect:

$ python tests/test_email.py

6 passed, 5 failed
  FAIL inline footer/no blank line keeps the request:
     got  ''
     want 'My account is locked. Please unlock it.'
  FAIL inline footer/no terminal punctuation still recovers the request:
     got  ''
     want 'Please unlock it.'
  FAIL inline footer/body is never emptied:
     got  ''
     want 'My account is locked.'
  FAIL inline footer/is not empty
  FAIL email_state/body keeps the request:
     got  ''
     want 'My account is locked.'

After:

$ python tests/test_email.py

11 passed, 0 failed
$ python tests/test_criteria.py     ->  34 passed, 0 failed
$ python tests/test_router.py       -> 106 passed, 0 failed
$ python -m compileall laya         ->  exit 0

Diff: laya/email.py +15/-1, tests/test_email.py new.

Happy to adjust the split wording or the test cases if you would rather the cleaning stay more
aggressive.

…graph `clean_email_body` decided which paragraphs to drop with `_DISCLAIMER`, and that pattern matches on the single word "confidential". A footer that runs on without a blank line therefore shares a paragraph with the sender's actual request, and the whole paragraph went -- request included: clean_email_body("My account is locked.\n" "This email is confidential and intended solely for the named addressee.\n" "Please unlock it.") # before: '' the whole body, request included, is gone # after: 'My account is locked. Please unlock it.' `Agent.predict` then received an empty body, silently. The decision now happens inside the paragraph: a paragraph is dropped whole only when every sentence in it is boilerplate, and otherwise only the boilerplate sentences go. A paragraph with no disclaimer match is returned untouched, so the line structure of ordinary mail is unchanged. Splitting on sentences rather than newlines is deliberate. A footer broken across lines is one sentence and still disappears whole; splitting on newlines as well left its first line behind. The cleaning now errs towards keeping text, because dropping the request is silent and severe while leaving one boilerplate line behind is neither. tests/test_email.py: 11 regression tests, all new (0 -> 11). On this commit's parent the five body-wiping cases fail with `got ''`; quoted-history removal, signature removal and wrapped/standalone footers are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`tests/test_email.py` is not picked up by anything: both workflows run a hardcoded list of
suites, so the 11 cases added in the previous commit would never execute in CI. Add it to both
lists — the `lint`-style explicit enumeration the other two suites already use.

- `.github/workflows/ci.yml`: new step, next to the routing/language-detection suite.
- `.github/workflows/release.yml`: appended to the existing pre-publish test step, so a failure
  in email cleaning blocks a release the same way a routing failure does.
@PerryLink
PerryLink force-pushed the fix/email-disclaimer-paragraph branch from 2ec716a to f22ff1a Compare September 21, 2026 15:52
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for fixing this. Splitting by sentence before dropping disclaimers prevents legitimate user requests from being deleted when footers lack a blank line separator. The test suite covers the edge cases well.

@NandhaKishorM
NandhaKishorM merged commit 4c2c6f9 into NandhaKishorM:main Sep 21, 2026
10 of 15 checks passed
ahmedembeddedxx added a commit to ahmedembeddedxx/laya that referenced this pull request Sep 23, 2026
The sign-off pattern accepted any short line that started with a closing
word, followed by anything made of word characters, spaces and ",!.".
So an opener like "Thanks for getting back to me." or "Best time to reach
me is after 5pm." was taken for the signature, and everything below it
(the actual request) was cut. For emails of nine lines or fewer the
search starts at line 1, so the most common reply opener in support mail
reduced the body to "Hi,". The bare "best" prefix also matched words like
"Bestellung" ("order" in German).

A sign-off line now has to be a closing phrase, optionally followed by up
to three capitalised name tokens ("Thanks, Alice", "Best regards, Dr.
Jane Doe"). A closing written as a sentence ("Thanks for your help,") is
no longer stripped and stays in the body. Where the two goals conflict,
the cleaner keeps text, as in NandhaKishorM#94.
PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 24, 2026
The English quote-header marker was `^\s*From:\s.+$`, which matches any line that starts
`From: `. Ordinary prose opens that way, so the request was deleted and the model answered an
empty body — the same silent failure this module has been fixed for twice already (NandhaKishorM#94, NandhaKishorM#227):

```python
from laya import email

body = ("Hi team,\n"
        "From: my side the integration works, but please refund the duplicate charge today.\n"
        "Thanks")
print(email.clean_email_body(body))

# before
# 'Hi team,'
#   -> "refund" in cleaned is False; the model never sees the request

# after
# 'Hi team,\nFrom: my side the integration works, but please refund the duplicate charge today.\nThanks'
```

The `De:` marker two lines below already had this exact fix, for the same reason and with a
comment saying so — `De:` also opens ordinary Portuguese/Spanish lines. `From:` opens ordinary
English lines, and a reply header always carries the sender, so it is now gated on an address
the same way:

```python
re.compile(r"^\s*From:\s.*[@<]", re.I),   # was r"^\s*From:\s.+$"
```

Deliberate choices tested:

- **Gate on an address rather than on the shape of the text.** A word-count or capitalisation rule would be guessing at what a sentence looks like; "a reply header names its sender" is a property of the header, and it is the rule `De:` already uses, so the two now read the same way instead of one being stricter for no stated reason.
- **A bare `From: Name` header is still cut, by its neighbours.** This is the part I got wrong first: I assumed `_SIGNATURE_MARKERS` covered `Sent:`, but `_HEADER_NEXT` only listed the Portuguese/Spanish `Enviado:`/`Data:`/`Fecha:` forms, so dropping the address from the marker leaked every English quote header written without one. `_HEADER_FROM_NAME`/`_HEADER_NEXT` now accept the English pair too — `From: <name>` followed by `Sent:` or a dated `Date:` — which is the translation of the rule that was already there for `De:`.
- **`To:` alone is still not enough**, matching the existing note that "`De: 10/09 / Para: 15/09` is how a leave request reads". `From: Alice Smith` followed by `To: Bob` and a request is kept, because a `To:` line is not evidence of a quote block.
- **When it cannot be told apart, the request wins.** A bare `From: Name` with no header neighbour is kept. A leaked quote is noise the model can read past; a deleted request is a wrong answer, which is the asymmetry NandhaKishorM#94 and NandhaKishorM#227 were both about.

The suite had **no `From:` case at all** before this, which is why it stayed green on the bug —
`python tests/test_email.py` reported `90 passed, 0 failed` on `main` with the bug present. The
new block covers five prose forms, the four quoted-header forms that must still cut, and the two
ambiguous forms that must be kept:

```
# with the marker reverted to r"^\s*From:\s.+$"
98 passed, 7 failed
  FAIL en/`From:` prose keeps the request: the reported case Hi team,
  FAIL en/`From:` prose keeps the request: lowercase Hi,
  FAIL en/`From:` prose keeps the request: uppercase Hi,
  FAIL en/`From:` prose keeps the request: extra spacing Hi,
  FAIL en/`From:` prose keeps the request: mid-body Hello,
  FAIL en/bare `From:` with no header neighbour is kept: From: Name then To: Hi,
  FAIL en/bare `From:` with no header neighbour is kept: From: Name alone Hi,

# restored
105 passed, 0 failed
```

The four quoted-header positive controls stay green in both states, so the fix is not simply
"stop cutting on `From:`".

```
$ python tests/test_email.py                  105 passed, 0 failed   (90 before)
$ python tests/test_lang_guess.py             52 passed, 0 failed
$ python tests/test_router.py                 all routing tests passed
$ python tests/test_criteria.py               128 passed, 0 failed
$ python tests/test_packaging.py              all packaging tests passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: `From:` followed by an address but no other header line is treated as a quote
header and cut, because that is what an address on a `From:` line overwhelmingly is in an email
body. A caller who writes prose that literally begins `From: <someone@example.com>` loses that
line; I could not construct a realistic case where that is a request, and the quoted-header case
is the common one.
@PerryLink PerryLink mentioned this pull request Sep 24, 2026
2 of 4 tasks
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 25, 2026
The English quote-header marker was `^\s*From:\s.+$`, which matches any line that starts
`From: `. Ordinary prose opens that way, so the request was deleted and the model answered an
empty body — the same silent failure this module has been fixed for twice already (NandhaKishorM#94, NandhaKishorM#227):

```python
from laya import email

body = ("Hi team,\n"
        "From: my side the integration works, but please refund the duplicate charge today.\n"
        "Thanks")
print(email.clean_email_body(body))

# before
# 'Hi team,'
#   -> "refund" in cleaned is False; the model never sees the request

# after
# 'Hi team,\nFrom: my side the integration works, but please refund the duplicate charge today.\nThanks'
```

The `De:` marker two lines below already had this exact fix, for the same reason and with a
comment saying so — `De:` also opens ordinary Portuguese/Spanish lines. `From:` opens ordinary
English lines, and a reply header always carries the sender, so it is now gated on an address
the same way:

```python
re.compile(r"^\s*From:\s.*[@<]", re.I),   # was r"^\s*From:\s.+$"
```

Deliberate choices tested:

- **Gate on an address rather than on the shape of the text.** A word-count or capitalisation rule would be guessing at what a sentence looks like; "a reply header names its sender" is a property of the header, and it is the rule `De:` already uses, so the two now read the same way instead of one being stricter for no stated reason.
- **A bare `From: Name` header is still cut, by its neighbours.** This is the part I got wrong first: I assumed `_SIGNATURE_MARKERS` covered `Sent:`, but `_HEADER_NEXT` only listed the Portuguese/Spanish `Enviado:`/`Data:`/`Fecha:` forms, so dropping the address from the marker leaked every English quote header written without one. `_HEADER_FROM_NAME`/`_HEADER_NEXT` now accept the English pair too — `From: <name>` followed by `Sent:` or a dated `Date:` — which is the translation of the rule that was already there for `De:`.
- **`To:` alone is still not enough**, matching the existing note that "`De: 10/09 / Para: 15/09` is how a leave request reads". `From: Alice Smith` followed by `To: Bob` and a request is kept, because a `To:` line is not evidence of a quote block.
- **When it cannot be told apart, the request wins.** A bare `From: Name` with no header neighbour is kept. A leaked quote is noise the model can read past; a deleted request is a wrong answer, which is the asymmetry NandhaKishorM#94 and NandhaKishorM#227 were both about.

The suite had **no `From:` case at all** before this, which is why it stayed green on the bug —
`python tests/test_email.py` reported `90 passed, 0 failed` on `main` with the bug present. The
new block covers five prose forms, the four quoted-header forms that must still cut, and the two
ambiguous forms that must be kept:

```
# with the marker reverted to r"^\s*From:\s.+$"
98 passed, 7 failed
  FAIL en/`From:` prose keeps the request: the reported case Hi team,
  FAIL en/`From:` prose keeps the request: lowercase Hi,
  FAIL en/`From:` prose keeps the request: uppercase Hi,
  FAIL en/`From:` prose keeps the request: extra spacing Hi,
  FAIL en/`From:` prose keeps the request: mid-body Hello,
  FAIL en/bare `From:` with no header neighbour is kept: From: Name then To: Hi,
  FAIL en/bare `From:` with no header neighbour is kept: From: Name alone Hi,

# restored
105 passed, 0 failed
```

The four quoted-header positive controls stay green in both states, so the fix is not simply
"stop cutting on `From:`".

```
$ python tests/test_email.py                  105 passed, 0 failed   (90 before)
$ python tests/test_lang_guess.py             52 passed, 0 failed
$ python tests/test_router.py                 all routing tests passed
$ python tests/test_criteria.py               128 passed, 0 failed
$ python tests/test_packaging.py              all packaging tests passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: `From:` followed by an address but no other header line is treated as a quote
header and cut, because that is what an address on a `From:` line overwhelmingly is in an email
body. A caller who writes prose that literally begins `From: <someone@example.com>` loses that
line; I could not construct a realistic case where that is a request, and the quoted-header case
is the common one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

clean_email_body deletes the whole body when a disclaimer footer shares its paragraph

2 participants