Repository navigation
fix(email): keep the request when a disclaimer footer shares its paragraph - #94
Merged
NandhaKishorM merged 2 commits intoSep 21, 2026
Conversation
…graph `clean_email_body` decided which paragraphs to drop with `_DISCLAIMER`, and that pattern matches on the single word "confidential". A footer that runs on without a blank line therefore shares a paragraph with the sender's actual request, and the whole paragraph went -- request included: clean_email_body("My account is locked.\n" "This email is confidential and intended solely for the named addressee.\n" "Please unlock it.") # before: '' the whole body, request included, is gone # after: 'My account is locked. Please unlock it.' `Agent.predict` then received an empty body, silently. The decision now happens inside the paragraph: a paragraph is dropped whole only when every sentence in it is boilerplate, and otherwise only the boilerplate sentences go. A paragraph with no disclaimer match is returned untouched, so the line structure of ordinary mail is unchanged. Splitting on sentences rather than newlines is deliberate. A footer broken across lines is one sentence and still disappears whole; splitting on newlines as well left its first line behind. The cleaning now errs towards keeping text, because dropping the request is silent and severe while leaving one boilerplate line behind is neither. tests/test_email.py: 11 regression tests, all new (0 -> 11). On this commit's parent the five body-wiping cases fail with `got ''`; quoted-history removal, signature removal and wrapped/standalone footers are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`tests/test_email.py` is not picked up by anything: both workflows run a hardcoded list of suites, so the 11 cases added in the previous commit would never execute in CI. Add it to both lists — the `lint`-style explicit enumeration the other two suites already use. - `.github/workflows/ci.yml`: new step, next to the routing/language-detection suite. - `.github/workflows/release.yml`: appended to the existing pre-publish test step, so a failure in email cleaning blocks a release the same way a routing failure does.
PerryLink
force-pushed
the
fix/email-disclaimer-paragraph
branch
from
September 21, 2026 15:52
2ec716a to
f22ff1a
Compare
Owner
|
Thank you for fixing this. Splitting by sentence before dropping disclaimers prevents legitimate user requests from being deleted when footers lack a blank line separator. The test suite covers the edge cases well. |
ahmedembeddedxx
added a commit
to ahmedembeddedxx/laya
that referenced
this pull request
Sep 23, 2026
The sign-off pattern accepted any short line that started with a closing
word, followed by anything made of word characters, spaces and ",!.".
So an opener like "Thanks for getting back to me." or "Best time to reach
me is after 5pm." was taken for the signature, and everything below it
(the actual request) was cut. For emails of nine lines or fewer the
search starts at line 1, so the most common reply opener in support mail
reduced the body to "Hi,". The bare "best" prefix also matched words like
"Bestellung" ("order" in German).
A sign-off line now has to be a closing phrase, optionally followed by up
to three capitalised name tokens ("Thanks, Alice", "Best regards, Dr.
Jane Doe"). A closing written as a sentence ("Thanks for your help,") is
no longer stripped and stays in the body. Where the two goals conflict,
the cleaner keeps text, as in NandhaKishorM#94.
PerryLink
added a commit
to PerryLink/laya
that referenced
this pull request
Sep 24, 2026
The English quote-header marker was `^\s*From:\s.+$`, which matches any line that starts `From: `. Ordinary prose opens that way, so the request was deleted and the model answered an empty body — the same silent failure this module has been fixed for twice already (NandhaKishorM#94, NandhaKishorM#227): ```python from laya import email body = ("Hi team,\n" "From: my side the integration works, but please refund the duplicate charge today.\n" "Thanks") print(email.clean_email_body(body)) # before # 'Hi team,' # -> "refund" in cleaned is False; the model never sees the request # after # 'Hi team,\nFrom: my side the integration works, but please refund the duplicate charge today.\nThanks' ``` The `De:` marker two lines below already had this exact fix, for the same reason and with a comment saying so — `De:` also opens ordinary Portuguese/Spanish lines. `From:` opens ordinary English lines, and a reply header always carries the sender, so it is now gated on an address the same way: ```python re.compile(r"^\s*From:\s.*[@<]", re.I), # was r"^\s*From:\s.+$" ``` Deliberate choices tested: - **Gate on an address rather than on the shape of the text.** A word-count or capitalisation rule would be guessing at what a sentence looks like; "a reply header names its sender" is a property of the header, and it is the rule `De:` already uses, so the two now read the same way instead of one being stricter for no stated reason. - **A bare `From: Name` header is still cut, by its neighbours.** This is the part I got wrong first: I assumed `_SIGNATURE_MARKERS` covered `Sent:`, but `_HEADER_NEXT` only listed the Portuguese/Spanish `Enviado:`/`Data:`/`Fecha:` forms, so dropping the address from the marker leaked every English quote header written without one. `_HEADER_FROM_NAME`/`_HEADER_NEXT` now accept the English pair too — `From: <name>` followed by `Sent:` or a dated `Date:` — which is the translation of the rule that was already there for `De:`. - **`To:` alone is still not enough**, matching the existing note that "`De: 10/09 / Para: 15/09` is how a leave request reads". `From: Alice Smith` followed by `To: Bob` and a request is kept, because a `To:` line is not evidence of a quote block. - **When it cannot be told apart, the request wins.** A bare `From: Name` with no header neighbour is kept. A leaked quote is noise the model can read past; a deleted request is a wrong answer, which is the asymmetry NandhaKishorM#94 and NandhaKishorM#227 were both about. The suite had **no `From:` case at all** before this, which is why it stayed green on the bug — `python tests/test_email.py` reported `90 passed, 0 failed` on `main` with the bug present. The new block covers five prose forms, the four quoted-header forms that must still cut, and the two ambiguous forms that must be kept: ``` # with the marker reverted to r"^\s*From:\s.+$" 98 passed, 7 failed FAIL en/`From:` prose keeps the request: the reported case Hi team, FAIL en/`From:` prose keeps the request: lowercase Hi, FAIL en/`From:` prose keeps the request: uppercase Hi, FAIL en/`From:` prose keeps the request: extra spacing Hi, FAIL en/`From:` prose keeps the request: mid-body Hello, FAIL en/bare `From:` with no header neighbour is kept: From: Name then To: Hi, FAIL en/bare `From:` with no header neighbour is kept: From: Name alone Hi, # restored 105 passed, 0 failed ``` The four quoted-header positive controls stay green in both states, so the fix is not simply "stop cutting on `From:`". ``` $ python tests/test_email.py 105 passed, 0 failed (90 before) $ python tests/test_lang_guess.py 52 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_criteria.py 128 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` Known limitation: `From:` followed by an address but no other header line is treated as a quote header and cut, because that is what an address on a `From:` line overwhelmingly is in an email body. A caller who writes prose that literally begins `From: <someone@example.com>` loses that line; I could not construct a realistic case where that is a request, and the quoted-header case is the common one.
2 of 4 tasks
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 25, 2026
The English quote-header marker was `^\s*From:\s.+$`, which matches any line that starts `From: `. Ordinary prose opens that way, so the request was deleted and the model answered an empty body — the same silent failure this module has been fixed for twice already (NandhaKishorM#94, NandhaKishorM#227): ```python from laya import email body = ("Hi team,\n" "From: my side the integration works, but please refund the duplicate charge today.\n" "Thanks") print(email.clean_email_body(body)) # before # 'Hi team,' # -> "refund" in cleaned is False; the model never sees the request # after # 'Hi team,\nFrom: my side the integration works, but please refund the duplicate charge today.\nThanks' ``` The `De:` marker two lines below already had this exact fix, for the same reason and with a comment saying so — `De:` also opens ordinary Portuguese/Spanish lines. `From:` opens ordinary English lines, and a reply header always carries the sender, so it is now gated on an address the same way: ```python re.compile(r"^\s*From:\s.*[@<]", re.I), # was r"^\s*From:\s.+$" ``` Deliberate choices tested: - **Gate on an address rather than on the shape of the text.** A word-count or capitalisation rule would be guessing at what a sentence looks like; "a reply header names its sender" is a property of the header, and it is the rule `De:` already uses, so the two now read the same way instead of one being stricter for no stated reason. - **A bare `From: Name` header is still cut, by its neighbours.** This is the part I got wrong first: I assumed `_SIGNATURE_MARKERS` covered `Sent:`, but `_HEADER_NEXT` only listed the Portuguese/Spanish `Enviado:`/`Data:`/`Fecha:` forms, so dropping the address from the marker leaked every English quote header written without one. `_HEADER_FROM_NAME`/`_HEADER_NEXT` now accept the English pair too — `From: <name>` followed by `Sent:` or a dated `Date:` — which is the translation of the rule that was already there for `De:`. - **`To:` alone is still not enough**, matching the existing note that "`De: 10/09 / Para: 15/09` is how a leave request reads". `From: Alice Smith` followed by `To: Bob` and a request is kept, because a `To:` line is not evidence of a quote block. - **When it cannot be told apart, the request wins.** A bare `From: Name` with no header neighbour is kept. A leaked quote is noise the model can read past; a deleted request is a wrong answer, which is the asymmetry NandhaKishorM#94 and NandhaKishorM#227 were both about. The suite had **no `From:` case at all** before this, which is why it stayed green on the bug — `python tests/test_email.py` reported `90 passed, 0 failed` on `main` with the bug present. The new block covers five prose forms, the four quoted-header forms that must still cut, and the two ambiguous forms that must be kept: ``` # with the marker reverted to r"^\s*From:\s.+$" 98 passed, 7 failed FAIL en/`From:` prose keeps the request: the reported case Hi team, FAIL en/`From:` prose keeps the request: lowercase Hi, FAIL en/`From:` prose keeps the request: uppercase Hi, FAIL en/`From:` prose keeps the request: extra spacing Hi, FAIL en/`From:` prose keeps the request: mid-body Hello, FAIL en/bare `From:` with no header neighbour is kept: From: Name then To: Hi, FAIL en/bare `From:` with no header neighbour is kept: From: Name alone Hi, # restored 105 passed, 0 failed ``` The four quoted-header positive controls stay green in both states, so the fix is not simply "stop cutting on `From:`". ``` $ python tests/test_email.py 105 passed, 0 failed (90 before) $ python tests/test_lang_guess.py 52 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_criteria.py 128 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` Known limitation: `From:` followed by an address but no other header line is treated as a quote header and cut, because that is what an address on a `From:` line overwhelmingly is in an email body. A caller who writes prose that literally begins `From: <someone@example.com>` loses that line; I could not construct a realistic case where that is a request, and the quoted-header case is the common one.
8 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #93
clean_email_bodycan return an empty string for a legitimate email._DISCLAIMERwas appliedto whole paragraphs and matches on the single word
confidential, so a footer that runs on withouta blank line sits in the same paragraph as the sender's request and both were deleted together:
The blank line is the only thing that matters. Adding one before the footer was already enough to
avoid the bug, which is why this needed a wrapped footer to surface.
Agent.predict/Router.predictthen receives a state with an emptybodyand answers, withconfidence, about an email it can no longer see. Nothing warns.
Change
The keep-or-drop decision now happens inside a paragraph instead of over the whole thing: a
paragraph is dropped only when every sentence in it is boilerplate, and otherwise just the
boilerplate sentences go. A paragraph with no disclaimer match is returned untouched, so the line
structure of ordinary mail is unchanged.
Two details are deliberate, and I tested the alternatives:
disappears whole. Splitting on newlines as well left its first line behind -- that variant
regressed the wrapped-footer case, so it is not what is here.
boilerplate line behind is neither. Where the two goals conflict, this change keeps the text.
Nothing else moved
Measured on
42626c3and on this branch, same inputs:'My account is locked. Please unlock it.''My account is locked.''''My account is locked.'On ... wrote:,>)--block)''''Known limitation
Sentence splitting needs a sentence boundary. Prose with no terminal punctuation directly followed
by the footer in the same paragraph still loses its first fragment:
It recovers more than before rather than less, and I did not find a rule that fixes this case
without breaking the wrapped-footer case above. Flagging it rather than hiding it.
Verification
laya/email.pyhad no test file. This addstests/test_email.py(11 cases) in the style of theexisting standalone suites, covering both directions: the request surviving the footer, and pure
footer paragraphs still being removed.
Before (
main,42626c3) -- 5 failures, all the same defect:After:
Diff:
laya/email.py+15/-1,tests/test_email.pynew.Happy to adjust the split wording or the test cases if you would rather the cleaning stay more
aggressive.