Skip to content

fix(email): clean Portuguese and Spanish replies, signatures and footers - #200

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
paulocarinhena:fix/email-clean-pt-es
Sep 23, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
paulocarinhena:fix/email-clean-pt-es

Conversation

@paulocarinhena

@paulocarinhena paulocarinhena commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

The problem

clean_email_body (and therefore email_state) only recognises English mail-client markers:
On … wrote:, ----- Original Message -----, From:, Regards, Sent from my iPhone, confidential.

The Router already sends Portuguese and Spanish states to laya-multilingual, but the cleaning step in front of it was a no-op for those languages. For a typical Brazilian reply, nothing is removed:

Olá equipe,

Fomos cobrados duas vezes na fatura de março. Por favor, estornem a cobrança duplicada hoje.

Atenciosamente,
João Silva

Enviado do meu iPhone

Esta mensagem pode conter informações confidenciais. Se você recebeu esta mensagem por engano, favor apagá-la.

-----Mensagem original-----
De: Maria <maria@acme.com>
Assunto: cancelar contrato
Queremos cancelar o contrato imediatamente.

Before this PR clean_email_body returns the whole text above unchanged.
After it returns only the request:

Olá equipe,

Fomos cobrados duas vezes na fatura de março. Por favor, estornem a cobrança duplicada hoje.

Why it matters

This goes beyond cosmetics. The quoted history is usually an earlier, different request. Here the new message asks for a refund, while the quoted one asks to cancel the contract. Both go into the same body, so questions such as churn_risk, refund_requested or category are answered over a mix of two messages. Laya does not generate text, so the cleaning step is the only thing that keeps that noise out of the forward pass. Signatures and footers also use up part of the context budget (512/1024 tokens).

What this PR changes

Only laya/email.py, and mostly its pattern lists. The algorithm is unchanged apart from two small additions: the wrapped attribution and the address-less Outlook header, both described below.

added
quoted-reply headers Gmail Em … escreveu: / El … escribió:; Outlook -----Mensagem original-----, Mensagem encaminhada, Mensaje original, Mensaje reenviado; Outlook De: header
wrapped attribution Gmail wraps long attribution lines (… Suporte < / suporte@x.com> escreveu:). The tail line now cuts too and takes its On/Em/El … head with it. This also fixes the same case in English.
signatures Atenciosamente, Att, Abraços, Abs, Cordialmente, Grato/a, Obrigado/a, Cumprimentos, Saudações, Saludos, Atentamente, Gracias; Enviado do meu iPhone / Enviado desde mi …
mobile / mail-app footers Enviado do meu Galaxy, Enviado do meu smartphone Samsung Galaxy., Enviado do Outlook para iOS, Obter o Outlook para Android, Enviado do Email para Windows, Enviado do Yahoo Mail no Android, and the English Get Outlook for iOS/Android. Only a line that is nothing but the footer matches. Such a line may be up to 60 characters, because Samsung's default footer is 41 (sign-offs keep the existing 40).
Outlook header without address Exchange often writes De: Maria Souza with no address. A bare De: cuts when the header's own Enviado: line, or a dated Data:/Fecha: line, follows it.
disclaimers Esta mensagem/Este e-mail … confidencial/sigiloso, uso exclusivo do destinatário, (Se você / Caso tenha) recebido esta mensagem por engano, Si ha recibido este mensaje por error; the Brazilian print footer Antes de imprimir, pense no meio ambiente / …verifique se é realmente necessário

Guarding against false positives

Dropping the sender's request is worse than leaving boilerplate behind (the same principle as #94). For that reason each new marker is narrower than its English counterpart:

  • Attribution headers must contain a digit (a date). Real client headers always do, whereas Em resposta ao que você escreveu: is ordinary body text.
  • De: must carry an address (@ or <). De: 10/09 a 15/09 is ordinary Portuguese/Spanish text.
  • Sign-offs must stand alone on their line. The English marker accepts trailing words ([\w ,!.]*). The PT/ES marker does not, so Obrigado pelo retorno, mas continua sem funcionar is kept.
  • Disclaimers are tied to "esta mensagem / este e-mail", never the bare word confidencial, so Preciso do contrato confidencial assinado is kept. destinado exclusivamente alone is not enough either (O valor é destinado exclusivamente ao pagamento is kept). It must refer to a recipient/person/entity.
  • PT/ES disclaimer alternatives use word boundaries (contesta mensagem does not match esta mensagem).
  • Device footers must be the whole line. Enviado do meu celular o comprovante ontem. is a request and is kept. Get only matches Get Outlook for …, never Get mail.
  • An address-less De: needs Enviado: or a dated Data: below it. Para: is not enough (De: 10/09 / Para: 15/09 is a leave request), and neither is an undated Data: amanhã cedo.
  • The print footer is matched on its environmental ending, so Antes de imprimir o boleto, confira o valor is kept.

English behaviour is otherwise unchanged. All existing tests pass untouched.

Tests

tests/test_email.py (already run by CI and by the release workflow) gains 31 cases:

  • 21 positive cases: full PT-BR reply, Outlook block with and without separator, Gmail attribution wrapped at the address and inside the name, the same wrapped case in English, short PT sign-off, Caso tenha recebido… footer, ES reply, ES footer, 6 mobile/mail-app footers (including Samsung's 41-character default), the English Outlook download line, two print-footer variants, and the address-less Outlook header followed by Enviado: and by a dated Data:.
  • 10 negative cases (text that must survive): confidencial inside a request, Obrigado opening a sentence, Em … escreveu: without a date, destinado exclusivamente inside a request, De: without an address, device words inside a request, a De:/Para: date range, De: followed by an undated Data:, antes de imprimir inside a request, and Get mail.
new code:  42 passed, 0 failed
on main:   21 passed, 21 failed   <- exactly the 21 positive cases, i.e. the tests reproduce the bug

The 10 must-survive cases pass on both, as do the 11 existing English tests, so no text that was kept before is dropped now.

Out of scope / follow-up

  • PT sign-offs with the name on the same line (Att, João) are deliberately not matched. That would require letting trailing words through, which is what makes Obrigado pelo retorno… risky.
  • Separate PR: lang.py sends short or unaccented Brazilian Portuguese to the English checkpoint ("Boa tarde, gostaria de cancelar o plano", "Voce pode me mandar a nota fiscal?"), because você/voce/vc, nao and gostaria are missing from _STOP["pt"].

clean_email_body only knew English markers, so for Portuguese and Spanish mail it removed nothing:
Gmail's "Em ... escreveu:" / "El ... escribió:", Outlook's "-----Mensagem original-----" and "De:"
header, "Atenciosamente" / "Enviado do meu iPhone" and the confidentiality footer all reached the
model. The quoted history is usually an earlier, different request, so it pulled choice and noul
answers towards the old message.

Add PT/ES markers to the existing lists, and cut a Gmail attribution line that was wrapped in two
(in English too). Each new marker is kept narrower than its English counterpart so that it cannot
eat the sender's request: attributions need a date, `De:` needs an address, sign-offs must stand
alone on their line, and footers are tied to "esta mensagem/este e-mail" rather than the bare word
"confidencial".
…rint footer

Three gaps that left Brazilian mail uncleaned:

- Mobile and mail-app footers: "Enviado do meu Galaxy", "Enviado do Outlook para iOS", "Obter o
  Outlook para Android", "Enviado do Email para Windows", "Enviado do Yahoo Mail no Android".
  Only a line that is nothing but the footer matches, and it may run to 60 characters because
  Samsung's default ("Enviado do meu smartphone Samsung Galaxy.") is 41.
- Outlook reply headers without an address ("De: Maria Souza"), which Exchange produces. The quoted
  history still reached the model. A bare `De:` now cuts when an `Enviado:` line or a dated `Data:`
  line follows it; `Para:` alone does not, since "De: 10/09 / Para: 15/09" is a leave request.
- The "Antes de imprimir, pense no meio ambiente" footer, matched on its environmental ending so
  that "Antes de imprimir o boleto, confira o valor" is kept.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks, this closes a gap nobody had noticed: the router already sent Portuguese and Spanish mail to the multilingual checkpoint, but the cleaning in front of it did nothing for those languages.

The email suite goes from 11 to 42 checks and passes, together with everything else. Merging.

@NandhaKishorM
NandhaKishorM merged commit 7a58ab6 into NandhaKishorM:main Sep 23, 2026
NandhaKishorM pushed a commit that referenced this pull request Sep 23, 2026
`_SIGNATURE_MARKERS` allowed `[\w ,!.]*` behind the closing word. So a body
line that starts with one of them and stays under the 40-character guard was
read as the first line of a signature. `clean_email_body` breaks at the first
match and drops everything after it, so the sender's actual request went too.
`email_state` cleans by default, so the model was scored on what was left.

Measured on 010bace, Python 3.14.7, macOS 25.6.0:

    clean_email_body("Hi,\n\nThanks for the quick reply.\n"
                     "Could you refund invoice 4411 as well?")
    # 'Hi,'

Two more of the same shape: "Thanks for looking into it." and "Best practice
would be a manual unlock." each cut the line that carried the request.

A closing line is the closing word, punctuation, and at most a name. The next
word separates the two readings. A name is capitalised. The "for" in "Thanks
for the quick reply." is not. The closing words stay case-insensitive and the
name does not, so the flag is scoped and not global. The optional second word
covers the lowercase closers ("in advance", "a lot", "very much").

This is the rule the file already states for its other languages. The
Portuguese and Spanish markers from #200 say: "unlike the English marker no
trailing words are allowed". This change brings the English marker in line.

Checked against 25 sign-off lines and 11 body lines. All 25 are still cut, and
all 11 are now kept. The old pattern matched 10 of those 11. Eleven of the 25
are in the test as positive controls. A later change that stops cutting real
signatures fails the suite too.

Rejected:

* A tighter length guard. It is already 40 characters, and "Thanks for the
  quick reply." is 27. Length does not separate the cases.
* A rule that the closing line must sit near the end. For a mail under nine
  lines, `min(int(len * 0.6), len - 8)` goes negative. The scan window then
  starts at line 1. A wider window would stop cutting signatures above a long
  address block. Case, not position, is the signal.

tests/test_email.py on 010bace: 57 passed / 4 failed before, 61 passed / 0
failed after. First measured on 573e5b6: 22 / 4 before, 26 / 0 after. The 16
test files in the CI list, ruff and compileall all pass after the change.
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…ers (NandhaKishorM#200)

* fix(email): clean Portuguese and Spanish replies, signatures and footers

clean_email_body only knew English markers, so for Portuguese and Spanish mail it removed nothing:
Gmail's "Em ... escreveu:" / "El ... escribió:", Outlook's "-----Mensagem original-----" and "De:"
header, "Atenciosamente" / "Enviado do meu iPhone" and the confidentiality footer all reached the
model. The quoted history is usually an earlier, different request, so it pulled choice and noul
answers towards the old message.

Add PT/ES markers to the existing lists, and cut a Gmail attribution line that was wrapped in two
(in English too). Each new marker is kept narrower than its English counterpart so that it cannot
eat the sender's request: attributions need a date, `De:` needs an address, sign-offs must stand
alone on their line, and footers are tied to "esta mensagem/este e-mail" rather than the bare word
"confidencial".

* fix(email): Brazilian mobile footers, address-less Outlook headers, print footer

Three gaps that left Brazilian mail uncleaned:

- Mobile and mail-app footers: "Enviado do meu Galaxy", "Enviado do Outlook para iOS", "Obter o
  Outlook para Android", "Enviado do Email para Windows", "Enviado do Yahoo Mail no Android".
  Only a line that is nothing but the footer matches, and it may run to 60 characters because
  Samsung's default ("Enviado do meu smartphone Samsung Galaxy.") is 41.
- Outlook reply headers without an address ("De: Maria Souza"), which Exchange produces. The quoted
  history still reached the model. A bare `De:` now cuts when an `Enviado:` line or a dated `Data:`
  line follows it; `Para:` alone does not, since "De: 10/09 / Para: 15/09" is a leave request.
- The "Antes de imprimir, pense no meio ambiente" footer, matched on its environmental ending so
  that "Antes de imprimir o boleto, confira o valor" is kept.
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…aKishorM#132)

`_SIGNATURE_MARKERS` allowed `[\w ,!.]*` behind the closing word. So a body
line that starts with one of them and stays under the 40-character guard was
read as the first line of a signature. `clean_email_body` breaks at the first
match and drops everything after it, so the sender's actual request went too.
`email_state` cleans by default, so the model was scored on what was left.

Measured on 010bace, Python 3.14.7, macOS 25.6.0:

    clean_email_body("Hi,\n\nThanks for the quick reply.\n"
                     "Could you refund invoice 4411 as well?")
    # 'Hi,'

Two more of the same shape: "Thanks for looking into it." and "Best practice
would be a manual unlock." each cut the line that carried the request.

A closing line is the closing word, punctuation, and at most a name. The next
word separates the two readings. A name is capitalised. The "for" in "Thanks
for the quick reply." is not. The closing words stay case-insensitive and the
name does not, so the flag is scoped and not global. The optional second word
covers the lowercase closers ("in advance", "a lot", "very much").

This is the rule the file already states for its other languages. The
Portuguese and Spanish markers from NandhaKishorM#200 say: "unlike the English marker no
trailing words are allowed". This change brings the English marker in line.

Checked against 25 sign-off lines and 11 body lines. All 25 are still cut, and
all 11 are now kept. The old pattern matched 10 of those 11. Eleven of the 25
are in the test as positive controls. A later change that stops cutting real
signatures fails the suite too.

Rejected:

* A tighter length guard. It is already 40 characters, and "Thanks for the
  quick reply." is 27. Length does not separate the cases.
* A rule that the closing line must sit near the end. For a mail under nine
  lines, `min(int(len * 0.6), len - 8)` goes negative. The scan window then
  starts at line 1. A wider window would stop cutting signatures above a long
  address block. Case, not position, is the signal.

tests/test_email.py on 010bace: 57 passed / 4 failed before, 61 passed / 0
failed after. First measured on 573e5b6: 22 / 4 before, 26 / 0 after. The 16
test files in the CI list, ruff and compileall all pass after the change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants