Repository navigation
fix(email): clean Portuguese and Spanish replies, signatures and footers - #200
Merged
NandhaKishorM merged 2 commits intoSep 23, 2026
Merged
Conversation
clean_email_body only knew English markers, so for Portuguese and Spanish mail it removed nothing: Gmail's "Em ... escreveu:" / "El ... escribió:", Outlook's "-----Mensagem original-----" and "De:" header, "Atenciosamente" / "Enviado do meu iPhone" and the confidentiality footer all reached the model. The quoted history is usually an earlier, different request, so it pulled choice and noul answers towards the old message. Add PT/ES markers to the existing lists, and cut a Gmail attribution line that was wrapped in two (in English too). Each new marker is kept narrower than its English counterpart so that it cannot eat the sender's request: attributions need a date, `De:` needs an address, sign-offs must stand alone on their line, and footers are tied to "esta mensagem/este e-mail" rather than the bare word "confidencial".
paulocarinhena
force-pushed
the
fix/email-clean-pt-es
branch
from
September 23, 2026 03:44
aaa644e to
580b655
Compare
…rint footer
Three gaps that left Brazilian mail uncleaned:
- Mobile and mail-app footers: "Enviado do meu Galaxy", "Enviado do Outlook para iOS", "Obter o
Outlook para Android", "Enviado do Email para Windows", "Enviado do Yahoo Mail no Android".
Only a line that is nothing but the footer matches, and it may run to 60 characters because
Samsung's default ("Enviado do meu smartphone Samsung Galaxy.") is 41.
- Outlook reply headers without an address ("De: Maria Souza"), which Exchange produces. The quoted
history still reached the model. A bare `De:` now cuts when an `Enviado:` line or a dated `Data:`
line follows it; `Para:` alone does not, since "De: 10/09 / Para: 15/09" is a leave request.
- The "Antes de imprimir, pense no meio ambiente" footer, matched on its environmental ending so
that "Antes de imprimir o boleto, confira o valor" is kept.
Owner
|
Thanks, this closes a gap nobody had noticed: the router already sent Portuguese and Spanish mail to the multilingual checkpoint, but the cleaning in front of it did nothing for those languages. The email suite goes from 11 to 42 checks and passes, together with everything else. Merging. |
This was referenced Sep 23, 2026
NandhaKishorM
pushed a commit
that referenced
this pull request
Sep 23, 2026
`_SIGNATURE_MARKERS` allowed `[\w ,!.]*` behind the closing word. So a body line that starts with one of them and stays under the 40-character guard was read as the first line of a signature. `clean_email_body` breaks at the first match and drops everything after it, so the sender's actual request went too. `email_state` cleans by default, so the model was scored on what was left. Measured on 010bace, Python 3.14.7, macOS 25.6.0: clean_email_body("Hi,\n\nThanks for the quick reply.\n" "Could you refund invoice 4411 as well?") # 'Hi,' Two more of the same shape: "Thanks for looking into it." and "Best practice would be a manual unlock." each cut the line that carried the request. A closing line is the closing word, punctuation, and at most a name. The next word separates the two readings. A name is capitalised. The "for" in "Thanks for the quick reply." is not. The closing words stay case-insensitive and the name does not, so the flag is scoped and not global. The optional second word covers the lowercase closers ("in advance", "a lot", "very much"). This is the rule the file already states for its other languages. The Portuguese and Spanish markers from #200 say: "unlike the English marker no trailing words are allowed". This change brings the English marker in line. Checked against 25 sign-off lines and 11 body lines. All 25 are still cut, and all 11 are now kept. The old pattern matched 10 of those 11. Eleven of the 25 are in the test as positive controls. A later change that stops cutting real signatures fails the suite too. Rejected: * A tighter length guard. It is already 40 characters, and "Thanks for the quick reply." is 27. Length does not separate the cases. * A rule that the closing line must sit near the end. For a mail under nine lines, `min(int(len * 0.6), len - 8)` goes negative. The scan window then starts at line 1. A wider window would stop cutting signatures above a long address block. Case, not position, is the signal. tests/test_email.py on 010bace: 57 passed / 4 failed before, 61 passed / 0 failed after. First measured on 573e5b6: 22 / 4 before, 26 / 0 after. The 16 test files in the CI list, ruff and compileall all pass after the change.
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
…ers (NandhaKishorM#200) * fix(email): clean Portuguese and Spanish replies, signatures and footers clean_email_body only knew English markers, so for Portuguese and Spanish mail it removed nothing: Gmail's "Em ... escreveu:" / "El ... escribió:", Outlook's "-----Mensagem original-----" and "De:" header, "Atenciosamente" / "Enviado do meu iPhone" and the confidentiality footer all reached the model. The quoted history is usually an earlier, different request, so it pulled choice and noul answers towards the old message. Add PT/ES markers to the existing lists, and cut a Gmail attribution line that was wrapped in two (in English too). Each new marker is kept narrower than its English counterpart so that it cannot eat the sender's request: attributions need a date, `De:` needs an address, sign-offs must stand alone on their line, and footers are tied to "esta mensagem/este e-mail" rather than the bare word "confidencial". * fix(email): Brazilian mobile footers, address-less Outlook headers, print footer Three gaps that left Brazilian mail uncleaned: - Mobile and mail-app footers: "Enviado do meu Galaxy", "Enviado do Outlook para iOS", "Obter o Outlook para Android", "Enviado do Email para Windows", "Enviado do Yahoo Mail no Android". Only a line that is nothing but the footer matches, and it may run to 60 characters because Samsung's default ("Enviado do meu smartphone Samsung Galaxy.") is 41. - Outlook reply headers without an address ("De: Maria Souza"), which Exchange produces. The quoted history still reached the model. A bare `De:` now cuts when an `Enviado:` line or a dated `Data:` line follows it; `Para:` alone does not, since "De: 10/09 / Para: 15/09" is a leave request. - The "Antes de imprimir, pense no meio ambiente" footer, matched on its environmental ending so that "Antes de imprimir o boleto, confira o valor" is kept.
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
…aKishorM#132) `_SIGNATURE_MARKERS` allowed `[\w ,!.]*` behind the closing word. So a body line that starts with one of them and stays under the 40-character guard was read as the first line of a signature. `clean_email_body` breaks at the first match and drops everything after it, so the sender's actual request went too. `email_state` cleans by default, so the model was scored on what was left. Measured on 010bace, Python 3.14.7, macOS 25.6.0: clean_email_body("Hi,\n\nThanks for the quick reply.\n" "Could you refund invoice 4411 as well?") # 'Hi,' Two more of the same shape: "Thanks for looking into it." and "Best practice would be a manual unlock." each cut the line that carried the request. A closing line is the closing word, punctuation, and at most a name. The next word separates the two readings. A name is capitalised. The "for" in "Thanks for the quick reply." is not. The closing words stay case-insensitive and the name does not, so the flag is scoped and not global. The optional second word covers the lowercase closers ("in advance", "a lot", "very much"). This is the rule the file already states for its other languages. The Portuguese and Spanish markers from NandhaKishorM#200 say: "unlike the English marker no trailing words are allowed". This change brings the English marker in line. Checked against 25 sign-off lines and 11 body lines. All 25 are still cut, and all 11 are now kept. The old pattern matched 10 of those 11. Eleven of the 25 are in the test as positive controls. A later change that stops cutting real signatures fails the suite too. Rejected: * A tighter length guard. It is already 40 characters, and "Thanks for the quick reply." is 27. Length does not separate the cases. * A rule that the closing line must sit near the end. For a mail under nine lines, `min(int(len * 0.6), len - 8)` goes negative. The scan window then starts at line 1. A wider window would stop cutting signatures above a long address block. Case, not position, is the signal. tests/test_email.py on 010bace: 57 passed / 4 failed before, 61 passed / 0 failed after. First measured on 573e5b6: 22 / 4 before, 26 / 0 after. The 16 test files in the CI list, ruff and compileall all pass after the change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
clean_email_body(and thereforeemail_state) only recognises English mail-client markers:On … wrote:,----- Original Message -----,From:,Regards,Sent from my iPhone,confidential.The
Routeralready sends Portuguese and Spanish states tolaya-multilingual, but the cleaning step in front of it was a no-op for those languages. For a typical Brazilian reply, nothing is removed:Before this PR
clean_email_bodyreturns the whole text above unchanged.After it returns only the request:
Why it matters
This goes beyond cosmetics. The quoted history is usually an earlier, different request. Here the new message asks for a refund, while the quoted one asks to cancel the contract. Both go into the same
body, so questions such aschurn_risk,refund_requestedorcategoryare answered over a mix of two messages. Laya does not generate text, so the cleaning step is the only thing that keeps that noise out of the forward pass. Signatures and footers also use up part of the context budget (512/1024 tokens).What this PR changes
Only
laya/email.py, and mostly its pattern lists. The algorithm is unchanged apart from two small additions: the wrapped attribution and the address-less Outlook header, both described below.Em … escreveu:/El … escribió:; Outlook-----Mensagem original-----,Mensagem encaminhada,Mensaje original,Mensaje reenviado; OutlookDe:header… Suporte </suporte@x.com> escreveu:). The tail line now cuts too and takes itsOn/Em/El …head with it. This also fixes the same case in English.Atenciosamente,Att,Abraços,Abs,Cordialmente,Grato/a,Obrigado/a,Cumprimentos,Saudações,Saludos,Atentamente,Gracias;Enviado do meu iPhone/Enviado desde mi …Enviado do meu Galaxy,Enviado do meu smartphone Samsung Galaxy.,Enviado do Outlook para iOS,Obter o Outlook para Android,Enviado do Email para Windows,Enviado do Yahoo Mail no Android, and the EnglishGet Outlook for iOS/Android. Only a line that is nothing but the footer matches. Such a line may be up to 60 characters, because Samsung's default footer is 41 (sign-offs keep the existing 40).De: Maria Souzawith no address. A bareDe:cuts when the header's ownEnviado:line, or a datedData:/Fecha:line, follows it.Esta mensagem/Este e-mail … confidencial/sigiloso,uso exclusivo do destinatário,(Se você / Caso tenha) recebido esta mensagem por engano,Si ha recibido este mensaje por error; the Brazilian print footerAntes de imprimir, pense no meio ambiente/…verifique se é realmente necessárioGuarding against false positives
Dropping the sender's request is worse than leaving boilerplate behind (the same principle as #94). For that reason each new marker is narrower than its English counterpart:
Em resposta ao que você escreveu:is ordinary body text.De:must carry an address (@or<).De: 10/09 a 15/09is ordinary Portuguese/Spanish text.[\w ,!.]*). The PT/ES marker does not, soObrigado pelo retorno, mas continua sem funcionaris kept.confidencial, soPreciso do contrato confidencial assinadois kept.destinado exclusivamentealone is not enough either (O valor é destinado exclusivamente ao pagamentois kept). It must refer to a recipient/person/entity.contesta mensagemdoes not matchesta mensagem).Enviado do meu celular o comprovante ontem.is a request and is kept.Getonly matchesGet Outlook for …, neverGet mail.De:needsEnviado:or a datedData:below it.Para:is not enough (De: 10/09/Para: 15/09is a leave request), and neither is an undatedData: amanhã cedo.Antes de imprimir o boleto, confira o valoris kept.English behaviour is otherwise unchanged. All existing tests pass untouched.
Tests
tests/test_email.py(already run by CI and by the release workflow) gains 31 cases:Caso tenha recebido…footer, ES reply, ES footer, 6 mobile/mail-app footers (including Samsung's 41-character default), the English Outlook download line, two print-footer variants, and the address-less Outlook header followed byEnviado:and by a datedData:.confidencialinside a request,Obrigadoopening a sentence,Em … escreveu:without a date,destinado exclusivamenteinside a request,De:without an address, device words inside a request, aDe:/Para:date range,De:followed by an undatedData:,antes de imprimirinside a request, andGet mail.The 10 must-survive cases pass on both, as do the 11 existing English tests, so no text that was kept before is dropped now.
Out of scope / follow-up
Att, João) are deliberately not matched. That would require letting trailing words through, which is what makesObrigado pelo retorno…risky.lang.pysends short or unaccented Brazilian Portuguese to the English checkpoint ("Boa tarde, gostaria de cancelar o plano","Voce pode me mandar a nota fiscal?"), becausevocê/voce/vc,naoandgostariaare missing from_STOP["pt"].