Handle XMPP uploads

[selfhoster1312]

Current status:

• for incoming messages
  • extract OOB URLs
  • download the URL in the background
  • forward the message for other bridges
• for outgoing messages
  • find the XMPP file upload component
  • get an upload token
  • perform the HTTP upload
  • send the message containing the new upload

Learned along the way:

• if you forget to set RemoteNickFormat in config, you're gonna have a bad time
• xmpp.Client.SendOOB actually ignores any Text body set, which is bad for OOB file uploads where the body must match the OOB url

Some gotchas:

• to my knowledge no client displays the OOB description along side the URL, so the description is placed in a message preceeding the OOB announcement
• when no description is attached, only the sender's name is sent in RemoteNickFormat style
• uploading HTTP files happens in the background so we don't block the task, but downloading files is still done in the foreground, which means the client could hang fixed
• are we handling multiple files in the same message correctly? yes we do

[selfhoster1312]

So right now the status is:

• receiving file uploads from XMPP works
• sending them only works with the media server enabled (because it provides the URL)

I need to implement OOB file upload when sending the attachments. Here's the status of that:

• find the XMPP file upload component
• get an upload token
• perform the HTTP upload
• send the message containing the new upload

Currently this depends on unreleased go-xmpp because of missing fields on the DiscoResult, see xmppo/go-xmpp#211

[selfhoster1312]

Not sure why golangci-lint did not find those issues locally but they're good catches. My Printf-style error should use the Errorf method instead of Error. As for wrapping errors… that's done with Log.WithErr(err).Warn… if i understand correctly.

Thank you CI gods <3

[poVoq]

Like discussed in the XMPP chat, it seems a bit odd to use the xmpp http_upload service for something the built in media-proxy was designed for, but on the other hand having the bot use the existing xmpp server features isn't bad per se and does simplify the setup significantly in some cases. For example some people run Matterbridge from their home networks with no option to publicly expose a http endpoint via the media-proxy.

[selfhoster1312]

it seems a bit odd to use the xmpp http_upload service for something the built in media-proxy was designed for

So if the xmpp bridge is about to send a file with an attachment, if that file already has a URL, it will not reupload it. However, on the receiving side i am not transmitting the URL. Maybe i should?

There's a tradeoff involved:

• using the 3rd party received URL may leak client IPs on the other side of the bridge (due to autopreview in clients)
• not using the 3rd party URL makes it impossible for the remote user to actually delete the content

I'm not sure what's the best approach here. Does anybody have a strong opinion? Otherwise, maybe this should just be a setting so that the bridge operator can choose for themselves.

Also noting that attachment handling is a very undocumented part of the codebase. I've added a few comments here and there in this PR, but i think this deserves a deeper conversation in the future so we don't ask those unfortunate questions on a bridge-by-bridge basis.

[poVoq]

A setting is good, but it should probably default to what requires the least additional setup, i.e. using the external http_upload in this case.

[selfhoster1312]

A setting is good, but it should probably default to what requires the least additional setup, i.e. using the external http_upload in this case.

Sorry if i wasn't clear. I meant when the xmpp bridge receives a remote HTTP attachment, it reuploads it on the bridge's file upload server. I was proposing a setting to disable that, which has nothing to do with the mediaserver. Or i could just disable the reupload entirely. Opinions?

The mediaserver integration is also a good question but i think it's a more general question of architecture with all the different bridges. I'll open an issue about this but i'm proposing we don't really answer it in this PR because it requires more overall thought.

[poVoq]

Ah ok. Yes that seems like a good privacy feature and is probably also useful to avoid some automated checks against hotloading images from the Discord cdn for example.

[selfhoster1312]

I rebased on #59 and it seems to work fine. MediaDownloadBlackList is also supported according to manual testing.

I just need to make sure about that FileInfo.Url thing before merging.

I would also like to have manual testing for #61 before merging #59, then these two.

[selfhoster1312]

Pushed a new commit using the new Bridge.HttpUpload helper 👍

[Wohlstand]

Quick question: did you the disguesting the fact the URL is owned by XMPP's HTTP file server or it's just a random URL shared from nowhere?

[selfhoster1312]

disguesting

Not sure what you mean sorry :D :D

If the question is about attachments from other bridges:

• the URL may be anything (discord URL, media server URL, etc)
• currently i'm always reuploading it to the chan's upload server but after some thinking i don't think this breaking change should be the default without further investigation
• if there is no URL provided (eg. mumble image upload), then it will be uploaded to the chan's upload server

If the question is about attachments coming from the XMPP channel:

• usually it will be a URL to the chan's HTTP upload server
• it may be actually anything else, it's just a URL basically

EDIT: no actually files are uploaded to the bot's HTTP upload server, not the chatroom's.

[selfhoster1312]

Ready for testing/review @poVoq

You can probably test the native file upload by using Discord's alwaysDownloadFiles. You can test that the "no upload component" warning works by further disabling the upload component on your vhost.

Looks good to me. There's still plenty of corner cases and profound questions about attachments and interop with other networks but i think it's deeper than XMPP integration and we should merge this then think about the bigger picture.

[poVoq]

Am I missing some newly necessary config option or so? Because right now if I try the version of this PR it just fails entirely to send files to XMPP. Files uploaded on Matrix only show the filename and an empty message on XMPP, and files uploaded on Discord just fail silently on XMPP fail silently when I don't add a text in Discord and when I add a text in discord, that text is shown twice on XMPP, but no image.

Nothing visible in the regular logs about this. It should probably show a warning or so when an upload fails.

(The version of this PR you shared a few weeks ago was working fine with this exact setup).

Also: please add a changelog entry and at least a note in the xmpp protocol documentation that a http_upload component for the bot account is required now.

[selfhoster1312]

Also: please add a changelog entry and at least a note in the xmpp protocol documentation that a http_upload component for the bot account is required now.

Very good point!!!!

when I don't add a text in Discord

Yeah discord attachments are broken. I don't think it's the fault of this PR. See #114 for discord attachment improvements (work in progress).

Files uploaded on Matrix only show the filename and an empty message on XMPP

That's not expected. I may have broken something by accident. Can you share debug logs maybe? I tested again xmpp<->xmpp is working fine on this branch, so except if it's a problem with captions (wihch we don't really support in XMPP), it might indicate a bug on the other bridge instead.

A quick read at how matrix handles upload indicates:

• media type detection may be wrong if the filename has no extension, which could explain the content not being displayed… but xmpp side it seems to be set correctly
• when there is no caption, the username is repeated once before every file upload, which is a little much if there's a lot of files
• you should have a lot more info in your debug logs :)

[selfhoster1312]

Rebased for testing. This will be the final round of review. Concerns about blocking uploads/downloads and message ID tracking will be addressed further, as deep discussions have taken place about these topics and work is under way.

[poVoq]

Sending image files from Discord or Matrix still result in an empty message on the XMPP side.

[poVoq]

Further testing: With the media-server turned off it is a bit better. There is still the empty message, but images shared from Matrix get uploaded to the XMPP http_upload correctly and shared as another message. However for images shared from Discord the situation is unchanged even with the media-server off.

The debug logs are hard to parse, but there is no error or warning as far as I can see.

[poVoq]

So there seem to be two separate issues the debug message for the failing Discord image share is this:

time="2026-03-01T18:38:13-01:00" level=debug msg="Announcing uploaded file to bridge-testing@chat.f-hub.org: text [D] desc `` urlhttps://cdn.discordapp.com/attachments/1449091024378003688/1477751831017951253/hanginthere.jpg?ex=69a5e723&is=69a495a3&hm=54835ea69d6e983f7058faeb6fc8d33cdb7063de42e501a3a74dc82675176fec&`" func=announceUploadedFile file="matterbridge/bridge/xmpp/helpers.go:46" prefix=xmpp`

Which is odd because it just repeats the Discord CDN link, while the working message for the Matrix image share is:

time="2026-03-01T18:36:55-01:00" level=debug msg="Announcing uploaded file to bridge-testing@chat.f-hub.org: text[m] desc `` urlhttps://uploads.f-hub.org/upload/16773380cf2acf27dcc9588b99232b9af23f6bb5/hf1k1rKgLLakCrqgmktKgQee7aoAH3vwuTr77n7V/onlyfix.jpg`" func=announceUploadedFile file="matterbridge/bridge/xmpp/helpers.go:46" prefix=xmpp`

Which uses the http_upload service as expected (as long as the media-server is off).

Edit: ok the lack of an Discord URL link, XMPP side seems to be unrelated to this PR.

[selfhoster1312]

So trying to reproduce the missing file uploads towards XMPP. Looks like it only happens on @poVoq 's server so far. I could not reproduce with two different prosody instance, nor with another ejabberd instance.

We have ruled out:

• CI caching producing wrong artifacts
• MUC server having an allowlist for OOB URLs
• a ejabberd specific thing (tested with another server)

[poVoq]

The problem was that my server had min_message_interval: 0.4 in the mod_muc settings as a spam prevention measure and that was triggered by the bot sending three messages in quick succession.

So it looks like this PR is finally good to merge.

[selfhoster1312]

Looks like the culprit was rate-limiting configured on @poVoq's server. No problem on this PR, everything is good!!!!!! <3