Make CDN purging more robust

[tobias]

We purge files from the CDN on deploy to prevent stale maven-metadata.xml files (and their checksum files). But we do this in a future to not delay the deploy, and there is no error handling around that. So if it fails we don't know about it and there is stale data in the cache.

We should:

• track failures
• add retry logic

[zerg000000]

Any hints on resolving this issue?

Here is a few directions I can think of

1. submit to in memory job queue, a background thread to purge the files. We can do error handling and retry, we can have more fine-grained control, like de-duplication / rate limit / throttle in this way. would need some refactoring.
2. add retry logic in the future. Get the job done, but don't have much control.
3. using resilience4j? less home baked retry logic, but we have one more dependency and at least two more components.

[tobias]

Hi @zerg000000! Thanks for taking a look at this.

I think it would be fine to do a simple retry in the future (option 2). I've seen no Sentry reports of purge failures since adding reporting, so I think failures here are rare.

A simple backoff/retry strategy would work. I think we should still report each failure to sentry even when we retry.