Velero AWS plugin: S3 compatibility broken with Qumulo in recent versions

[Mise]

Hello,

We are attempting to integrate Velero and Velero plugin for AWS with Qumulo storage, which the vendor claims is fully compatible with AWS S3.

However, it seems that the requests made by the Velero AWS plugin to our storage include parameters that it does not recognize. For example, some requests fail due to the inclusion of the x-id parameter, for example:

GET https://storage.domain.tld:443/bucketname/backups/test-backup-qumulo/test-backup-qumulo-volumeinfo.json.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=blablaaws4_request&X-Amz-Date=20241015T093024Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=blabla"

Other requests, such as PutObject requests, fail similarly. The storage vendor asserts that x-id parameter is not part of the S3 specification, and I could not find any information about it in the S3 API reference.

We tried Velero versions 1.13.2 and 1.14.1 and AWS plugin versions 1.9.0 and 1.10, which exhibit the same behavior, and a combination of older Velero v1.12.4 with AWS plugin v1.8.2, which work correctly.

Could you please advise why newer Velero and plugin versions are including the x-id parameter in their S3 requests, and whether there is a way to configure Velero to omit this parameter when communicating with our S3 storage?

Thank you!

[kaovilai]

dupe of #8265

[kaovilai]

bump to aws-sdk-go-v2 forced addition of some headers and drop usage of older "legacy" (as aws determined) apis.

Everyone who claims S3 compatible are only via testing/replicating functionality, which can get out of date and eventually incompatible with what aws uses overtime.

[kaovilai]

"x-id" is an expected part of requests made by aws-sdk-go-v2
https://github.com/aws/aws-sdk-go-v2/blob/04e7aca073a0a7ed479aa37cad88a1cf58a979a1/service/s3/internal/customizations/presign_test.go#L35-L41

[blackpiglet]

/assign @kaovilai

[kzakhark]

"x-id" is an expected part of requests made by aws-sdk-go-v2 https://github.com/aws/aws-sdk-go-v2/blob/04e7aca073a0a7ed479aa37cad88a1cf58a979a1/service/s3/internal/customizations/presign_test.go#L35-L41

Is there any indication of why it is an expected part, as there's no mention of this parameter in the API reference?

[kaovilai]

Ask aws sdk go v2 team. We don't know.

[kaovilai]

aws-sdk-go v1 is expected to go into unsupported state in mid 2025. You should ask Qumulo if they would eventually implement support for aws-sdk-go-v2 or they have an alternative SDK they prefer us to use.

[kaovilai]

@blackpiglet this should be labeled Area/Storage/s3-compatible not aws :)

[kaovilai]

I'm assuming it's for debugging with customers where aws only have to ask for request URL, and not method.

More people understand URL than HTTP request methods (PUT/GET)

[reasonerjt]

@Mise

which the vendor claims is fully compatible with AWS S3.
So it seems it's not b/c the AWS SDK does not work with it.

Could you test using the velero with the v1.8.x version of velero-plugin-for-s3?