Binary support for s390x

[pandurangkhandeparker]

Thank you for contributing to Velero!

Signed-off-by: Pandurang Alias Aradhya Khandeparker [email protected]

Please add a summary of your change

This PR aims to add s390x support to Velero binary.

Does your change fix a particular issue?

Fixes #2622

Please indicate you've done the following:

• Accepted the DCO. Commits without the DCO will delay acceptance.
• Created a changelog file or added /kind changelog-not-required as a comment on this pull request.
• Updated the corresponding documentation in site/content/docs/main.

[reasonerjt]

IMHO, given we don't have resources or bandwidth to verify on zLinux, shall we let this remain downstream?

[rposts]

@reasonerjt If access to s390x system to run tests is an issue then IBM now has a public program for requesting virtual machines for s390x. A member of the Velero team can put in a request via the following form. When approved and sign-up is complete resources can be set up and delegated as appropriate. https://www.ibm.com/community/z/open-source/virtual-machines-request/

Please let us know what is required for s390x binary verification.

It will be great for s390x to be part of several ecosystem that Velero already supports which includes ppc64le and arm as well.

[kaovilai]

Are we open to unsupported binary release (at your own risk)?

[rposts]

Are we open to unsupported binary release (at your own risk)?

I think we should try to maintain fidelity with other archs - at a minimum s390x binary should pass associated testcases .

[kaovilai]

at a minimum s390x binary should pass associated testcases

How do you propose velero obtain the infra to test this? Running in github actions emulated would be very slow.

[rposts]

at a minimum s390x binary should pass associated testcases

How do you propose velero obtain the infra to test this? Running in github actions emulated would be very slow.
@kaovilai - pls see above "..A member of the Velero team can put in a request via the following form. When approved and sign-up is complete resources can be set up and delegated as appropriate. https://www.ibm.com/community/z/open-source/virtual-machines-request/ "

This is a real s390x VM which can be used to run the testcases.

[kaovilai]

If it can be added as a github actions runner maybe. That way is not another separate test pipeline.. only maintainers can add new hosted runners to the repo tho.

[kaovilai]

It's not just access, but lack of maintainers bandwidth to prioritize signing up for a new account and hooking it up to CI.

If you have a fork that produces binaries I think it'll be an easier pill to swallow for maintainers to link to your fork for an unofficial s390x binary.

Tho personally I think unverified binary is ok upstream.

[rposts]

@kaovilai Thanks for following up on this PR. For this iteration, is it possible to release artifacts similar to this ppc64le implementation? I am unable to determine ppc64le CI is running extensive tests.

[sseago]

@kaovilai Thanks for following up on this PR. For this iteration, is it possible to release artifacts similar to this ppc64le implementation? I am unable to determine ppc64le CI is running extensive tests.

Looks like that added build support for ppc64le but explicitly disabled publishing docker images. So I think this means the testing infra does not test on this env, but it allows users to build easily from a checkout. I'm not 100% sure though.

[rposts]

@kaovilai Thanks for following up on this PR. For this iteration, is it possible to release artifacts similar to this ppc64le implementation? I am unable to determine ppc64le CI is running extensive tests.

Looks like that added build support for ppc64le but explicitly disabled publishing docker images. So I think this means the testing infra does not test on this env, but it allows users to build easily from a checkout. I'm not 100% sure though.

@sseago It seems to publish binaries as well.

[kaovilai]

See if this would work for you in the meantime.

docker container create --name velero-s390x --platform=linux/s390x registry.redhat.io/oadp/oadp-velero-rhel9:1.4.0 && \
docker cp velero-s390x:/velero ./velero && \
docker container rm velero-s390x && \
docker image rm registry.redhat.io/oadp/oadp-velero-rhel9:1.4.0

you can see which tags are available at https://catalog.redhat.com/software/containers/oadp/oadp-velero-rhel9/64510133c1647dae8134a158

[rposts]

See if this would work for you in the meantime.

docker container create --name velero-s390x --platform=linux/s390x registry.redhat.io/oadp/oadp-velero-rhel9:1.4.0 && \
docker cp velero-s390x:/velero ./velero && \
docker container rm velero-s390x && \
docker image rm registry.redhat.io/oadp/oadp-velero-rhel9:1.4.0

you can see which tags are available at https://catalog.redhat.com/software/containers/oadp/oadp-velero-rhel9/64510133c1647dae8134a158

@kaovilai Yes - that works. However, this effort is to ensure we have artifacts available here as well.

[rposts]

@kaovilai let me know how we can assist further in enabling s390x artifacts. Seems like we have overcome technical issues.

[kaovilai]

@reasonerjt Is it okay to ship unverified s390x binary? Without adding test infra, change is trivial. We can tag it velero/velero:s390x-community? WDYT?

[kaovilai]

@rposts @pandurangkhandeparker

1. Pr has conflicts to be resolved.
2. https://github.com/linux-on-ibm-z/vmware-tanzu-velero/ could host the binaries on ghcr.io or other registries. If that is acceptable, we can certainly link it from velero.io That would be easier to swallow I’m assuming.
3. Let’s wait for others who have some authority on velero/velero repo to chime in if you want this PR pushed here. We should probably also add builds for windows/arm64 since that’s a thing now.

[weshayutin]

@pandurangkhandeparker I would suggest folks from IBM join the community calls if they want Z builds.
IMHO there is NOT a large enough install base of customers with Z to warrent any upstream work here. 0/

[rposts]

@kaovilai

1. Pr has conflicts to be resolved.
   I'll get my team to look into it.
2. https://github.com/linux-on-ibm-z/vmware-tanzu-velero/ could host the binaries on ghcr.io or other registries. If that is acceptable, we can certainly link it from velero.io That would be easier to swallow I’m assuming.
   In my opinion having artifacts hosted here with other archs will be more compelling for end users than hosting it elsewhere.

[kaovilai]

having artifacts hosted here

that implies upstream have enough bandwidth to test/verify. Unless we clarify that we don't test all or subset of published images.

The only way to get it added without verification is to make it clear that the published binaries and images are "as-is".

[rposts]

having artifacts hosted here

that implies upstream have enough bandwidth to test/verify. Unless we clarify that we don't test all or subset of published images.

The only way to get it added without verification is to make it clear that the published binaries and images are "as-is".

I believe we are already doing that for other archs other than amd64? I am unable to see e2e test results for these archs.

[kaovilai]

I believe we are already doing that for other archs other than amd64?

There are other cases NOT in github actions (GHA) pull requests tests such as nightlies which uses more resouces on cloud providers than would be possible via kind cluster on GHA.

@danfengliu do you know which arches are being verified in nightlies.

[kaovilai]

ICYMI PR tests only a subset of e2e tests filtered by labels.

[rposts]

got it - tx @kaovilai

[rposts]

@kaovilai unfortunately the developer who had raised this PR is no longer available to resolve the PR conflicts. I will be happy to get this reworked via different PR if it makes sense. Tx.

[kaovilai]

@rposts If you feel this work is important, and would like to discuss more, I invite you to join our community meetings.

Bi-weekly community meeting alternating every week between Beijing Friendly timezone and EST/Europe Friendly Timezone (Google Calendar, iCal)

The Beijing timezone one would get you in front of more folks who maintain the DockerHub repo.

[Repana-Chowdappa]

Hi @kaovilai @rposts - I am part of IBM Cloudpak on Z team, even we are also looking for this velero support on s390x platform for CP4AIOP's product. We have a customer requirement to enable AIOP's which has dependency on velero-v1.12.4-linux-s390x.tar.gz equavalent to velero-v1.12.4-linux-amd64.tar.gz.

Please let us know if anything required in enabling s390x artifacts here. We are able to build binaries locally using this repo, so we don't see any technical challenges.

[kaovilai]

@Repana-Chowdappa the challenges are not technical in nature. I suggest joining the next Beijing community call if you want this.

[rposts]

@kaovilai Resurrecting this thread - I tried joining couple of community meetings (China friendly time zone) - Jan 28th/Feb11 2025 - however it seemed that meetings did not take place (or I was not allowed to join). My connection timed out after 10 minutes. Is there any other way to plead s390x case differently?

I also noticed from the comments above that there is growing interests to have this available on s390x and perhaps now is the time to reconsider it. Community can already access s390x VM instance as noted earlier.

Thanks for your ongoing support.

[kaovilai]

@rposts Jan 28th were skipped due to Chinese new year

Feb 11th were technical issue with the link, so @reasonerjt created new link to use and communicated in slack.

We will use new link for meeting going forward at least the CN timezone one

You can join this slack via https://slack.k8s.io/

We will get hackmd docs updated.

Community meetings remain the place to get maintainers attention on development/release changes if it cannot be done asyncronously effectively.

[ranjeetsingh-23]

Hi @kaovilai ,
Any idea as when this PR will be merged and velero cli support for 390x be available?
Thanks,

[reasonerjt]

@ranjeetsingh-23 I'm discussing with @rposts on slack.

[reasonerjt]

Approved per discussion with @rposts, who confirmed that he will help with the s390x specific issues.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.56%. Comparing base (e5c7c7f) to head (2a2621c).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7505   +/-   ##
=======================================
  Coverage   59.56%   59.56%           
=======================================
  Files         370      370           
  Lines       40239    40239           
=======================================
  Hits        23969    23969           
  Misses      14771    14771           
  Partials     1499     1499           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kaovilai]

do we want this in for 1.16?

[Repana-Chowdappa]

do we want this in for 1.16?

Hi @kaovilai ,
Yes, we need this ASAP, so 1.16.0 would be good.

[Repana-Chowdappa]

Hi @kaovilai - Can we please merge this PR?

[kaovilai]

looks like 1.16 branch was cut.