[Packaging] Use CentOS Stream 8 to build `el8` RPM package #20918

jiasli · 2022-01-06T13:12:38Z

Description

#11204 added el8 for RHEL 8 but was reverted by #11325.

As CentOS 7's RPM (4.11.3) can't generate SHA256 digests and causes failure on FIPS-enabled RHEL system (#20719), we have to use CentOS Stream 8's RPM (4.14.3) to build.

Now we can see Payload SHA256 digest is calculated as expected:

$ docker run -it --rm quay.io/centos/centos:stream8

# curl 'https://artprodcus3.artifacts.visualstudio.com/A0fb41ef4-5012-48a9-bf39-4ee3de03ee35/29ec6040-b234-4e31-b139-33dc4287b756/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2F6dXJlLXNkay9wcm9qZWN0SWQvMjllYzYwNDAtYjIzNC00ZTMxLWIxMzktMzNkYzQyODdiNzU2L2J1aWxkSWQvMTI3ODY5MC9hcnRpZmFjdE5hbWUveXVtLWNlbnRvc19zdHJlYW040/content?format=file&subPath=%2Fazure-cli-2.32.0-1.el8.x86_64.rpm' -o azure-cli.rpm

# rpm --checksig --verbose azure-cli.rpm
azure-cli.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK

References

CentOS 8 has been deprecated, so we shouldn't use it anymore. See https://wiki.centos.org/About/Product
CentOS Stream 8 docker image is distributed as quay.io/centos/centos:stream8 at https://quay.io/repository/centos/centos?tab=tags. See https://wiki.centos.org/FAQ/CentOSStream

jiasli · 2022-01-06T13:13:00Z

scripts/release/rpm/Dockerfile.centos_stream8

+ARG cli_version=dev
+
+RUN yum update -y
+RUN yum install -y wget rpm-build gcc libffi-devel python3-devel openssl-devel make bash diffutils patch dos2unix python3-virtualenv perl


coreutils is not compatible with preinstalled coreutils-single, so we don't need to install coreutils.

# dnf list installed | grep coreutils coreutils-single.x86_64 8.30-12.el8 @baseos # yum install -y coreutils Last metadata expiration check: 0:13:00 ago on Thu Jan 6 09:04:58 2022. Error: Problem: problem with installed package coreutils-single-8.30-12.el8.x86_64 - package coreutils-8.30-12.el8.x86_64 conflicts with coreutils-single provided by coreutils-single-8.30-12.el8.x86_64 - package coreutils-8.30-12.el8.x86_64 conflicts with coreutils-single provided by coreutils-single-8.30-10.el8.x86_64 - package coreutils-8.30-12.el8.x86_64 conflicts with coreutils-single provided by coreutils-single-8.30-8.el8.x86_64 - package coreutils-8.30-12.el8.x86_64 conflicts with coreutils-single provided by coreutils-single-8.30-9.el8.x86_64 - cannot install the best candidate for the job (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

perl is not installed by rpm-build on CentOS Stream 8, so we should manually install it.

jiasli · 2022-01-06T13:17:24Z

For unknown reason, the rpm builder on CentOS Stream 8 is extremely slow at step Processing files: azure-cli-2.32.0-1.el8.x86_64:

https://dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_apis/build/builds/1278426/logs/507

2022-01-06T10:46:32.3684420Z [0mProcessing files: azure-cli-2.32.0-1.el8.x86_64
2022-01-06T11:19:00.3310021Z Provides: azure-cli = 2.32.0-1.el8 azure-cli(x86-64) = 2.32.0-1.el8 config(az

I thought this could be related to compression:

$ docker run -it --rm quay.io/centos/centos:stream

# yum install -y rpm-build

# rpm --showrc | grep _binary_payload
-13: _binary_payload    w2.xzdio

# grep -R _binary_payload /usr/lib/rpm/
/usr/lib/rpm/macros:#%_binary_payload   w9.gzdio
/usr/lib/rpm/redhat/macros:%_binary_payload w2.xzdio

but the the result is exactly the same on centos:7, while the rpm-build on latest fedora uses zstd: https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression

$ docker run -it --rm fedora:36

# yum install -y rpm-build

# rpm --showrc | grep _binary_payload
-13: _binary_payload    w19.zstdio

# grep -R _binary_payload /usr/lib/rpm/
/usr/lib/rpm/macros:#%_binary_payload   w9.gzdio
/usr/lib/rpm/redhat/macros:%_binary_payload w19.zstdio

jiasli · 2022-01-06T14:19:55Z

@glaubitz, just wondering if you have any experience with the above problem (#20918 (comment))? Any insights are greatly appreciated. 😊

glaubitz · 2022-01-10T10:10:45Z

@glaubitz, just wondering if you have any experience with the above problem (#20918 (comment))? Any insights are greatly appreciated. blush

Did you check the process list during this step? Or maybe iotop or strace?

jiasli · 2022-02-09T07:06:42Z

azure-pipelines.yml


- job: BuildYumPackageMariner
-  displayName: Build Yum Package Mariner
+- job: BuildRpmPackageMariner


RPM is the actual package and YUM is the package manager (see https://packages.microsoft.com/yumrepos/azure-cli/). YUM has been replaced by DNF in many modern systems. So RPM should be the correct name here, like DEB (package) vs APT (package manager).

jiasli · 2022-02-09T07:08:07Z

scripts/release/rpm/pipeline.sh

+# mkdir -p ${BUILD_STAGINGDIRECTORY}/docker
+# docker save azure/azure-cli:${IMAGE}-builder | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_${IMAGE}-builder.tar.gz &
+# docker save azure/azure-cli:${IMAGE} | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_${IMAGE}.tar.gz &
+# wait


Actually we never rehydrated these files. We disable them to save space on ADO.

azure-pipelines.yml

jiasli · 2022-02-09T07:55:16Z

azure-pipelines.yml

+  displayName: Test Rpm Package
  timeoutInMinutes: 120
-  dependsOn: BuildYumPackage
+  dependsOn: BuildRpmPackageCentOS7


As discussed in

Add support for 'dependsOn' for individual matrix jobs microsoft/azure-pipelines-yaml#420

Add cross-product matrix strategy microsoft/azure-pipelines-yaml#20

https://developercommunity.visualstudio.com/t/support-for-dependson-for-individual-matrix-jobs/1007464

dependsOn only works for the base job (all matrixes), and it can't work on a specific matrix. Since CentOS Stream 8's rpm builder is extremely slow for some reason, TestRpmPackage will wait for it to finish, making the whole pipeline much slower.

Here we make 2 deferent jobs:

BuildRpmPackageCentOS7

BuildRpmPackageCentOSStream8

so that TestRpmPackage can start right after BuildRpmPackageCentOS7 finishes, like before.

Co-authored-by: ZelinWang <[email protected]>

jiasli · 2022-02-09T09:00:51Z

I tried to verify the RPM on a RHEL 8.4 VM without FIPS:

$ curl 'https://artprodcus3.artifacts.visualstudio.com/A0fb41ef4-5012-48a9-bf39-4ee3de03ee35/29ec6040-b234-4e31-b139-33dc4287b756/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2F6dXJlLXNkay9wcm9qZWN0SWQvMjllYzYwNDAtYjIzNC00ZTMxLWIxMzktMzNkYzQyODdiNzU2L2J1aWxkSWQvMTM1NTA2My9hcnRpZmFjdE5hbWUvcnBtLWNlbnRvc19zdHJlYW040/content?format=file&subPath=%2Fazure-cli-2.33.0-1.el8.x86_64.rpm' -o azure-cli.rpm

$ rpm --checksig --verbose azure-cli.rpm
azure-cli.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK

After enabling FIPS following https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening#switching-the-system-to-fips-mode_using-the-system-wide-cryptographic-policies, MD5 digest: OK is no loner shown:

$ rpm --checksig --verbose azure-cli.rpm
azure-cli.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
$

Not sure if this is expected.

jiasli · 2022-02-09T11:32:17Z

After code signing, we got

$ rpm --checksig --verbose azure-cli-2.33.0-1.el8.x86_64.rpm
azure-cli-2.33.0-1.el8.x86_64.rpm:
    Header V4 RSA/SHA256 Signature, key ID be1229cf: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA256 Signature, key ID be1229cf: OK

$ sudo yum localinstall azure-cli-2.33.0-1.el8.x86_64.rpm
...
Installed:
  azure-cli-2.33.0-1.el8.x86_64

Complete!

This indicate the package now

has SHA256 so can be installed on FIPS-enabled machines
is correctly signed

archoversight · 2022-02-17T18:13:51Z

\o/

jiasli · 2022-03-11T03:53:05Z

azure-pipelines.yml

+- job: BuildRpmPackageCentOS7
+  displayName: Build Rpm Package CentOS 7
+  # Uncomment this line to disable this job on Pull Requests
+  # condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual', 'Schedule'))


This line was removed by #20608. Bring it back as a comment.

el8

74c2205

jiasli commented Jan 6, 2022

View reviewed changes

yonzhan assigned jiasli Jan 6, 2022

yonzhan added this to the Jan 2022 (2022-02-08) milestone Jan 6, 2022

yonzhan requested review from calvinhzy and evelyn-ys January 6, 2022 13:21

jiasli mentioned this pull request Jan 6, 2022

Failure on SLES 15 SP3 VM: No module named azure.cli.__main__ #20839

Open

update

521ceea

jiasli requested review from ahelal and fabnord January 6, 2022 15:12

jiasli changed the title ~~{Packaging} Use CentOS Stream 8 to build el8 RPM package~~ [Packaging] Use CentOS Stream 8 to build el8 RPM package Jan 6, 2022

jiasli requested review from chalamalasetty, chasewilson, mfrw and yonzhan January 7, 2022 02:07

yonzhan approved these changes Jan 7, 2022

View reviewed changes

yonzhan modified the milestones: Jan 2022 (2022-02-08), Feb 2022 (2022-03-01) Jan 29, 2022

jiasli added 3 commits February 9, 2022 10:51

Merge remote-tracking branch 'azure/dev' into el8

756e664

Add comment

36d3389

comment out docker save

ec44277

jiasli marked this pull request as ready for review February 9, 2022 05:58

jiasli requested review from kairu-ms and wangzelin007 as code owners February 9, 2022 05:58

jiasli mentioned this pull request Feb 9, 2022

[Packaging] Add SHA256 digest for RPM #20852

Closed

jiasli added 2 commits February 9, 2022 14:59

Rename yum to rpm

3ca5d20

Merge remote-tracking branch 'azure/dev' into el8

7b5a07e

jiasli commented Feb 9, 2022

View reviewed changes

wangzelin007 reviewed Feb 9, 2022

View reviewed changes

azure-pipelines.yml Outdated Show resolved Hide resolved

Do not use matrix

340c4be

jiasli commented Feb 9, 2022

View reviewed changes

Update azure-pipelines.yml

0b6287c

Co-authored-by: ZelinWang <[email protected]>

yonzhan modified the milestones: Feb 2022 (2022-03-01), Mar 2022 (2022-04-05) Feb 26, 2022

jiasli added 2 commits March 11, 2022 11:41

Merge remote-tracking branch 'azure/dev' into el8

e88e373

update

7a6e98c

jiasli force-pushed the el8 branch from edf621b to 7a6e98c Compare March 11, 2022 03:52

jiasli commented Mar 11, 2022

View reviewed changes

Update test_rpm_in_docker.sh

5050e7e

jiasli force-pushed the el8 branch from 49aa05d to 5050e7e Compare March 11, 2022 08:05

mfrw approved these changes Mar 13, 2022

View reviewed changes

calvinhzy approved these changes Mar 15, 2022

View reviewed changes

wangzelin007 approved these changes Mar 15, 2022

View reviewed changes

jiasli merged commit cd39b08 into Azure:dev Mar 15, 2022

jiasli deleted the el8 branch March 15, 2022 07:38

jiasli mentioned this pull request Mar 16, 2022

{Packaging} Use Red Hat Universal Base Image 8 to build el8 RPM package #21655

Merged

This was referenced Mar 25, 2022

{Packaging} Not build Mariner RPM on PullRequest CI #21800

Merged

rpmbuild 4.17.0 is slow and stuck at Processing files for a long time #21820

Open

{Packaging} Revert BuildRpmPackageCentOS7.ArtifactName to yum #21887

Merged

[Packaging] Use CentOS Stream 8 to build el8 RPM package #20918

[Packaging] Use CentOS Stream 8 to build el8 RPM package #20918

Uh oh!

Conversation

jiasli commented Jan 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

References

Uh oh!

jiasli Jan 6, 2022

Choose a reason for hiding this comment

Uh oh!

jiasli Jan 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiasli commented Jan 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiasli commented Jan 6, 2022

Uh oh!

glaubitz commented Jan 10, 2022

Uh oh!

jiasli Feb 9, 2022

Choose a reason for hiding this comment

Uh oh!

jiasli Feb 9, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jiasli Feb 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiasli commented Feb 9, 2022

Uh oh!

jiasli commented Feb 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

archoversight commented Feb 17, 2022

Uh oh!

jiasli Mar 11, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

[Packaging] Use CentOS Stream 8 to build `el8` RPM package #20918

[Packaging] Use CentOS Stream 8 to build `el8` RPM package #20918

jiasli commented Jan 6, 2022 •

edited

Loading

jiasli Jan 6, 2022 •

edited

Loading

jiasli commented Jan 6, 2022 •

edited

Loading

jiasli Feb 9, 2022 •

edited

Loading

jiasli commented Feb 9, 2022 •

edited

Loading