SuSE/RPM merge specfiles and only build a single rpm for RH/SuSE

[philfry]

This PR contains the following changes:

• merge rpm and SuSE specfiles
• remove zypper repository file, analogous to #1202
• remove SuSE build script
• re-add lost %dir %{_datadir}/%{name}

I'm not sure how the publishing works, though. Probably I'll need a little help here.

Testing done

# import the rpm signing key and announce it to rpm
gpg --import credentials/test.secret.gpg
echo '%_gpg_name Bogus Test (This is test only key) <[email protected]>' > ~/.rpmmacros

# remove the msi thingy
sed -i '/^export MSI/d' setup.mk

# get the war
export ver="2.414.1"
wget -c https://get.jenkins.io/war-stable/${ver}/jenkins.war

# build the rpm
WAR=${PWD}/jenkins.war make rpm BRAND=./branding/jenkins.mk BUILDENV=./env/release.mk VERSION=${ver}

# as I use podman instead of docker I needed some minor adaptions
sed -i 's/docker/podman/' requirements.txt molecule/default/molecule.yml
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# get the dokken images
podman pull docker.io/dokken/{oraclelinux-{8,9},opensuse-leap-15}:latest

# run the tests
molecule test -p oraclelinux-8
molecule test -p oraclelinux-9
molecule test -p opensuse-leap-15

PLAY RECAP *********************************************************************
oraclelinux-8              : ok=16   changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
oraclelinux-9              : ok=10   changed=4    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0
opensuse-leap-15           : ok=16   changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

### Submitter checklist
- [x] Make sure you are opening from a **topic/feature/bugfix branch** (right side) and not your main branch!
- [x] Ensure that the pull request title represents the desired changelog entry
- [x] Please describe what you did
- [x] Link to relevant issues in GitHub or Jira
- [x] Link to relevant pull requests, esp. upstream and downstream changes
- [x] Ensure you have provided tests - that demonstrates feature works or fixes the issue

Fixes: #419

[basil]

This looks really great! To get rid of the suse.publish target in the Makefile, the suse/publish/publish.sh script, and the SUSE_URL, SUSEDIR, and SUSE_WEBDIR environment variables, we just need to get rid of the SUSE stage from https://github.com/jenkins-infra/release/blob/8951254c4a415d24d81b8b9c351b9bb6a07408f8/Jenkinsfile.d/core/package#L226-L248 and rename the existing "Red Hat" stage to something more generic like "RPM", as well as doing the same thing in https://github.com/jenkinsci/jenkins/blob/ba4690c5a461d3c7b4f375912c6c7ff7b02323ee/.github/workflows/publish-release-artifact.yml#L177-L211. Note that while the jenkinsci/jenkins and jenkinsci/packaging repositories have unstable weekly (e.g., master) and stable LTS (e.g., stable-2.414) branches, jenkins-infra/release only has a single branch that is used for all releases, implying that for jenkins-infra/release we should either retain the SUSE stage for LTS releases but skip it on weekly releases (doable, but requires some conditional logic) or backport this PR to the stable-2.414 of jenkinsci/packaging and then remove SUSE entirely from jenkins-infra/release and both the master and stable-2.414 branches of jenkinsci/jenkins. I have no strong preference either way, but it seems like less work in the long term to do the backport and be done with this SUSE stuff sooner rather than later.

A final (but very minor) point is that there was previously a nonsensical "1.2" version for the jenkins-2.373-1.2.noarch.rpm (SUSE) RPM, which seemed to exist only to differentiate it from the "1.1" version used by the jenkins-2.373-1.1.noarch.rpm (Red Hat) RPM. Since we're now combining these into one product, I wonder if it wouldn't be too hard to remove the "-1.1" from the (sole remaining) RPM, which would make it match our Debian product (e.g. jenkins_2.373_all.deb). The "1.1" has no meaning in terms of Jenkins versioning. See #345.

[philfry]

A final (but very minor) point is that there was previously a nonsensical "1.2" version for the jenkins-2.373-1.2.noarch.rpm (SUSE) RPM, which seemed to exist only to differentiate it from the "1.1" version used by the jenkins-2.373-1.1.noarch.rpm (Red Hat) RPM. Since we're now combining these into one product, I wonder if it wouldn't be too hard to remove the "-1.1" from the (sole remaining) RPM, which would make it match our Debian product (e.g. jenkins_2.373_all.deb). The "1.1" has no meaning in terms of Jenkins versioning. See #345.

An RPM is required to have a release number, so we cannot skip it completely. I changed it simply to "1" now.

[philfry]

To get rid of the suse.publish target in the Makefile, the suse/publish/publish.sh script, and the SUSE_URL, SUSEDIR, and SUSE_WEBDIR environment variables, we just need to get rid of the SUSE stage from https://github.com/jenkins-infra/release/blob/8951254c4a415d24d81b8b9c351b9bb6a07408f8/Jenkinsfile.d/core/package#L226-L248 […]

Sounds good.
I checked whether or not I could use the redhat repo out of the box with openSUSE 15. Even tough it gives me a warning:

Warning: File 'repomd.xml' from repository 'Jenkins' is unsigned.

the redhat rpm installed without problems.
So probably before removing /opensuse/repodata or redirecting it to /redhat/repodata (guessing), it's probably a good idea to let rpm/publish/publish.sh sign the repomd.md as well.

But isn't the suse publisher still needed for the indexGenerator.py to provide at least the installation instructions for zypper? Sure, we could also include the zypper instructions on the redhat (or "generic rpm") page.

[basil]

Sure, we could also include the zypper instructions on the redhat (or "generic rpm") page.

Yes, that would work just fine. The more we can simplify things, the better, and having one fewer variant to maintain sounds like a great simplification.

[philfry]

Disclaimer: work-in-progress-commit, I expect it to break some tests. Also some things still need a little polish.

The build and publish scripts of openSUSE and Red Hat have been merged into a common rpm target.

Notable changes for openSUSE:

• zypper config no longer included in the rpm
• initv scripts no longer included in the rpm
• logrotate config no longer included in the rpm

Notable changes for Red Hat:

• repomd.xml is now signed and the repo file has repo_gpgcheck enabled to be openSUSE compilant

As the suse and redhat targets have been united to rpm the webserver and repo paths need to be adapted. I think it would be a good idea to install redirects from /suse and /redhat to /rpm once published.

Tested with:

# remove previous docroot
rm -rf pkg.jenkins.io

# import the rpm signing key and announce it to rpm
gpg --import credentials/test.secret.gpg
echo '%_gpg_name Bogus Test (This is test only key) <[email protected]>' > ~/.rpmmacros

# get the war
export ver="2.414.1"
wget -c https://get.jenkins.io/war-stable/${ver}/jenkins.war

# build the rpm and publish
WAR=${PWD}/jenkins.war MSI=/dev/null \
	make rpm.publish \
	BRAND=./branding/jenkins.mk VERSION=${ver} \
	GPG_KEYNAME="Bogus Test" CREDENTIAL=${PWD}/credentials/test.mk PKGSERVER=${USER}@localhost \
	WEBSERVER=localhost

# run test webserver
WAR=/dev/null MSI=/dev/null make test.local.setup

[dduportal]

Hello @philfry , are you still willing to work on this contribution?

[philfry]

Oh, I completely forgot about this one.
Yes, I can try to catch up the changes made to master.

[philfry]

The PR should be fine, even all checks have passed (yay!). Feel free to test.

[dduportal]

Many thanks @philfry! We are on huge infra changes (jenkins-infra/helpdesk#4774 and then jenkins-infra/helpdesk#3705 for info) which might predates your PR here (unless we deem this PR helpful to simplify the current publishing process). Mentioning in case you have no feedback in the upcoming weeks.

As the suse and redhat targets have been united to rpm the webserver and repo paths need to be adapted. I think it would be a good idea to install redirects from /suse and /redhat to /rpm once published.

Thanks for detailing this! We'll have to coordinate on the jenkins-infra side with this PR, but that helps a lot. I believe we'll prepare draft PR on the infra as code to prepare these redirects setups ahead of time.

[dduportal]

As described in jenkins-infra/helpdesk#4774 (comment), we'll integrate this change first (before any other changes) as it greatly simplifies and helps us on our packaging work on the infrastructure.

The prerequisites on the infrastructure side:

• Map this change in jenkins-infra/release master branch used to release the weekly releases (no impact on LTS as it uses another branch stable-2.xxx) to unify tghe Redhat and Suse pipeline stages (https://github.com/jenkins-infra/release/blob/fe568e4a06b76dc73fd5d1b049f2b1cfd847c6a8/Jenkinsfile.d/core/package#L203-L248) as mentioned by Basil above
• Prepare the /rpm/ directory on the pkg.jenkins.io service. I'm not unsure about how to initially fill it:
  • Should it be empty? But then redirections from /opensuse or /redhat would break for distributions using pinned older versions, or the package index update would loose the older versions?
  • Or shouldn't it be a copy of redhat assuming we may loose opensuse older versions? (we could the other way around but redhats users are more represented that opensuse users as far as I can tell)?
• Prepare adaptation of the monitoring to the new /rpm prefix for weeklies

[dduportal]

Prepare the /rpm/ directory on the pkg.jenkins.io service. I'm not unsure about how to initially fill it:

* Should it be empty? But then redirections from `/opensuse` or `/redhat` would break for distributions using pinned older versions, or the package index update would loose the older versions?

* Or shouldn't it be a copy of `redhat` assuming we may loose `opensuse` older versions? (we could the other way around but redhats users are more represented that opensuse users as far as I can tell)?

@timja @basil @daniel-beck @MarkEWaite I propose to try with an empty /rpm directory for this weekly without redirections and advertise both RedHat / SuSe user to adapt their (yum/rpm/zypper) configuration to point to /rpm.
=> it would avoid breaking existing users, but they would not see the new weekly version of Jenkins to appear if they don't take action.

I'm wondering what strategy to adopt for LTS though (on 10 december): should we do the same (at the risk of users complaining they don't see the .3 LTS until they change their RPM package URL)? Or should we, instead, make things breaks explicitly to push forward?

[timja]

Is it possible to redirect requests for both to the new one? or at least redhat?

Would be better if users were transparently upgraded.

We can try with empty weekly initially anyway

[dduportal]

Is it possible to redirect requests for both to the new one? or at least redhat?

Yes, it would be the same as putting a symlink (except redirect is better as the client is informed of the redirect). I can prepare to have both.

Would be better if users were transparently upgraded.

Yup, transparent upgrade looks better for me as well. But at the expense of installations with a pinned version if they reinstall the pinned old version since the new index might be regenerated without the "old" packages.

We can try with empty weekly initially anyway

I think we are inline: packaging change can always be reverted tomorrow (and RH/SuSe packages regenerated) without any consequence. I'm taking care of backuping the current pkg.jenkins.io content.

@timja WDYT if we keep providing the "old" endpoints under a new name to cover the users with "pinned" versions so they can always change their repo. URL if they do not want to upgrade?

That would looks like the following:

• pkg.jenkins.io/rpm serves the new (signed) RPM index but only provides versions >=2.537
• pkg.jenkins.io/opensuse -> redirects to pkg.jenkins.io/rpm
• pkg.jenkins.io/redhat -> redirects to pkg.jenkins.io/rpm
• pkg.jenkins.io/opensuse-legacy serves the old package index (currently under pkg.jenkins.io/opensuse before the redirections) and only provides versions under <2.537 for Opensuse
• pkg.jenkins.io/redhat-legacy serves the old package index (currently under pkg.jenkins.io/opensuse before the redirections) and only provides versions under <2.537 for RedHat

=> I propose we "try" for tomorrow's release, see the result, and we'll discuss for LTS. Does it look good?

[timja]

Yes sounds good

[dduportal]

Yes sounds good

See jenkins-infra/helpdesk#4774 (comment). I'm finishing all PRs and I'll proceed to merge

[dduportal]

Lets roll: jenkins-infra/helpdesk#4774 (comment)

[dduportal]

Released with success!

Note: 3 (hot)fixes were required, but they were hard to be tested until the publication (mainly due to the infrastructure legacy complexity):

• The repomd.xml file was initially not found because it is expected in the "RPM website" directory, not in the "RPM" directory (first one is only package metadatas, second one is where the binaries/RPMs are put to be deployed to mirrors) => dec339d
• The file repomd.xml.key is mandatory on OpenSuse with the GPG armored public key inside it. It was missing after the PR because was not set for rpm/redhat before. Fixed by adding it => bd2ac3a
• Fixup of commit above, along with unifying the shell code and ensuring that jenkins.repo specifies the repomd.xml.key as public GPG key (to potentially get rid of all the duplicated jenkins*.key in the future): 5851fba

[dduportal]

Many thanks @philfry for your contribution!

[dduportal]

Additional discovery: this PR did implicitly brought #409 to openSUSE installers (ref. https://issues.jenkins.io/browse/JENKINS-76296)