Add all previous minor stable versions to current minor fast and stable

[sdodson]

Today there's much confusion and uncertainty about the upgrade process
due to multiple factors. This may lead some people to switch to the next
channel before they're able to upgrade to the next minor. If they were on
a version that's not in the next channel they may be presented no upgrade
paths even if application of an additional z-stream update would unlock
a minor upgrade.

Hopefully this shouldn't be necessary when we push 4.7 out because the CVO
now knows which additional channels contain the current version enablign the
console to only present those channels when appropriate discouraging people
from changing prematurely.

/cc @wking

[sdodson]

I thought about doing the same for 4.3 for benefit of the ~ 130 4.2 clusters that are still out there, but 4.2 is EOL and now even 4.3 is EOL.

Additionally, adding older 4.2 into the 4.3 channels will require many additional blocks to be created because our edge management was much different back then. If we choose to pursue that we need to very carefully ensure that we're not introducing new edges, for instance 4.3.1 does include edges from 4.2.16 which would not currently be present in the graph but would if we were to backfill all stable 4.2.z into 4.3 channels.

[wking]

Comparing:

$ for i in 3 4 5; do for weight in candidate fast stable; do FROM="${weight}-4.${i}"; TO="${weight}-4.$((i+1))"; diff -U0 --label "${FROM}" --label "${TO}" <(yaml2json < "channels/${FROM}.yaml" | jq -r '.versions[]' | grep "^4[.]${i}[.]" | sort -V) <(yaml2json < "channels/${TO}.yaml" | jq -r '.versions[]' | grep "^4[.]${i}[.]" | sort -V); done; done
--- candidate-4.3
+++ candidate-4.4
@@ -1,7 +0,0 @@
-4.3.0
-4.3.0-0.hotfix-2020-09-30-133631
-4.3.0-rc.0
-4.3.0-rc.3
-4.3.1
-4.3.2
-4.3.3
@@ -15,3 +8,3 @@
-4.3.14
-4.3.15
-4.3.17
+4.3.14+amd64
+4.3.15+amd64
+4.3.17+amd64
--- fast-4.3
+++ fast-4.4
@@ -2 +1,0 @@
-4.3.0-0.hotfix-2020-09-30-133631
--- stable-4.3
+++ stable-4.4
@@ -2 +1,0 @@
-4.3.0-0.hotfix-2020-09-30-133631
--- candidate-4.4
+++ candidate-4.5
@@ -1,17 +0,0 @@
-4.4.0
-4.4.0-rc.0
-4.4.0-rc.1
-4.4.0-rc.2
-4.4.0-rc.4
-4.4.0-rc.6
-4.4.0-rc.7
-4.4.0-rc.8
-4.4.0-rc.9
-4.4.0-rc.10
-4.4.0-rc.11
-4.4.0-rc.12
-4.4.0-rc.13
-4.4.2
-4.4.3
-4.4.4
-4.4.5
@@ -19 +1,0 @@
-4.4.7
--- candidate-4.5
+++ candidate-4.6
@@ -1,14 +0,0 @@
-4.5.0
-4.5.0-0.hotfix-2020-08-24-185832
-4.5.0-rc.1
-4.5.0-rc.2
-4.5.0-rc.4
-4.5.0-rc.5
-4.5.0-rc.6
-4.5.0-rc.7
-4.5.1
-4.5.1-rc.0
-4.5.2
-4.5.3
-4.5.4
-4.5.5
--- fast-4.5
+++ fast-4.6
@@ -1 +0,0 @@
-4.5.0-0.hotfix-2020-08-24-185832
@@ -10,0 +10 @@
+4.5.10
@@ -11,0 +12 @@
+4.5.12
--- stable-4.5
+++ stable-4.6
@@ -1,15 +0,0 @@
-4.5.0-0.hotfix-2020-08-24-185832
-4.5.1
-4.5.2
-4.5.3
-4.5.4
-4.5.5
-4.5.6
-4.5.7
-4.5.8
-4.5.9
-4.5.11
-4.5.13
-4.5.14
-4.5.15
-4.5.16

But this is a step in the direction I want. However, I don't like violating candidate -> fast -> stable. Can we include these in candidate too to avoid:

$ diff -U0 --label candidate-4.3 --label fast-4.3 <(yaml2json <channels/candidate-4.4.yaml | jq -r '.versions[]' | sort -V) <(yaml2json <channels/fast-4.4.yaml | jq -r '.versions[]' | sort -V) | grep '^+\|---'
--- candidate-4.3
+++ fast-4.3
+4.3.0
+4.3.1
+4.3.2
+4.3.3

and such?

[wking]

#!/usr/bin/env python3

import functools
import re

import yaml


SEMVER = re.compile('^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$')


def version_key(v, minor):
    match = SEMVER.match(v)
    groups = match.groupdict()
    v_minor = int(groups['minor'])
    v_patch = int(groups['patch'])
    if v_minor != minor:
        v_patch = -v_patch
    return (int(groups['major']), v_minor, v_patch, v)


data = {}
minors = [3,4,5]
weights = ['candidate', 'fast', 'stable']
for minor in minors:
    data[minor] = {}
    for weight in weights:
        with open('channels/{}-4.{}.yaml'.format(weight, minor)) as f:
            data[minor][weight] = yaml.safe_load(f)

for minor, next_minor in zip(minors, minors[1:]):
    for weight in weights:
        versions = set(data[minor][weight]['versions'])
        versions.union(v for v in data[next_minor][weight]['versions'] if v.startswith('4.{}.'.format(minor)))
        data[minor][weight]['versions'] = sorted(versions, key=functools.partial(version_key, minor=minor))

for minor in minors:
    for weight in weights:
        with open('channels/{}-4.{}.yaml'.format(weight, minor), 'w') as f:
            yaml.safe_dump(data[minor][weight], f, default_flow_style=False)

Strips comments (which I'm fine with), but I think something like this will do what I want.

[sdodson]

yeah i'm fine with ensuring that candidate always includes what's in fast and i'm fine with dropping all comments, don't know if i'll get to update this soon though, biggest thing for me is I'd like to get 4.6 channels (minus stable-4.6) squared away ASAP

[wking]

Ah, 4.6 will need prompting forward a minor. I'll update my script...

[wking]

I bumped my script and pushed a fixup to your branch. If it looks good, squash it in.

[wking]

although we might want to run the script fresh on master to pick up #508 and similar.

[sdodson]

Can you fill out fast-4.6 with 4.5.z versions as well? We'll withold from stable-4.6 until upgrades are ready to go stable.

[sdodson]

@wking PTAL again even though most changes came from you :-)

[LalatenduMohanty]

/lgtm
/hold for @wking to take a look

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LalatenduMohanty, sdodson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [LalatenduMohanty,sdodson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[LalatenduMohanty]

/hold cancel

[wking]

Some follow-up in #549