Skip to content

README_RELEASE_HELM_CHART.md

Latest commit

 

History

History
1214 lines (888 loc) · 45.9 KB

README_RELEASE_HELM_CHART.md

File metadata and controls

1214 lines (888 loc) · 45.9 KB

Table of contents

You can find the prerequisites to release Apache Airflow in README.md. This document details the steps for releasing Helm Chart.

Collect ambiguities during the release (for a follow-up doc PR)

These instructions are imperfect. Every release uncovers at least one command that has drifted, one step that is under-documented, or one automation that silently did the wrong thing. As you run through this document, jot down any such observations in a scratch file kept outside the repo (anywhere that is not tracked by git — a note in your home directory, a scratchpad, a gist). Once the release has landed, turn those notes into a follow-up PR against this document.

Keeping the scratch file out of the repo avoids accidentally committing release-manager notes along with the release-prep PR, and makes it obvious that the notes are input to the next doc PR rather than something to keep around long-term.

Perform review of security issues that are marked for the release

We are keeping track of security issues in the Security Issues repository currently. As a release manager, you should have access to the repository. Please review and ensure that all security issues marked for the release have been addressed and resolved. Ping security team (comment in the issues) if anything missing or the issue does not seem to be addressed.

Additionally, the dependabot alerts and code scanning alerts should be reviewed and security team should be pinged to review and resolve them.

Prepare the Apache Airflow Helm Chart Release Candidate

Pre-requisites

Instructions for installing the pre-requisites are explained in the steps below.

Set environment variables

  • Set environment variables
# Set Version
export VERSION=1.1.0
export VERSION_SUFFIX=rc1

# Set AIRFLOW_REPO_ROOT to the path of your git repo
export AIRFLOW_REPO_ROOT=$(pwd -P)

# Example after cloning
git clone https://github.com/apache/airflow.git airflow
cd airflow
export AIRFLOW_REPO_ROOT=$(pwd -P)

Setup k8s environment (mainly helm chart)

This will install Helm in the recent version and enter the environment where you can run helm commands and installs the necessary python dependencies (including towncrier in the k8s virtual environment).

breeze k8s setup-env
breeze k8s shell

Install the helm-gpg plugin, if you have not installed it already. This command will install the plugin using commit sha of version of the plugin that is known to work with latest openssl and reviewed by us.

helm plugin install https://github.com/technosophos/helm-gpg --version 6303407eb63deaeb1b2f24de611e3468a27ec05b

You can also update/uninstall/list the plugin with other helm plugin commands. For more information, run:

helm plugin --help

Build Release Notes

Before creating the RC, you need to build and commit the release notes for the release:

Preview with:

towncrier build --draft --version=${VERSION} --date=2021-12-15 --dir chart --config chart/newsfragments/config.toml

Then remove the --draft flag to have towncrier build the release notes for real.

The significant changes section does require some reformatting - look at prior releases as an example.

If no significant changes where added in this release, add the header and put "No significant changes.".

Then, get the rest of the entries, categorize them into the appropriate sections, and add it to the release notes.

git log --oneline helm-chart/1.1.0..main --pretty='format:- %s' -- chart/ docs/helm-chart/

Add changelog annotations to Chart.yaml

Once the release notes have been built, run the script to generate the changelog annotations.

./dev/chart/build_changelog_annotations.py

Verify the output looks right (only entries from this release), then put them below artifacthub.io/changes in Chart.yaml, replace previous change logs if a new release, append if another RC. Should look like for example:

annotations:
  artifacthub.io/changes: |
    - kind: added
      description: Add resources for `cleanup` and `createuser` jobs
      links:
        - name: "#19263"
          url: https://github.com/apache/airflow/pull/19263

Make sure that all the release notes changes are submitted as PR and merged. Changes in release notes should also automatically (via prek trigger updating of the reproducible_build.yaml) file which is uses to reproducibly build the chart package and source tarball.

You can leave the k8s environment now:

exit

Update minimum version of Kubernetes

The minimum version of Kubernetes should be updated according to https://github.com/apache/airflow/blob/main/README.md#requirements in two places:

Build RC artifacts

The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any modification than renaming – i.e. the contents of the files must be the same between voted release candidate and final release. Because of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix.

Make sure you have the upstream remote set up pointing to the apache git repo (per the standard convention upstream → apache/airflow, origin → your fork — see contributing-docs/10_working_with_git.rst). If needed, add it with:

git remote add upstream git@github.com:apache/airflow.git
git fetch upstream
  • We currently release Helm Chart from main branch:

For releasing 1.2x.0

git checkout apache/chart/v1-2x-test

For releasing 2.x.x and onwards

git checkout upstream/main
  • Clean the checkout: (note that this step will also clean any IDE settings you might have so better to do it in a checked out version you only use for releasing)
git clean -fxd
rm -rf dist/*

  • Verify your GPG signing key is ready.

    Before you spend 10+ minutes building artifacts only to discover that signing fails, run these checks once:

    # 1. The apache.org key has a secret signing subkey available locally.
gpg --list-secret-keys apache.org

# 2. Signing actually works (exits 0, writes a .asc, verifies cleanly).
echo test > /tmp/sign-check && \
    gpg --yes --armor --local-user apache.org \
        --output /tmp/sign-check.asc --detach-sig /tmp/sign-check && \
    gpg --verify /tmp/sign-check.asc /tmp/sign-check && \
    rm -f /tmp/sign-check /tmp/sign-check.asc && \
    echo "GPG signing OK"

# 3. The fingerprint of your signing (sub)key appears in the Airflow KEYS file.
#    Without this, PMC verifiers cannot validate the release.
FINGERPRINT=$(gpg --list-keys --with-colons apache.org | awk -F: '/^fpr:/ {print $10; exit}')
curl -fsS https://dist.apache.org/repos/dist/release/airflow/KEYS | \
    grep -q "${FINGERPRINT}" && echo "Key ${FINGERPRINT} is in KEYS" || \
    echo "MISSING: add your key to KEYS before releasing"

    If any of these fail, fix them before the build step. For first-time release managers, adding your key to the KEYS file is a separate PR against https://dist.apache.org/repos/dist/release/airflow/ (SVN).

    sign.sh defaults to SIGN_WITH=apache.org. If your apache.org uid resolves to multiple keys (rare), set SIGN_WITH explicitly to the fingerprint of the key you want to use.

  • Generate the source tarball:

breeze release-management prepare-helm-chart-tarball --version ${VERSION}

Note: The version suffix is only used for the RC tag and tag message. The version in the tarball is without the suffix, so the tarball can be released as-is when the vote passes.

  • Generate the binary Helm Chart release:
VERSION_SUFFIX= breeze release-management prepare-helm-chart-package --sign-email jedcunningham@apache.org

Note: we temporarily unset VERSION_SUFFIX when preparing the package, as we do not want it set and the flag defaults to the env var

Warning: you need the helm gpg plugin to sign the chart (instructions to install it above)

This should generate two files:

  • dist/airflow-${VERSION}.tgz
  • dist/airflow-${VERSION}.tgz.prov

The second one is a provenance file as described in https://helm.sh/docs/topics/provenance/. It can be used to verify integrity of the Helm chart.

  • Generate SHA512/ASC
pushd ${AIRFLOW_REPO_ROOT}/dist
${AIRFLOW_REPO_ROOT}/dev/sign.sh airflow-*.tgz airflow-*-source.tar.gz
popd

  • Move the artifacts to ASF dev dist repo, Generate convenience index.yaml & Publish them

    # First clone the repo
cd ${AIRFLOW_REPO_ROOT}/..
[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
svn update --set-depth=infinity asf-dist/dev/airflow

# Create new folder for the release
cd asf-dist/dev/airflow/helm-chart
svn mkdir ${VERSION}${VERSION_SUFFIX}

# Move the artifacts to svn folder
mv ${AIRFLOW_REPO_ROOT}/dist/airflow-${VERSION}.tgz* ${VERSION}${VERSION_SUFFIX}/
mv ${AIRFLOW_REPO_ROOT}/dist/airflow-chart-${VERSION}-source.tar.gz* ${VERSION}${VERSION_SUFFIX}/
cd ${VERSION}${VERSION_SUFFIX}

###### Generate index.yaml file - Start
# Download the latest index.yaml on Airflow Website
curl https://airflow.apache.org/index.yaml --output index.yaml

# Replace the URLs from "https://downloads.apache.org" to "https://archive.apache.org"
# as the downloads.apache.org only contains latest releases.
sed -i 's|https://downloads.apache.org/airflow/helm-chart/|https://archive.apache.org/dist/airflow/helm-chart/|' index.yaml

# Generate / Merge the new version with existing index.yaml
helm repo index --merge ./index.yaml . --url "https://dist.apache.org/repos/dist/dev/airflow/helm-chart/${VERSION}${VERSION_SUFFIX}"

###### Generate index.yaml file - End

# Commit the artifacts
svn add *
svn commit -m "Add artifacts for Helm Chart ${VERSION}${VERSION_SUFFIX}"

  • Remove old Helm Chart versions from the dev repo

    cd ..
export PREVIOUS_VERSION_WITH_SUFFIX=1.0.0rc1
svn rm ${PREVIOUS_VERSION_WITH_SUFFIX}
svn commit -m "Remove old Helm Chart release: ${PREVIOUS_VERSION_WITH_SUFFIX}"

  • Push Tag for the release candidate

    cd ${AIRFLOW_REPO_ROOT}
git push upstream tag helm-chart/${VERSION}${VERSION_SUFFIX}

Publish rc documentation

Documentation is an essential part of the product and should be made available to users. In our cases, documentation for the released versions is published in S3 bucket, and the site is kept in a separate repository - apache/airflow-site, but the documentation source code and build tools are available in the apache/airflow repository, so you need to run several workflows to publish the documentation. More details about it can be found in Docs README showing the architecture and workflows including manual workflows for emergency cases.

You should use the breeze command to publish the documentation. The command does the following:

  1. Triggers Publish Docs to S3.
  2. Triggers workflow in apache/airflow-site to refresh
  3. Triggers S3 to GitHub Sync
breeze workflow-run publish-docs --ref helm-chart/${VERSION}${VERSION_SUFFIX} --site-env staging helm-chart

Prepare issue for testing status of rc

Create an issue for testing status of the RC (PREVIOUS_RELEASE should be the previous release version (for example 1.4.0).

cat <<EOF
Status of testing of Apache Airflow Helm Chart ${VERSION}${VERSION_SUFFIX}
EOF

Content is generated with:

breeze release-management generate-issue-content-helm-chart \
--previous-release helm-chart/${PREVIOUS_VERSION_WITH_SUFFIX} --current-release helm-chart/${VERSION}${VERSION_SUFFIX}

Copy the URL of the issue.

Prepare Vote email on the Apache Airflow release candidate

Subject:

cat <<EOF
[VOTE] Release Apache Airflow Helm Chart ${VERSION} based on ${VERSION}${VERSION_SUFFIX}
EOF
if [ "$OS" = "Darwin" ]; then  # MacOS (BSD date)
  export VOTE_END_TIME=$(date -u -v "+${VOTE_DURATION_IN_HOURS}H" -v "+10M" +'%Y-%m-%d %H:%M')
else  # Linux
  export VOTE_END_TIME=$(date --utc -d "now + $VOTE_DURATION_IN_HOURS hours + 10 minutes" +'%Y-%m-%d %H:%M')
fi
export TIME_DATE_URL="to?iso=$(date --utc -d "now + 72 hours + 10 minutes" +'%Y%m%dT%H%M')&p0=136&font=cursive"

Body:

cat <<EOF
Hello Apache Airflow Community,

This is a call for the vote to release Helm Chart version ${VERSION}.

The release candidate is available at:
https://dist.apache.org/repos/dist/dev/airflow/helm-chart/${VERSION}${VERSION_SUFFIX}/

airflow-chart-${VERSION}-source.tar.gz - is the "main source release" that comes with INSTALL instructions.
airflow-${VERSION}.tgz - is the binary Helm Chart release.

Public keys are available at: https://www.apache.org/dist/airflow/KEYS

For convenience "index.yaml" has been uploaded (though excluded from voting), so you can also run the below commands.

helm repo add apache-airflow-dev https://dist.apache.org/repos/dist/dev/airflow/helm-chart/${VERSION}${VERSION_SUFFIX}/
helm repo update
helm install airflow apache-airflow-dev/airflow

airflow-${VERSION}.tgz.prov - is also uploaded for verifying Chart Integrity, though not strictly required for releasing the artifact based on ASF Guidelines.

$ helm gpg verify airflow-${VERSION}.tgz
gpg: Signature made Thu Jan  6 21:33:35 2022 MST
gpg:                using RSA key E1A1E984F55B8F280BD9CBA20BB7163892A2E48E
gpg: Good signature from "Jed Cunningham <jedcunningham@apache.org>" [ultimate]
plugin: Chart SHA verified. sha256:b33eac716e0416a18af89fb4fa1043fcfcf24f9f903cda3912729815213525df

The documentation is available at https://airflow.staged.apache.org/docs/helm-chart/${VERSION}/index.html.

The vote will be open for at least 72 hours ($VOTE_END_TIME UTC) or until the necessary number of votes is reached.

https://www.timeanddate.com/countdown/$TIME_DATE_URL

Please vote accordingly:

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason

Only votes from PMC members are binding, but members of the community are
encouraged to test the release and vote with "(non-binding)".

The test procedure for PMC members is described in:
https://github.com/apache/airflow/blob/main/dev/README_RELEASE_HELM_CHART.md#verify-the-release-candidate-by-pmc-members

The test procedure for contributors and members of the community who would like to test this RC is described in:
https://github.com/apache/airflow/blob/main/dev/README_RELEASE_HELM_CHART.md#verify-the-release-candidates-by-contributors

Consider this my (binding) +1.

For license checks, the .rat-excludes files are included, so you can run the following to verify licenses (just update your path to rat):

tar -xvf airflow-chart-${VERSION}-source.tar.gz
cd airflow-chart-${VERSION}
java -jar apache-rat-0.18.jar chart -E .rat-excludes

Please note that the version number excludes the \`rcX\` string, so it's now
simply ${VERSION}. This will allow us to rename the artifact without modifying
the artifact checksums when we actually release it.

The status of testing the Helm Chart by the community is kept here:

<TODO COPY LINK TO THE ISSUE CREATED>

Thanks,
<your name>
EOF

Note, you need to update the helm gpg verify output and verify the end of the voting period in the body.

Note, For RC2/3 you may refer to shorten vote period as agreed in mailing list thread.

Verify the release candidate by PMC members

The PMC members should verify the releases in order to make sure the release is following the Apache Legal Release Policy.

At least 3 (+1) votes should be recorded in accordance to Votes on Package Releases

The legal checks include:

  • checking if the packages are present in the right dist folder on svn
  • verifying if all the sources have correct licences
  • verifying if release manager signed the releases with the right key
  • verifying if all the checksums are valid for the release

SVN check

The files should be present in the sub-folder of Airflow dist - helm-chart

The following files should be present (7 files):

  • airflow-chart-${VERSION}-source.tar.gz + .asc + .sha512
  • airflow-{VERSION}.tgz + .asc + .sha512
  • airflow-{VERSION}.tgz.prov

Source tarball reproducibility check

The source tarball should be reproducible. This means that if you build it twice, you should get the same result. This is important for security reasons, as it ensures that the source code has not been tampered with.

  1. Go to airflow repository root (for example if you cloned it to ../airflow then cd ../airflow)
cd ../airflow
AIRFLOW_REPO_ROOT=$(pwd -P)
  1. Set the version of the release you are checking
VERSION=12.0.1
VERSION_SUFFIX=rc1
VERSION_RC=${VERSION}${VERSION_SUFFIX}
  1. Check-out the branch from which the release was made and cleanup dist folder:
git checkout helm-chart/${VERSION_RC}
rm -rf dist/*
  1. Build the source tarball and package. Since you are not releasing the package, you should ignore version check and skip tagging. There is no need to specify version as it is stored in Chart.yaml of the rc tag.
breeze release-management prepare-helm-chart-tarball --version-suffix ${VERSION_SUFFIX} --ignore-version-check --skip-tagging
VERSION_SUFFIX= breeze release-management prepare-helm-chart-package

Note

In case VERSION_SUFFIX is exported, we temporarily unset VERSION_SUFFIX when preparing the package, as we do not want it set and the flag defaults to the env var.

  1. Compare the produced tarball binary with ones in SVN: As a PMC member, you should be able to clone the SVN repository:
cd ${AIRFLOW_REPO_ROOT}/..
[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
svn update --set-depth=infinity asf-dist/dev/airflow

Or update it if you already checked it out:

cd asf-dist/dev/airflow
svn update .

Set an environment variable: SVN_REPO_ROOT to the root of folder where you have asf-dist checked out:

cd asf-dist/
export SVN_REPO_ROOT=$(pwd -P)
diff ${AIRFLOW_REPO_ROOT}/dist/airflow-chart-${VERSION}-source.tar.gz ${SVN_REPO_ROOT}/dev/airflow/helm-chart/${VERSION_RC}/airflow-chart-${VERSION}-source.tar.gz
diff ${AIRFLOW_REPO_ROOT}/dist/airflow-${VERSION}.tgz ${SVN_REPO_ROOT}/dev/airflow/helm-chart/${VERSION_RC}/airflow-${VERSION}.tgz

There should be no differences reported. If you see "binary files differ" message, it means that the source tarball is not reproducible. This is not a blocker for the release, you can unpack the sources of the tarball and compare the two tarballs and check the differences for example with diff -R <DIR1> <DIR2>. It could be that our reproducible build script is not working correctly yet, and we need to fix it (so checking the differences would be helpful also to find out what is wrong).

Before proceeding next you want to go to the SVN directory

cd ${SVN_REPO_ROOT}/dev/airflow/helm-chart/${VERSION_RC}

Licence check

You can run this command to do it for you (including checksum verification for your own security):

# Checksum value is taken from https://downloads.apache.org/creadur/apache-rat-0.18/apache-rat-0.18-bin.tar.gz.sha512
wget -q https://archive.apache.org/dist/creadur/apache-rat-0.18/apache-rat-0.18-bin.tar.gz -O /tmp/apache-rat-0.18-bin.tar.gz
echo "315b16536526838237c42b5e6b613d29adc77e25a6e44a866b2b7f8b162e03d3629d49c9faea86ceb864a36b2c42838b8ce43d6f2db544e961f2259e242748f4  /tmp/apache-rat-0.18-bin.tar.gz" | sha512sum -c -
tar -xzf /tmp/apache-rat-0.18-bin.tar.gz -C /tmp
  • Unpack the release source archive (the <package + version>-source.tar.gz file) to a folder
  • Enter the sources folder run the check
rm -rf /tmp/apache/airflow-helm-chart-src && mkdir -p /tmp/apache/airflow-helm-chart-src && tar -xzf ${SVN_REPO_ROOT}/dev/airflow/helm-chart/${VERSION_RC}/airflow-chart-*-source.tar.gz --strip-components 1 -C /tmp/apache/airflow-helm-chart-src
cp ${AIRFLOW_REPO_ROOT}/.rat-excludes /tmp/apache/airflow-helm-chart-src/.rat-excludes
java -jar /tmp/apache-rat-0.18/apache-rat-0.18.jar --input-exclude-file /tmp/apache/airflow-helm-chart-src/.rat-excludes /tmp/apache/airflow-helm-chart-src/ | grep -E "! |INFO: "

where .rat-excludes is the file in the root of Chart source code.

You should see no files reported as Unknown or with wrong licence and summary of the check similar to:

INFO: Apache Creadur RAT 0.18 (Apache Software Foundation)
INFO: Excluding patterns: .git-blame-ignore-revs, .github/*, .git ...
INFO: Excluding MISC collection.
INFO: Excluding HIDDEN_DIR collection.
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
INFO: RAT summary:
INFO:   Approved:  15615
INFO:   Archives:  2
INFO:   Binaries:  813
INFO:   Document types:  5
INFO:   Ignored:  2392
INFO:   License categories:  2
INFO:   License names:  2
INFO:   Notices:  216
INFO:   Standards:  15609
INFO:   Unapproved:  0
INFO:   Unknown:  0

There should be no files reported as Unknown or Unapproved. The files that are unknown or unapproved should be shown with a line starting with !.

For example:

! Unapproved:         1    A count of unapproved licenses.
! /CODE_OF_CONDUCT.md

Signature check

Make sure you have imported into your GPG the PGP key of the person signing the release. You can find the valid keys in KEYS.

You can import the whole KEYS file:

wget https://dist.apache.org/repos/dist/release/airflow/KEYS
gpg --import KEYS

You can also import the keys individually from a keyserver. The below one uses Kaxil's key and retrieves it from the default GPG keyserver OpenPGP.org:

gpg --keyserver keys.openpgp.org --receive-keys CDE15C6E4D3A8EC4ECF4BA4B6674E08AD7DE406F

You should choose to import the key when asked.

Note that by being default, the OpenPGP server tends to be overloaded often and might respond with errors or timeouts. Many of the release managers also uploaded their keys to the GNUPG.net keyserver, and you can retrieve it from there.

gpg --keyserver keys.gnupg.net --receive-keys CDE15C6E4D3A8EC4ECF4BA4B6674E08AD7DE406F

Once you have the keys, the signatures can be verified by running this:

for i in *.asc
do
   echo -e "Checking $i\n"; gpg --verify $i
done

This should produce results similar to the below. The "Good signature from ..." is indication that the signatures are correct. Do not worry about the "not certified with a trusted signature" warning. Most of the certificates used by release managers are self-signed, and that's why you get this warning. By importing the key either from the server in the previous step or from the KEYS page, you know that this is a valid key already. To suppress the warning you may edit the key's trust level by running gpg --edit-key <key id> trust and entering 5 to assign trust level ultimate.

Checking airflow-1.0.0.tgz.asc
gpg: assuming signed data in 'airflow-1.0.0.tgz'
gpg: Signature made Sun 16 May 01:25:24 2021 BST
gpg:                using RSA key CDE15C6E4D3A8EC4ECF4BA4B6674E08AD7DE406F
gpg:                issuer "kaxilnaik@apache.org"
gpg: Good signature from "Kaxil Naik <kaxilnaik@apache.org>" [unknown]
gpg:                 aka "Kaxil Naik <kaxilnaik@gmail.com>" [unknown]
gpg: WARNING: The key's User ID is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: CDE1 5C6E 4D3A 8EC4 ECF4  BA4B 6674 E08A D7DE 406F

Checking airflow-chart-1.0.0-source.tar.gz.asc
gpg: assuming signed data in 'airflow-chart-1.0.0-source.tar.gz'
gpg: Signature made Sun 16 May 02:24:09 2021 BST
gpg:                using RSA key CDE15C6E4D3A8EC4ECF4BA4B6674E08AD7DE406F
gpg:                issuer "kaxilnaik@apache.org"
gpg: Good signature from "Kaxil Naik <kaxilnaik@apache.org>" [unknown]
gpg:                 aka "Kaxil Naik <kaxilnaik@gmail.com>" [unknown]
gpg: WARNING: The key's User ID is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: CDE1 5C6E 4D3A 8EC4 ECF4  BA4B 6674 E08A D7DE 406F

SHA512 sum check

Run this:

for i in *.sha512
do
    echo "Checking $i"; shasum -a 512 `basename $i .sha512 ` | diff - $i
done

You should get output similar to:

Checking airflow-1.0.0.tgz.sha512
Checking airflow-chart-1.0.0-source.tar.gz.sha512

Verify the release candidates by Contributors

Contributors can run below commands to test the Helm Chart against any Kubernetes cluster they already have configured. Replace the version with the one being voted on:

helm repo add apache-airflow-dev https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.1.0rc1/
helm repo update
helm install airflow apache-airflow-dev/airflow

You can then perform any other verifications to check that it works as you expected by upgrading the Chart or installing by overriding default of values.yaml.

End-to-end verification with Breeze and KinD

If you don't already have a Kubernetes cluster handy, the breeze k8s commands give you a clean room: a fresh KinD cluster plus the chart deployed from your local checkout. This runs the same chart files that were packaged into the RC tarball, so it doubles as a chart-content sanity check.

  1. Check out the RC tag so the local chart/ directory is the candidate under vote:

    git checkout helm-chart/${VERSION_RC}

  2. Create the cluster, configure namespaces, build a k8s-ready airflow image, and load it into KinD. On main these are bundled into a single breeze k8s deploy-cluster (which also installs the shared k8s tooling — kind, kubectl, helm — into Breeze's pinned virtualenv, i.e. it includes the work breeze k8s setup-env would do); on older tags, run setup-env plus the constituent commands in sequence:

    breeze k8s deploy-cluster --rebuild-base-image                  # main
# or, on older tags:
breeze k8s setup-env
breeze k8s create-cluster
breeze k8s configure-cluster
breeze ui compile-assets  # ensures the UI is built so the API server can serve index.html
breeze k8s build-k8s-image --rebuild-base-image
breeze k8s upload-k8s-image

  3. Helm-install the chart into the cluster:

    breeze k8s deploy-airflow

    The command prints the forwarded localhost:<port> URL of the airflow API server (default credentials admin / admin).

  4. Confirm the deployment is healthy:

    breeze k8s status                            # cluster + connection
curl -s http://localhost:<port>/api/v2/monitor/health
curl -s http://localhost:<port>/api/v2/version

    You want all four metadatabase, scheduler, triggerer, dag_processor health entries to be healthy, and the version endpoint to report the airflow version pinned by the chart's values.yaml.

  5. Open the forwarded URL in your browser and poke around the UI — automated health checks pass even when the UI assets fail to load or auth is broken, so a human eyeball over the running deployment is worth the 30 seconds. The URL breeze k8s deploy-airflow printed in step 3 (http://localhost:<port>) lands on the login page; sign in with admin / admin. Suggested manual sanity checks:

    • the login page renders (UI assets served correctly);
    • after login, the Dags list loads and shows the example Dags bundled with the image;
    • open one example Dag, trigger a run, and watch a task instance reach success (exercises scheduler → triggerer → executor and log retrieval end-to-end);
    • the Admin → Connections and Admin → Variables pages render without backend errors.

    If the forwarded port has been reclaimed (e.g. you re-ran the deploy or your shell rotated), breeze k8s status re-prints the current URL.

  6. Tear down the cluster when you're done:

    breeze k8s delete-cluster

Switch back to your working branch (git checkout <branch>) before running any other release-prep commands.

Publish the final release

Re-establish your shell like when prepared

# Set Version
export VERSION=1.1.0
export VERSION_SUFFIX=rc1

# Set AIRFLOW_REPO_ROOT to the path of your git repo
export AIRFLOW_REPO_ROOT=$(pwd -P)

breeze k8s setup-env
breeze k8s shell

Summarize the voting for the release

Once the vote has been passed, you will need to send a result vote to dev@airflow.apache.org:

Subject:

[RESULT][VOTE] Release Apache Airflow Helm Chart 1.1.0 based on 1.1.0rc1

Message:

Hello all,

The vote to release Apache Airflow Helm Chart version 1.1.0 based on 1.1.0rc1 is now closed.

The vote PASSED with 4 binding "+1", 4 non-binding "+1" and 0 "-1" votes:

"+1" Binding votes:

  - Kaxil Naik
  - Jarek Potiuk
  - Ash Berlin-Taylor
  - Xiaodong Deng

"+1" Non-Binding votes:

  - Jed Cunningham
  - Ephraim Anierobi
  - Dennis Akpenyi
  - Ian Stanton

Vote thread:
https://lists.apache.org/thread.html/r865f041e491a2a7a52e17784abf0d0f2e35c3bac5ae8a05927285558%40%3Cdev.airflow.apache.org%3E

I'll continue with the release process and the release announcement will follow shortly.

Thanks,
<your name>

Publish release to SVN

You need to migrate the RC artifacts that passed to this repository: https://dist.apache.org/repos/dist/release/airflow/helm-chart/ (The migration should include renaming the files so that they no longer have the RC number in their filenames.)

The best way of doing this is to svn cp between the two repos (this avoids having to upload the binaries again, and gives a clearer history in the svn commit logs):

# First clone the repo
cd ${AIRFLOW_REPO_ROOT}/..
[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
svn update --set-depth=infinity asf-dist/release/airflow

# Create new folder for the release
cd asf-dist/release/airflow/helm-chart
export AIRFLOW_SVN_RELEASE_HELM=$(pwd -P)
svn mkdir ${VERSION}
cd ${VERSION}

# Move the artifacts to svn folder & commit (don't copy or copy & remove - index.yaml)
for f in ../../../../dev/airflow/helm-chart/${VERSION}${VERSION_SUFFIX}/*; do svn cp $f $(basename $f); done
svn rm index.yaml
svn commit -m "Release Airflow Helm Chart Check ${VERSION} from ${VERSION}${VERSION_SUFFIX}"

Verify that the packages appear in Airflow Helm Chart.

Publish release tag

Create and push the release tag:

cd "${AIRFLOW_REPO_ROOT}"
git checkout helm-chart/${VERSION}${VERSION_SUFFIX}
git tag -s helm-chart/${VERSION} -m "Apache Airflow Helm Chart ${VERSION}"
git push upstream helm-chart/${VERSION}

Publish final documentation

Documentation is an essential part of the product and should be made available to users. In our cases, documentation for the released versions is published in S3 bucket, and the site is kept in a separate repository - apache/airflow-site, but the documentation source code and build tools are available in the apache/airflow repository, so you need to run several workflows to publish the documentation. More details about it can be found in Docs README showing the architecture and workflows including manual workflows for emergency cases.

You should use the breeze command to publish the documentation. The command does the following:

  1. Triggers Publish Docs to S3.
  2. Triggers workflow in apache/airflow-site to refresh
  3. Triggers S3 to GitHub Sync
breeze workflow-run publish-docs --ref helm-chart/${VERSION} --site-env live helm-chart

Update index.yaml in airflow-site

The index.yaml file on the Airflow website allows users to easily add the Apache Airflow Helm repository with:

helm repo add apache-airflow https://airflow.apache.org

To update index.yaml for the newly released chart version, follow these steps:

git clone https://github.com/apache/airflow-site.git airflow-site
cd airflow-site
curl https://dist.apache.org/repos/dist/dev/airflow/helm-chart/${VERSION}${VERSION_SUFFIX}/index.yaml -o index.yaml
cp ${AIRFLOW_SVN_RELEASE_HELM}/${VERSION}/airflow-${VERSION}.tgz .
helm repo index --merge ./index.yaml . --url "https://downloads.apache.org/airflow/helm-chart/${VERSION}"
rm airflow-${VERSION}.tgz
mv index.yaml landing-pages/site/static/index.yaml
git checkout -b feature/new-chart-release-${VERSION}
git add .
# Note: commit will re-add the license at the top, need to add another time
git commit -m "Add Apache Airflow Helm Chart Release ${VERSION} to chart index file"
git add .
git commit -m "Add Apache Airflow Helm Chart Release ${VERSION} to chart index file"
git push
# and finally open a PR

When the PR is merged, the Build docs workflow in airflow-site is triggered automatically by the push to main and will deploy the updated site.

You can check the latest chart version that is in the index.yaml with:

curl -s https://airflow.apache.org/index.yaml | yq '.entries.airflow[0].version' -

You can see when ArtifactHUB will next check for a new version by logging in and going to the ArtifactHUB control panel.

Wait for ArtifactHUB to discover new release

As we link out to ArtifactHUB in all of our release communications, we now wait until ArtifactHUB has discovered the new release. This can take 60 minutes or longer to happen after the index change PR from above is merged.

Notify developers of release

Subject:

cat <<EOF
[ANNOUNCE] Apache Airflow Helm Chart version ${VERSION} Released
EOF

Body:

cat <<EOF
Dear Airflow community,

I am pleased to announce that we have released Apache Airflow Helm chart ${VERSION} 🎉 🎊

The source release, as well as the "binary" Helm Chart release, are available:

📦   Official Sources: https://airflow.apache.org/docs/helm-chart/${VERSION}/installing-helm-chart-from-sources.html
📦   ArtifactHub: https://artifacthub.io/packages/helm/apache-airflow/airflow
📚   Docs: https://airflow.apache.org/docs/helm-chart/${VERSION}/
🚀   Quick Start Installation Guide: https://airflow.apache.org/docs/helm-chart/${VERSION}/quick-start.html
🛠️   Release Notes: https://airflow.apache.org/docs/helm-chart/${VERSION}/release_notes.html

Thanks to all the contributors who made this possible.

Cheers,
<your name>
EOF

Send the same email to announce@apache.org, except change the opening line to Dear community,. It is more reliable to send it via the web ui at https://lists.apache.org/list.html?announce@apache.org (press "c" to compose a new thread)

Send announcements about security issues fixed in the release

The release manager should review and mark as READY all the security issues fixed in the release. Such issues are marked as affecting < <JUST_RELEASED_VERSION> in the CVE management tool at https://cveprocess.apache.org/. Then the release manager should announced the issues via the tool.

Once announced, each of the issue should be linked with a 'reference' with tag 'vendor advisory' with the URL to the announcement published automatically by the CVE management tool. Note that the announce@apache.org is moderated, and the link to the email thread will not be published immediately, that's why it is recommended to add the link to users@airflow.apache.org which takes usually few seconds to be published after the CVE tool sends them.

The ASF Security will be notified and will submit to the CVE project and will set the state to 'PUBLIC'.

Add release data to Apache Committee Report Helper

Add the release data (version and date) at: https://reporter.apache.org/addrelease.html?airflow

Update Announcements page

Update "Announcements" page at the Official Airflow website

Create release on GitHub

Create a new release on GitHub with the release notes and assets from the release svn.

  • Create a new release in https://github.com/apache/airflow/releases/new ("Draft new release" button)
  • Select the tag, e.g. helm-chart/1.20.0
  • Use title: Apache Airflow Helm Chart 1.20.0
  • Copy the release notes RST from the release. Need to convert all headings with ^^^^^ as underline to H1 in markdown with a # as prefix and all headings with """"" as underline to H2 in markdown with a ## prefix.
  • Use binaries (all chart files, signatures and source tar) from the SVN
  • Click on "Publish release"

Close the milestone

Use: https://github.com/apache/airflow/milestones

Before closing the milestone on GitHub, make sure that all PR marked for it are either part of the release (was cherry picked) or postponed to the next release, then close the milestone. Create the next one if it hasn't been already (it probably has been). Update the new milestone in the Currently we are working on issue make sure to update the last updated timestamp as well.

Close the testing status issue

Don't forget to thank the folks who tested and close the issue tracking the testing status.

Thank you everyone. New Helm-Chart is released.

I invite everyone to help improve the chart for the next release, a list of open issues can be found [here](https://github.com/apache/airflow/issues?q=is%3Aissue%20state%3Aopen%20label%3Aarea%3Ahelm-chart).

Update issue template with the new release

Updating issue templates in .github/ISSUE_TEMPLATE/1-airflow_bug_report.yml.yml with the new version

Announce the release on the community slack

Post this in the #announce channel:

cat <<EOF
We've just released Apache Airflow Helm Chart ${VERSION} 🎉

📦 ArtifactHub: https://artifacthub.io/packages/helm/apache-airflow/airflow
📚 Docs: https://airflow.apache.org/docs/helm-chart/${VERSION}/
🚀 Quick Start Installation Guide: https://airflow.apache.org/docs/helm-chart/${VERSION}/quick-start.html
🛠 Release Notes: https://airflow.apache.org/docs/helm-chart/${VERSION}/release_notes.html

Thanks to all the contributors who made this possible.
EOF

Announce about the release in social media

Announcement is done from official Apache-Airflow accounts.

Make sure attach the release image generated with Figma to the post. If you don't have access to the account ask a PMC member to post.

Tweet and post on Linkedin about the release:

cat <<EOF
We've just released Apache Airflow Helm chart ${VERSION} 🎉

📦 ArtifactHub: https://artifacthub.io/packages/helm/apache-airflow/airflow
📚 Docs: https://airflow.apache.org/docs/helm-chart/${VERSION}/
🛠️ Release Notes: https://airflow.apache.org/docs/helm-chart/${VERSION}/release_notes.html

Thanks to all the contributors who made this possible.
EOF

Bump chart version in Chart.yaml

Bump the chart version to the next version in chart/Chart.yaml in main. Do not add -dev suffix to the version.

Remove old releases

We should keep the old version a little longer than a day or at least until the updated index.yaml is published. This is to avoid errors for users who haven't run helm repo update.

It is probably ok if we leave last 2 versions on release svn repo too.

# http://www.apache.org/legal/release-policy.html#when-to-archive
cd cd ${AIRFLOW_REPO_ROOT}/asf-dist/release/airflow/helm-chart
export PREVIOUS_VERSION=1.0.0
svn rm ${PREVIOUS_VERSION}
svn commit -m "Remove old Helm Chart release: ${PREVIOUS_VERSION}"

Additional processes

Fixing released documentation

Sometimes we want to rebuild the documentation with some fixes that were merged in main branch, for example when there are html layout changes or typo fixes, or formatting issue fixes.

In this case the process is as follows:

  • When you want to re-publish helm-chart/X.Y.Z docs, create (or pull if already created) helm-chart/X.Y.Z-docs branch
  • Cherry-pick changes you want to add and push to the main apache/airflow repo
  • Run the publishing workflow.

In case you are releasing latest released version of helm-chart (which should be most of the cases), run this:

breeze workflow-run publish-docs --site-env live --ref helm-chart/X.Y.Z-docs \
   --skip-tag-validation \
   helm-chart

In case you are releasing an older version of helm-chart, you should skip writing to the stable folder

breeze workflow-run publish-docs --site-env live --ref helm-chart/X.Y.Z-docs \
   --skip-tag-validation \
   --skip-write-to-stable-folder \
   helm-chart