feat(ci): automate release process

[milinddethe15]

What this PR does / why we need it:
To release a newer version of trainer, user has to run make release VERSION=1.0.0 GITHUB_TOKEN=<token> and open PR with the generated commit.

• Release PR check: validate semver, ensure tag doesn’t exist and verify manifests, chart version, and Python API version match VERSION.
• Release workflow: create release branch/tag, build Python API dist, publish it to PyPI (requires PYPI_API_TOKEN secret in repo) and create a GitHub release using git-cliff-generated changelog.

This methods ensures release PR can be created by anyone and multiple maintainers can approve a release by LGTM on PR.

More detail in: #3148 (comment)

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #2155

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Pull Request Test Coverage Report for Build 21821785755

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 2 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.8%) to 51.998%

---

Files with Coverage Reduction	New Missed Lines	%
pkg/runtime/framework/plugins/registry.go	2	0.0%

Totals
Change from base Build 21715897523:	0.8%
Covered Lines:	1288
Relevant Lines:	2477

---

💛 - Coveralls

[jaiakash]

/retest

[Krishna-kg732]

curious: the SDK release workflow uses OIDC trusted publishing for PyPI (no secrets needed), but this PR uses PYPI_API_TOKEN. Was there a specific reason for choosing the API token approach over trusted publishing? Just want to understand the tradeoff — both work fine for our release cadence, but OIDC avoids managing secrets

[Krishna-kg732]

Existing release branches use release-X.Y format (release-1.9, release-2.0, release-2.1), per issue #2155. This will create release-2.2.0 instead. Should be :

Suggested change

	BRANCH="release-${VERSION}"
	MAJOR_MINOR=$(echo "$VERSION" | cut -d. -f1,2)
	BRANCH="release-${MAJOR_MINOR}"

[Krishna-kg732]

Suggested change

	# Copyright 2024 The Kubeflow Authors.
	# Copyright 2026 The Kubeflow Authors.

[Krishna-kg732]

Suggested change

	sed -i "s/__version__ = \".*\"/__version__ = \"$NEW_VERSION\"/" "$PYTHON_API_VERSION_FILE"
	echo "Updated Python API version to $NEW_VERSION"

$PYTHON_API_VERSION_FILE is git-added but never modified by the script. The init.py version won't be updated, and check-release.yaml will fail on the mismatch. Add before git add:

[milinddethe15]

@Krishna-kg732 It's a draft PR and If you'd like to work on it, please feel free to take it over since I won’t be able to work on it this month.

[andreyvelich]

@Krishna-kg732 Given that Trainer v2.2 release is coming, it would be great if you could finalize this work!
Feel free to open separate PR to automate release process.

[jaiakash]

Hi @Krishna-kg732 feel free to take this up.

We do example change log generation with git-cliff on the https://github.com/kubeflow/sdk repo, Check this kubeflow/sdk#99

You can try replicating this.

let me know if need more help for this.

[milinddethe15]

@Krishna-kg732 If you haven’t started yet, please wait until next week. I will try to work on it over the weekend.

For this PR, the only remaining task is testing a release on the forked repo.

[Krishna-kg732]

@Krishna-kg732 If you haven’t started yet, please wait until next week. I will try to work on it over the weekend.

For this PR, the only remaining task is testing a release on the forked repo.

I’ve already implemented the release workflow and will be opening a separate PR shortly.
Please feel free to review it when you have time.

[jaiakash]

Hi @Krishna-kg732 actually we need this feature. Already raised PR for that, can you help to review that please.
Check this #3231

[Krishna-kg732]

Thanks Akash, I’ll take a look at #3231 shortly and review it.
If there’s overlap with the release workflow changes I’ve implemented, we can consolidate into a single approach.

[milinddethe15]

I have tested this automation in my fork repo for release version v4.0.0

Steps:

1. Created the release commit using make target:

make release VERSION=4.0.0 GITHUB_TOKEN=<token>

2. Opened a PR with above commit on Master branch where check release action will match all the tags to version in VERSION file

3. Once the release PR is merged to master, [release](https://github.com/milinddethe15/kf-trainer/actions/runs/22284743190/job/64461002955) action will be triggered where,

• python_api build
• branch creation (release-X.Y)
• publish pypi package (for testing/verifying the upload, I have used my personal account)
• Create tag, github release with changelog
• trigger dockerimage build and publish & publish helm chart with appropriate tags (I haven't tested actual upload of images and chart but via github action logs, its confirmed that it fails only because of permission error, see below)

Chart:

Image:

5. And finally the github release is published: https://github.com/milinddethe15/kf-trainer/releases/tag/v4.0.0

---

Also added release doc for users to understand release flow: https://github.com/milinddethe15/kf-trainer/blob/feat/automate-release/docs/release/README.md