CI: add PyPI Trusted-Publishing “publish” job to wheels workflow (#61669)

[EvMossan]

• closes ENH: Switch to trusted publishing for package upload to PyPI in CI #61669
• all code checks passed (pre-commit & CI)
• added an entry in doc/source/whatsnew/v3.0.0.rst

Summary

This PR enables Trusted Publishing (OIDC) uploads to PyPI when a GitHub release is published.

What’s changed

• .github/workflows/wheels.yml

  • adds a new publish job that
    1. downloads all wheel / sdist artifacts from upstream jobs;
    2. excludes Pyodide wheels (*pyodide*.whl);
    3. runs pypa/gh-action-pypi-publish@v1 in the pypi environment.

• doc/source/whatsnew/v3.0.0.rst

  • adds a single Build / CI line announcing the switch to Trusted Publishing

• doc/source/development/maintaining.rst

  • drop manual twine step and note Trusted Publishing

No other files or CI matrix settings were changed.

Release prerequisites

• GitHub repo must have an environment named pypi (OIDC token permission enabled).
• The pandas project on PyPI must list pandas-dev/pandas → “pypi” as a Trusted Publisher (see https://docs.pypi.org/trusted-publishers/).

[EvMossan]

CI failed in the docstring-validation step with

AttributeError: 'getset_descriptor' object has no attribute '__module__'. Did you mean: '__reduce__'?

This occurs before my code runs, so it isn’t caused by the changes in this PR.
It looks related to the latest numpydoc release.
I’ll re-run CI once the upstream fix lands.

[EvMossan]

All comments addressed and conversations resolved - ready for another look. Thanks!

[jorisvandenbossche]

@EvMossan thanks a lot for working on this! (and @EpicWink for the review)

@mroeschke I updated the PyPI configuration to enable trusted publishing on that side.

I suppose we can only test this really once doing a next pre-release?
Could we already test the first part of the added job by temporarily commenting out the last pypa/gh-action-pypi-publish step, but so at least we can check the downloading and filtering of the artifacts is working?

[jorisvandenbossche]

@EpicWink you suggested to add this, but I am wondering if that is needed? As I understand, this wheels.yml workflow will run on the commit pushed to main with a tag, and so that way it will already run for a release? And we can then upload the wheels from that run?

[EpicWink]

A GitHub release publish happens after a push, allowing you to review the CI, and write release notes in the GitHub release, before triggering the publish to PyPI. Up to you how you want the release workflow to be, if you want to publish unconditionally.

An alternative is to split the workflow into two files (and therefore two workflows), where the publish job downloads the artefacts from the build job of the other workflow, and the publish workflow is triggered from the build workflow's completion.

[jorisvandenbossche]

@EpicWink thanks for the explanation, got it! Since we still manually create the github release (not automated from CI when alls builds passed), it indeed seems fine to only run the PyPI publish step afterwards.

[jorisvandenbossche]

Is this line needed? (it's not included in the example in https://docs.pypi.org/trusted-publishers/using-a-publisher/)

[EpicWink]

It's not necessary, and as far as I know there's no recommendation to hard-code it. When the upload API 2.0 PEP lands, this URL and action will change anyway.

[EpicWink]

I suppose we can only test this really once doing a next pre-release?

Yes, fortunately PyPI (production) supports uploading pre-releases.

Could we already test the first part of the added job by temporarily commenting out the last pypa/gh-action-pypi-publish step, but so at least we can check the downloading and filtering of the artifacts is working?

That could work, as long as it's in a protected tag, but you would be creating a tag just to test this, unless you change the workflow definition further to support a non-version-tag ref.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[jorisvandenbossche]

OK, let's give this a try. Thanks @EvMossan and @EpicWink!

[EvMossan]

Thank you!

[jorisvandenbossche]

FWIW, I have tested the workflow when releasing 2.3.3, and we still have some issue to resolve. Essentially the problem was that on the "release" event, we didn't actually run the jobs to build the sdist and wheels (their if does not include a check for release), and so then the publish step also did not run because this has needs: on the sdist/wheel build ..

This should be easy to solve by updating the if: clauses.
Longer term, I think it would still be nice to avoid rebuilding the wheels for the release event, but reusing the ones that were build (if successfully) from pushing the tag (#61718 (comment))