RFC: Support automated stub generation

[Tpt]

PyO3 stub generation implementation is moving forward well and starts to be in a fairly good state. It would be great to expose it in Maturin.

For context: PyO3 stub generation works by inserting into the built binaries extra symbols, each containing a JSON object describing some element (a module, a class...) and pointing to other elements by identifiers. The pyo3-introspection crate then extracts the fragments from the binaries, parse them and build stub files. The stubs generated are designed in such a way that they are never "too narrow" i.e. they should not introduce a type error in working code. We aim at having as good as possible compatibility in the JSON fragment, providing backward compatibility (newer pyo3-introspection can read fragments generated by older pyo3) and best effort forward compatibility (older pyo3-introspection should write not too bad stubs from data from newer pyo3).

I see two ways to expose stubs in PyO3:

1. a new maturin build-stubs command that would run a debug build and write the stub files in a given output directory.
2. an option --introspect-stubs or --generate-stubs to the build and develop commands that would be available on binary-only pyo3 projects and automatically add the stubs to the built artifacts. We might start by having this option as disabled by default then enable it by default if we are confident the stubs are always correct. Added automatic type stub generation with pyo3-introspection #2940 goes into this direction.

Both are useful, 1. seems very useful to experiment with stubs and in hybrid Rust/Python projects, and 2 seems convenient in binary only projects as a "just works" mechanism.

If we have agreement on the objectives, a possible timeline might be:

1. maturin build-stubs added soon relying on the pyo3-introspection 0.28 release. It covers nearly everything except doc comments and advanced features like custom imports, generics... and might be a good experiments start. We would make it clear this is a "preview" unstable feature.
2. When we are happy with the stubs state and after new PyO3 releases, implement the --generate-stubs option.
3. In long term, if we are confident about stubs quality, switch --generate-stubs as a default behavior in a possible maturin breaking release.

[messense]

IMO it's fine to go with --generate-stubs in the beginning, if we really want a build stubs command I think pyo3-introspection should include a cli? Could be useful for setuptools-rust?.

[Tpt]

Thank you!

IMO it's fine to go with --generate-stubs in the beginning

Great! Will prepare an MR

if we really want a build stubs command I think pyo3-introspection should include a cli?
Could be useful for setuptools-rust?.

Stubs are extracted from a done build so if we include a cli to pyo3-introspection it means that either it needs to call a build tool like maturin, introducing a circular dependency or mandate the user to point toward an existing binary, making usage more painful. However having this CLI would be indeed quite useful for setuptools-rust. So, what about:

• adding a small CLI to pyo3-introspection taking for input the path to a binary and a target output directory
• having maturin build-stubs as an "integrated" experience for people wanting to generate the stubs directly from Rust source code (with support of cross-compilation...)
  What do you think?

[messense]

Sounds good to me.