feat(build): design “profiles”

[tdejager]

We can generate packages in different ways, editable or not, release vs debug, etc. The different configurations depend on what a backend support. We want to figure out an intuitive general way of defining these properties, configuring them and using them.

Conan, cmake, meson use “profiles” for this which seems useful. More research is required.

Proposal

[Hofer-Julian]

We should also think about which profiles make sense for pixi-build-python. For example, if you use it with maturin, you'll want to switch between debug and release mode.

For future reference, this is how you can do this: PyO3/maturin#2314 (comment)

[lucascolley]

SciPy build modes

What are we building?

Output	How?	Description
Python wheel	python -m build --wheel --no-isolation	via meson-python
Conda package	build.sh, build-output.sh	build a wheel, pip install the wheel
Editable Python package	spin install	pip install . -e via meson-python
Regular Python package	spin install --no-editable	pip install . via meson-python
Regular Python package	spin build	Directly using meson {setup, compile, install}. Installs output to a directory, not as a package in the environment. Many different profiles used via this. Other development tasks (spin test, spin docs) then use this directory automatically.
Source distribution	spin sdist	pyproject-build . --sdist

Where does pixi build fit in?

Some thoughts for now:

• It would be nice for pixi-build-python to support building wheels and sdists.
• pixi-build-python seems to already be providing an alternative for building a conda package via pixi build, which does not involve building a wheel. This supports recursive source dependencies if you fiddle with the manifest. pixi-build-python is effectively covering spin install right now, using pip install . under the hood
• spin install and spin install --no-editable could be covered by different profiles for pixi install, backed by pixi-build-python. Again supporting source deps.
• For SciPy, we may want the default profile for pixi install to be neither of those, but rather spin build.
  • This would then be in scope for a pixi-build-meson backend, which would not need any Python-specific logic. We would want highly customisable profiles here.
• Perhaps a pixi-build-meson-python backend could have dependencies on pixi-build-meson and pixi-build-python, delegating to each for the appropriate modes, and be very thin itself?

One thing I'm not clear on is the UI for selecting build profiles when there are multiple Pixi packages in your environment that would be built on pixi install. Will we need a different environment for each combination of build profiles one wants to use?

What would that enable?

• Pixi workspaces could have source or git dependencies on a SciPy package, which would be built into a conda package with pixi-build-python
• a Pixi workspace for SciPy development could then make use of the different modes and profiles of pixi install. This should be able to integrate seamlessly with tasks for testing, linting etc. which use spin directly.

---

Something that goes past this scope here is crafting a workspace with environments which 'pick-and-choose' packages to use from source/git, with the rest coming from their usual source. This means both:

• developing SciPy and picking e.g. numpy and meson from source on the fly, without carrying around exponentially many environments for different package combinations.
• spinning up ipython in my workspace with 3 out of the 10 packages using pixi-build, the rest from conda-forge.

I just discussed with @tdejager and @baszalmstra the possibility of something like cargo's patching being helpful here. But this still needs a lot of thought.

---

@rgommers I think this will be of interest to you, any thoughts much appreciated.