[BUG] setuptools>=65.2.0 fails to find Python.h in Blender embedded Python environment

[peisenwang]

setuptools version

setuptools>=65.2.0

Python version

Python 3.10.2

OS

Ubuntu Linux x86_64

Additional environment information

Running in Blender 3.3.0's embedded Python environment.

Description

Trying to install OpenEXR inside Blender's embedded Python environment.

The package contains C code where Python.h is included. To add Python.h to the environment I downloaded Python source code matching the version from python.org, and extracted the header files under Include into blender/3.3/python/include/python3.10/. Then I run blender/3.3/python/bin/python3.10 -m pip install OpenEXR to install OpenEXR.

This worked before with setuptools under 65.1.1, but after udpating setuptools over 65.2.0 the installation failed with error: OpenEXR.cpp:1:10: fatal error: Python.h: No such file or directory

Expected behavior

Successfully compiling and installing OpenEXR inside Blender embedded Python.

How to Reproduce

1. Download latest version (currentlly 3.3.0) of Blender from https://www.blender.org/download/ and extract it.
2. Setup pip: blender-3.3.0-linux-x64/3.3/python/bin/python3.10 -m ensurepip
3. Install setuptools: blender-3.3.0-linux-x64/3.3/python/bin/python3.10 -m pip install setuptools==65.2.0
4. Download Python sourcefile tarball from python.org, extract, and copy files under Include into blender python's include directory:
   cp -r Python-3.10.2/Include/* blender-3.3.0-linux-x64/3.3/python/include/python3.10/
5. Try installing OpenEXR: blender-3.3.0-linux-x64/3.3/python/bin/python3.10 -m pip install OpenEXR
6. Switching to setuptools under 65.2.0 works fine:
   blender-3.3.0-linux-x64/3.3/python/bin/python3.10 -m pip install setuptools==65.1.1
blender-3.3.0-linux-x64/3.3/python/bin/python3.10 -m pip install OpenEXR

Output

  Preparing metadata (setup.py) ... done                                        
Building wheels for collected packages: OpenEXR                                 
  Building wheel for OpenEXR (setup.py) ... error                               
  error: subprocess-exited-with-error                                           
                                                                                
  × python setup.py bdist_wheel did not run successfully.                       
  │ exit code: 1                                                                
  ╰─> [16 lines of output]                                                      
      Looking for libOpenEXR...                                                 
      running bdist_wheel                                                       
      running build                                                             
      running build_py                                                          
      creating build                                                            
      creating build/lib.linux-x86_64-cpython-310                               
      copying Imath.py -> build/lib.linux-x86_64-cpython-310                    
      running build_ext                                                         
      building 'OpenEXR' extension                                              
      creating build/temp.linux-x86_64-cpython-310                              
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wa
ll -I/home/sybren/buildbot-builder/linux_glibc217_x86_64_cmake/build_deps/deps/R
elease/sqlite/include -I/home/sybren/buildbot-builder/linux_glibc217_x86_64_cmak
e/build_deps/deps/Release/bzip2/include -I/home/sybren/buildbot-builder/linux_gl
ibc217_x86_64_cmake/build_deps/deps/Release/lzma/include -I/home/sybren/buildbot
-builder/linux_glibc217_x86_64_cmake/build_deps/deps/Release/zlib/include -fPIC 
-fPIC -fPIC -I/usr/include/OpenEXR -I/usr/local/include/OpenEXR -I/opt/local/inc
lude/OpenEXR -I/usr/include/Imath -I/usr/local/include/Imath -I/opt/local/includ
e/Imath -I/home/sybren/buildbot-builder/linux_glibc217_x86_64_cmake/build_deps/d
eps/Release/python/include/python3.10 -c OpenEXR.cpp -o build/temp.linux-x86_64-
cpython-310/OpenEXR.o -g -DVERSION=\"1.3.9\"                                    
      OpenEXR.cpp:1:10: fatal error: Python.h: No such file or directory        
          1 | #include <Python.h>                                               
            |          ^~~~~~~~~~                                               
      compilation terminated.                                                   
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
   
  note: This error originates from a subprocess, and is likely not a problem wit
h pip.
  ERROR: Failed building wheel for OpenEXR
  Running setup.py clean for OpenEXR
Failed to build OpenEXR

[wqh17101]

I meet the same questions. May the reason is the same one . #3605
setuptools hack the original distutils.sysconfig.

[cbay]

Same here, although that's a more general issue that has nothing to do with Blender.

I compile Python with --prefix=/usr/local and install it with make install DESTDIR=/tmp/python, which is quite common when building a package (a Debian package, in my case).

$ /tmp/python/usr/local/bin/python3-config --includes
-I/tmp/python/usr/local/include/python3.10 -I/tmp/python/usr/local/include/python3.1

But setuptools doesn't use those paths as it strips the prefix, for instance:

$ /tmp/python/usr/local/bin/python3 -m pip install setproctitle==1.2.2 --no-binary setproctitle
Collecting setproctitle==1.2.2
  Using cached setproctitle-1.2.2.tar.gz (23 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: setproctitle
  Building wheel for setproctitle (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for setproctitle (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running bdist_wheel
      running build
      running build_ext
      building 'setproctitle' extension
      creating build
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/src
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DSPT_VERSION=1.2.2 -DHAVE_SYS_PRCTL_H=1 -I/usr/local/include/python3.10 -c src/setproctitle.c -o build/temp.linux-x86_64-cpython-310/src/setproctitle.o
      In file included from src/spt.h:15,
                       from src/setproctitle.c:14:
      src/spt_python.h:16:10: fatal error: Python.h: No such file or directory
       #include <Python.h>
                ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for setproctitle
Failed to build setproctitle

I assume that comes from this merge.

[cbay]

The issue is fixed (see #3657 for more details).