pipx unable to create venv when using Python from Microsoft Store

[dechamps]

On my Windows 11 23H2 (22631.2661) system, with Python 3.12.1 installed from the Microsoft Store, and following the pipx Windows installation instructions, this is what happens when I try to install anything:

PS C:\Users\etien> pipx install --verbose pycowsay
pipx >(setup:824): pipx version is 1.3.3
pipx >(setup:825): Default python interpreter is 'C:\Users\etien\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe'
pipx >(rmdir:55): removing directory C:\Users\etien\AppData\Local\pipx\pipx\trash
The system cannot find the path specified.
pipx >(rmdir:71): Failed to delete C:\Users\etien\AppData\Local\pipx\pipx\trash. You may need to delete it manually.
pipx >(package_name_from_spec:370): Determined package name: pycowsay
pipx >(package_name_from_spec:371): Package name determined in 0.0s
creating virtual environment...
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe -m venv --without-pip C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay
pipx >(run_subprocess:171): running <checking pip's availability>
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe -c import sysconfig; print(sysconfig.get_path('purelib'))
No pyvenv.cfg file
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\shared\Scripts\python.exe -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe --version
pipx >(_parsed_package_to_package_or_url:137): cleaned package spec: pycowsay
installing pycowsay...
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe -m pip --no-input install pycowsay
pipx >(subprocess_post_check_handle_pip_error:322): 'C:\\Users\\etien\\AppData\\Local\\pipx\\pipx\\venvs\\pycowsay\\Scripts\\python.exe -m pip --no-input install pycowsay' failed
pipx >(subprocess_post_check_handle_pip_error:337): Fatal error from pip prevented installation. Full pip output in file:
    C:\Users\etien\AppData\Local\pipx\pipx\Logs\cmd_2023-12-23_10.41.50_pip_errors.log

pipx >(rmdir:55): removing directory C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay
The system cannot find the path specified.
pipx >(rmdir:67): Failed to delete C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay. Will move it to a temp folder to delete later.
Error installing pycowsay.

C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\Logs\cmd_2023-12-23_10.50.12.log (note the strange path that doesn't match the above, more on that below) contains:

   108.7ms (setup:822): 2023-12-23 10:50:12
   108.7ms (setup:823): C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\pipx\__main__.py install pycowsay
   108.7ms (setup:824): pipx version is 1.3.3
   108.7ms (setup:825): Default python interpreter is 'C:\Users\etien\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe'
   109.7ms (rmdir:55): removing directory C:\Users\etien\AppData\Local\pipx\pipx\trash
   118.5ms (rmdir:71): Failed to delete C:\Users\etien\AppData\Local\pipx\pipx\trash. You may need to delete it manually.
   122.7ms (package_name_from_spec:370): Determined package name: pycowsay
   122.7ms (package_name_from_spec:371): Package name determined in 0.0s
   123.7ms (run_subprocess:171): running C:\Users\etien\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe -m venv --without-pip C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay
   206.2ms (run_subprocess:185): stdout:
   206.2ms (run_subprocess:187): stderr: Actual environment location may have moved due to redirects, links or junctions.
  Requested location: "C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe"
  Actual location:    "C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe"
   206.2ms (run_subprocess:188): returncode: 0
   206.2ms (run_subprocess:171): running <checking pip's availability>
   273.4ms (run_subprocess:185): stdout: ModuleSpec(name='pip', loader=<_frozen_importlib_external.SourceFileLoader object at 0x000001D5DD4E3620>, origin='C:\\Users\\etien\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\\LocalCache\\Local\\pipx\\pipx\\shared\\Lib\\site-packages\\pip\\__init__.py', submodule_search_locations=['C:\\Users\\etien\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\\LocalCache\\Local\\pipx\\pipx\\shared\\Lib\\site-packages\\pip'])
   273.4ms (run_subprocess:188): returncode: 0
   273.4ms (run_subprocess:171): running C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe -c import sysconfig; print(sysconfig.get_path('purelib'))
   313.8ms (run_subprocess:185): stdout:
   313.8ms (run_subprocess:188): returncode: 106
   313.8ms (run_subprocess:171): running C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\shared\Scripts\python.exe -c import sysconfig; print(sysconfig.get_path('purelib'))
   376.8ms (run_subprocess:185): stdout: C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\shared\Lib\site-packages
   376.8ms (run_subprocess:188): returncode: 0
   376.8ms (run_subprocess:171): running C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe --version
   384.8ms (run_subprocess:185): stdout:
   384.8ms (run_subprocess:187): stderr: No pyvenv.cfg file
   384.8ms (run_subprocess:188): returncode: 106
   384.8ms (_parsed_package_to_package_or_url:137): cleaned package spec: pycowsay
   384.8ms (run_subprocess:171): running C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe -m pip --no-input install pycowsay
   393.0ms (run_subprocess:188): returncode: 106
   393.0ms (subprocess_post_check_handle_pip_error:322): 'C:\\Users\\etien\\AppData\\Local\\pipx\\pipx\\venvs\\pycowsay\\Scripts\\python.exe -m pip --no-input install pycowsay' failed
   394.1ms (subprocess_post_check_handle_pip_error:337): Fatal error from pip prevented installation. Full pip output in file:
    C:\Users\etien\AppData\Local\pipx\pipx\Logs\cmd_2023-12-23_10.50.12_pip_errors.log
   395.1ms (rmdir:55): removing directory C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay
   402.7ms (rmdir:67): Failed to delete C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay. Will move it to a temp folder to delete later.
   403.8ms (cli:888): PipxError: Error installing pycowsay.
Traceback (most recent call last):
  File "C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\pipx\main.py", line 885, in cli
    return run_pipx_command(parsed_pipx_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\pipx\main.py", line 212, in run_pipx_command
    return commands.install(
           ^^^^^^^^^^^^^^^^^
  File "C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\pipx\commands\install.py", line 81, in install
    venv.install_package(
  File "C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\pipx\venv.py", line 254, in install_package
    raise PipxError(f"Error installing {full_package_description(package_name, package_or_url)}.")
pipx.util.PipxError: Error installing pycowsay.
   404.7ms (cli:896): pipx finished.

C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\Logs\cmd_2023-12-23_15.04.13_pip_errors.log contains:

PIP STDOUT
----------

PIP STDERR
----------
No pyvenv.cfg file

Additional information:

PS C:\Users\etien> pipx --version
1.3.3
PS C:\Users\etien> (Get-Command python3).Path
C:\Users\etien\AppData\Local\Microsoft\WindowsApps\python3.exe
PS C:\Users\etien> python3 -c 'import sys; print(sys.executable)'
C:\Users\etien\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe
PS C:\Users\etien> (Get-Command pipx).Path
c:\users\etien\appdata\local\packages\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\localcache\local-packages\python312\scripts\pipx.exe
PS C:\Users\etien> pipx environment
The system cannot find the path specified.
Failed to delete C:\Users\etien\AppData\Local\pipx\pipx\trash. You may need to delete it manually.
Environment variables (set by user):

PIPX_HOME=
PIPX_BIN_DIR=
PIPX_MAN_DIR=
PIPX_SHARED_LIBS=
PIPX_DEFAULT_PYTHON=
USE_EMOJI=

Derived values (computed by pipx):

PIPX_HOME=C:\Users\etien\AppData\Local\pipx\pipx
PIPX_BIN_DIR=C:\Users\etien\.local\bin
PIPX_MAN_DIR=C:\Users\etien\.local\share\man
PIPX_SHARED_LIBS=C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\shared
PIPX_LOCAL_VENVS=C:\Users\etien\AppData\Local\pipx\pipx\venvs
PIPX_LOG_DIR=C:\Users\etien\AppData\Local\pipx\pipx\Logs
PIPX_TRASH_DIR=C:\Users\etien\AppData\Local\pipx\pipx\trash
PIPX_VENV_CACHEDIR=C:\Users\etien\AppData\Local\pipx\pipx\Cache
PIPX_DEFAULT_PYTHON=C:\Users\etien\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe
USE_EMOJI=true

The output of this command is especially intruiguing:

PS C:\Users\etien> C:\Users\etien\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe -m venv --without-pip C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay
Actual environment location may have moved due to redirects, links or junctions.
  Requested location: "C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe"
  Actual location:    "C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe"

And indeed, on my system C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay doesn't exist at all, but C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay does (and contains pyvenv.cfg). It looks like this is because of path redirection. It is not clear to me if this is what's causing the problem of if it's just a red herring.

[dechamps]

Here's what I've been unable to determine so far:

1. venv creation seems to be working correctly. If I comment out pipx's code that moves the venv to the trash on installation failures, I am left with what looks like a sane venv in C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay with a working Scripts\python.exe.
2. pipx itself is running with path redirection enabled (it's running as C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.496.0_x64__qbz5n2kfra8p0\python3.12.exe which definitely has it).
3. The failing step involves pipx attempting to run C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe. That is (correctly) redirected to C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe.
4. venvs\pycowsay\Scripts\python.exe complains that it can't find pyvenv.cfg.
   • Monitoring file accesses using Process Monitor reveals that it is trying, and failing, to open C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\pyvenv.cfg. This suggests that this file access is not being redirected, otherwise it would have found it in C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay\pyvenv.cfg (which does exist)
5. If I run C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe myself on the command line, it works.

My main hypothesis at this point is:

• When pipx runs C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe, it can find the EXE and start it, because pipx itself is running with path redirection.
• I suspect venvs\pycowsay\Scripts\python.exe believes it's running as C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe, and therefore looks for pyvenv.cfg in C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\pyvenv.cfg.
• However venvs\pycowsay\Scripts\python.exe itself does not run with path redirection enabled (I think that makes sense because that EXE is not part of the Python Windows app package)
• Therefore, when venvs\pycowsay\Scripts\python.exe tries to open C:\Users\etien\AppData\Local\pipx\pipx\venvs\pycowsay\pyvenv.cfg, the access is not redirected to the real location (C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay\pyvenv.cfg), and fails.

Assuming my theory checks out, I suspect the cleanest fix would be to make sure pipx runs the venv Scripts\python.exe from its real location, not the redirected location, so that Scripts\python.exe looks for pyvenv.cfg in the correct directory.

[dechamps]

Unsurprisingly, things work just fine if I manually point pipx to the real location:

PS C:\Users\etien> $Env:PIPX_HOME = 'C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx'
PS C:\Users\etien> pipx install --verbose pycowsay
pipx >(setup:824): pipx version is 1.3.3
pipx >(setup:825): Default python interpreter is 'C:\Users\etien\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe'
pipx >(mkdir:77): creating directory C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\.cache
pipx >(package_name_from_spec:370): Determined package name: pycowsay
pipx >(package_name_from_spec:371): Package name determined in 0.0s
creating virtual environment...
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe -m venv --without-pip C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay
pipx >(run_subprocess:171): running <checking pip's availability>
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\shared\Scripts\python.exe -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe --version
pipx >(_parsed_package_to_package_or_url:137): cleaned package spec: pycowsay
installing pycowsay...
pipx >(run_subprocess:171): running C:\Users\etien\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\Local\pipx\pipx\venvs\pycowsay\Scripts\python.exe -m pip --no-input install pycowsay
pipx >(run_subprocess:171): running <fetch_info_in_venv commands>
pipx >(get_venv_metadata_for_package:339): get_venv_metadata_for_package: 137ms
pipx >(_parsed_package_to_package_or_url:137): cleaned package spec: pycowsay
pipx >(mkdir:77): creating directory C:\Users\etien\.local\share\man\man6
pipx >(run_subprocess:171): running <checking pip's availability>
pipx >(needs_upgrade:77): Time since last upgrade of shared libs, in seconds: 10226. Upgrade will be run by pipx if greater than 2592000.
pipx >(run_subprocess:171): running <checking pip's availability>
  installed package pycowsay 0.0.0.2, installed using Python 3.12.1
  These apps are now globally available
    - pycowsay.exe
  These manual pages are now globally available
    - man6\pycowsay.6
done! ✨ 🌟 ✨

At least that's one workaround for affected users.

[dechamps]

Possibly related/similar:

• Unable to create Virtual Environment with MS Store Python python-poetry/poetry#5331
• Python from the Windows store alias paths tox-dev/platformdirs#90 (pipx uses platformdirs)
• fix: fix venv paths redirected by Python from MS Store python-poetry/poetry#5931