Skip to content

Allow PEP 508 URL spec as editable #9471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
uranusjr wants to merge 4 commits into from
Closed

Allow PEP 508 URL spec as editable #9471

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9e97705
Allow PEP 508 URL spec as editable
uranusjr Jan 18, 2021
9b11a41
Update docs to always use PEP 508 instead of #egg=
uranusjr Jan 18, 2021
9defcc0
Make sure 'NAME @ URL' actually resembles a URL
uranusjr Feb 3, 2021
0fcde33
Build editable from a req string, not just the URL
uranusjr Mar 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 54 additions & 53 deletions docs/html/reference/pip_install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ for the name and project version (this is in theory slightly less reliable
than using the ``egg_info`` command, but avoids downloading and processing
unnecessary numbers of files).

Any URL may use the ``#egg=name`` syntax (see :ref:`VCS Support`) to
explicitly state the project name.
The :pep:`508` requirement syntax can be used to explicitly state the project
name (see :ref:`VCS Support`).

Satisfying Requirements
-----------------------
Expand Down Expand Up @@ -396,12 +396,13 @@ the :ref:`--editable <install_--editable>` option) or not.
(referencing a specific commit) if and only if the install is done using the
editable option.

The "project name" component of the URL suffix ``egg=<project name>``
is used by pip in its dependency logic to identify the project prior
to pip downloading and analyzing the metadata. For projects
where ``setup.py`` is not in the root of project, the "subdirectory" component
is used. The value of the "subdirectory" component should be a path starting
from the root of the project to where ``setup.py`` is located.
The "project name" of a requirement specified by URL can be explicitly set
with the :pep:`508` requirement syntax. This value is used by pip in its
dependency logic to identify the project prior to pip downloading and analyzing
the metadata. For projects where ``setup.py`` is not in the root of project,
the "subdirectory" component is used. The value of the "subdirectory" component
should be a path starting from the root of the project to where ``setup.py``
is located.

If your repository layout is::

Expand All @@ -418,13 +419,13 @@ Then, to install from this repository, the syntax would be:

.. code-block:: shell

python -m pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir"
python -m pip install -e "pkg @ vcs+protocol://repo_url/#subdirectory=pkg_dir"

.. tab:: Windows

.. code-block:: shell

py -m pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir"
py -m pip install -e "pkg @ vcs+protocol://repo_url/#subdirectory=pkg_dir"


Git
Expand All @@ -441,17 +442,17 @@ pip currently supports cloning over ``git``, ``git+http``, ``git+https``,

Here are the supported forms::

[-e] git+http://git.example.com/MyProject#egg=MyProject
[-e] git+https://git.example.com/MyProject#egg=MyProject
[-e] git+ssh://git.example.com/MyProject#egg=MyProject
[-e] git+file:///home/user/projects/MyProject#egg=MyProject
[-e] MyProject@git+http://git.example.com/MyProject
[-e] MyProject@git+https://git.example.com/MyProject
[-e] MyProject@git+ssh://git.example.com/MyProject
[-e] MyProject@git+file:///home/user/projects/MyProject

Passing a branch name, a commit hash, a tag name or a git ref is possible like so::

[-e] git+https://git.example.com/MyProject.git@master#egg=MyProject
[-e] git+https://git.example.com/[email protected]#egg=MyProject
[-e] git+https://git.example.com/MyProject.git@da39a3ee5e6b4b0d3255bfef95601890afd80709#egg=MyProject
[-e] git+https://git.example.com/MyProject.git@refs/pull/123/head#egg=MyProject
[-e] MyProject@git+https://git.example.com/MyProject.git@master
[-e] MyProject@git+https://git.example.com/[email protected]
[-e] MyProject@git+https://git.example.com/MyProject.git@da39a3ee5e6b4b0d3255bfef95601890afd80709
[-e] MyProject@git+https://git.example.com/MyProject.git@refs/pull/123/head

When passing a commit hash, specifying a full hash is preferable to a partial
hash because a full hash allows pip to operate more efficiently (e.g. by
Expand All @@ -467,18 +468,18 @@ The supported schemes are: ``hg+file``, ``hg+http``, ``hg+https``,

Here are the supported forms::

[-e] hg+http://hg.myproject.org/MyProject#egg=MyProject
[-e] hg+https://hg.myproject.org/MyProject#egg=MyProject
[-e] hg+ssh://hg.myproject.org/MyProject#egg=MyProject
[-e] hg+file:///home/user/projects/MyProject#egg=MyProject
[-e] MyProject@hg+http://hg.myproject.org/MyProject
[-e] MyProject@hg+https://hg.myproject.org/MyProject
[-e] MyProject@hg+ssh://hg.myproject.org/MyProject
[-e] MyProject@hg+file:///home/user/projects/MyProject

You can also specify a revision number, a revision hash, a tag name or a local
branch name like so::

[-e] hg+http://hg.example.com/MyProject@da39a3ee5e6b#egg=MyProject
[-e] hg+http://hg.example.com/MyProject@2019#egg=MyProject
[-e] hg+http://hg.example.com/[email protected]#egg=MyProject
[-e] hg+http://hg.example.com/MyProject@special_feature#egg=MyProject
[-e] MyProject@hg+http://hg.example.com/MyProject@da39a3ee5e6b
[-e] MyProject@hg+http://hg.example.com/MyProject@2019
[-e] MyProject@hg+http://hg.example.com/[email protected]
[-e] MyProject@hg+http://hg.example.com/MyProject@special_feature

Subversion
^^^^^^^^^^
Expand All @@ -487,14 +488,14 @@ pip supports the URL schemes ``svn``, ``svn+svn``, ``svn+http``, ``svn+https``,

Here are some of the supported forms::

[-e] svn+https://svn.example.com/MyProject#egg=MyProject
[-e] svn+ssh://svn.example.com/MyProject#egg=MyProject
[-e] svn+ssh://[email protected]/MyProject#egg=MyProject
[-e] MyProject@svn+https://svn.example.com/MyProject
[-e] MyProject@svn+ssh://svn.example.com/MyProject
[-e] MyProject@svn+ssh://[email protected]/MyProject

You can also give specific revisions to an SVN URL, like so::

[-e] svn+svn://svn.example.com/svn/MyProject#egg=MyProject
[-e] svn+http://svn.example.com/svn/MyProject/trunk@2019#egg=MyProject
[-e] MyProject@svn+svn://svn.example.com/svn/MyProject
[-e] MyProject@svn+http://svn.example.com/svn/MyProject/trunk@2019

which will check out revision 2019. ``@{20080101}`` would also check
out the revision from 2008-01-01. You can only check out specific
Expand All @@ -508,16 +509,16 @@ pip supports Bazaar using the ``bzr+http``, ``bzr+https``, ``bzr+ssh``,

Here are the supported forms::

[-e] bzr+http://bzr.example.com/MyProject/trunk#egg=MyProject
[-e] bzr+sftp://[email protected]/MyProject/trunk#egg=MyProject
[-e] bzr+ssh://[email protected]/MyProject/trunk#egg=MyProject
[-e] bzr+ftp://[email protected]/MyProject/trunk#egg=MyProject
[-e] bzr+lp:MyProject#egg=MyProject
[-e] MyProject@bzr+http://bzr.example.com/MyProject/trunk
[-e] MyProject@bzr+sftp://[email protected]/MyProject/trunk
[-e] MyProject@bzr+ssh://[email protected]/MyProject/trunk
[-e] MyProject@bzr+ftp://[email protected]/MyProject/trunk
[-e] MyProject@bzr+lp:MyProject

Tags or revisions can be installed like so::

[-e] bzr+https://bzr.example.com/MyProject/trunk@2019#egg=MyProject
[-e] bzr+http://bzr.example.com/MyProject/[email protected]#egg=MyProject
[-e] MyProject@bzr+https://bzr.example.com/MyProject/trunk@2019
[-e] MyProject@bzr+http://bzr.example.com/MyProject/[email protected]

Using Environment Variables
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand All @@ -527,8 +528,8 @@ makes it possible to reference private repositories without having to store
access tokens in the requirements file. For example, a private git repository
allowing Basic Auth for authentication can be refenced like this::

[-e] git+http://${AUTH_USER}:${AUTH_PASSWORD}@git.example.com/MyProject#egg=MyProject
[-e] git+https://${AUTH_USER}:${AUTH_PASSWORD}@git.example.com/MyProject#egg=MyProject
[-e] MyProject@git+http://${AUTH_USER}:${AUTH_PASSWORD}@git.example.com/MyProject
[-e] MyProject@git+https://${AUTH_USER}:${AUTH_PASSWORD}@git.example.com/MyProject

.. note::

Expand Down Expand Up @@ -827,14 +828,14 @@ You can install local projects or VCS projects in "editable" mode:
.. code-block:: shell

python -m pip install -e path/to/SomeProject
python -m pip install -e git+http://repo/my_project.git#egg=SomeProject
python -m pip install -e SomeProject@git+http://repo/my_project.git

.. tab:: Windows

.. code-block:: shell

py -m pip install -e path/to/SomeProject
py -m pip install -e git+http://repo/my_project.git#egg=SomeProject
py -m pip install -e SomeProject@git+http://repo/my_project.git


(See the :ref:`VCS Support` section above for more information on VCS-related syntax.)
Expand Down Expand Up @@ -956,7 +957,7 @@ Examples

py -m pip install SomePackage # latest version
py -m pip install SomePackage==1.0.4 # specific version
py -m pip install 'SomePackage>=1.0.4' # minimum version
py -m pip install "SomePackage>=1.0.4" # minimum version


#. Install a list of requirements specified in a file. See the :ref:`Requirements files <Requirements Files>`.
Expand Down Expand Up @@ -1027,21 +1028,21 @@ Examples

.. code-block:: shell

python -m pip install -e git+https://git.repo/some_pkg.git#egg=SomePackage # from git
python -m pip install -e hg+https://hg.repo/some_pkg.git#egg=SomePackage # from mercurial
python -m pip install -e svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage # from svn
python -m pip install -e git+https://git.repo/some_pkg.git@feature#egg=SomePackage # from 'feature' branch
python -m pip install -e "git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path" # install a python package from a repo subdirectory
python -m pip install -e SomePackage@git+https://git.repo/some_pkg.git # from git
python -m pip install -e SomePackage@hg+https://hg.repo/some_pkg # from mercurial
python -m pip install -e SomePackage@svn+svn://svn.repo/some_pkg/trunk/ # from svn
python -m pip install -e SomePackage@git+https://git.repo/some_pkg.git@feature # from 'feature' branch
python -m pip install -e SomePackage@git+https://git.repo/some_repo.git#subdirectory=subdir_path # install a python package from a repo subdirectory

.. tab:: Windows

.. code-block:: shell

py -m pip install -e git+https://git.repo/some_pkg.git#egg=SomePackage # from git
py -m pip install -e hg+https://hg.repo/some_pkg.git#egg=SomePackage # from mercurial
py -m pip install -e svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage # from svn
py -m pip install -e git+https://git.repo/some_pkg.git@feature#egg=SomePackage # from 'feature' branch
py -m pip install -e "git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path" # install a python package from a repo subdirectory
py -m pip install -e SomePackage@git+https://git.repo/some_pkg.git # from git
py -m pip install -e SomePackage@hg+https://hg.repo/some_pkg # from mercurial
py -m pip install -e SomePackage@svn+svn://svn.repo/some_pkg/trunk/ # from svn
py -m pip install -e SomePackage@git+https://git.repo/some_pkg.git@feature # from 'feature' branch
py -m pip install -e SomePackage@git+https://git.repo/some_repo.git#subdirectory=subdir_path # install a python package from a repo subdirectory

#. Install a package with `setuptools extras`_.

Expand Down
2 changes: 1 addition & 1 deletion docs/html/user_guide.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ In practice, there are 4 common uses of Requirements files:
``sometag``. You'd reference it in your requirements file with a line like
so::

git+https://myvcs.com/some_dependency@sometag#egg=SomeDependency
SomeDependency @ git+https://myvcs.com/some_dependency@sometag

If ``SomeDependency`` was previously a top-level requirement in your
requirements file, then **replace** that line with the new line. If
Expand Down
1 change: 1 addition & 0 deletions news/1289.feature.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add support to install a PEP 508 URL-specified requirement as editable.
19 changes: 16 additions & 3 deletions src/pip/_internal/req/constructors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,24 @@ def parse_editable(editable_req):
- a requirement name
- an URL
- extras
- editable options
Accepted requirements:
svn+http://blahblah@rev#egg=Foobar[baz]&subdirectory=version_subdir
.[some_extra]
- svn+http://blahblah@rev#egg=Foobar[baz]&subdirectory=version_subdir
- local_path[some_extra]
- Foobar[extra] @ svn+http://blahblah@rev#subdirectory=subdir ; markers
"""
try:
req = Requirement(editable_req)
except InvalidRequirement:
pass
else:
if req.url and "://" in req.url:
# Join the marker back into the name part. This will be parsed out
# later into a Requirement again.
if req.marker:
name = f"{req.name} ; {req.marker}"
else:
name = req.name
return (name, req.url, req.extras)
Copy link
Member

@sbidoul sbidoul Jan 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, on second look... this will let non VCS URLs go through ? So we need to merge #9436 first and see how we can do the link.is_vcs here too.

Copy link
Member Author

@uranusjr uranusjr Jan 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good point, Requirement() does not check if the URL is actually valid.

Copy link
Member Author

@uranusjr uranusjr Jan 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing we ca do here is to add an extra check, and only treat the string as PEP 508 if the URL part contains ://. This means pseudo URL strings will automatically fall back to the old parsing logic, and can be handled independently to this new logic.

Copy link
Member Author

@uranusjr uranusjr Feb 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sbidoul I added an additional check to prevent pseudo-URLs from going through here. Do you think it’d help?

Copy link
Member

@sbidoul sbidoul Feb 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@uranusjr I have not tested (I whish I had more time for pip...) but this version would accept any URLs ? Editables URLs must be file:// or vcs+...:// urls only, correct ? And even file URLs must point to a local directory, not a file.

Copy link
Member Author

@uranusjr uranusjr Feb 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. There are already checks in a later stage to error out on invalid URL values, so I think it’s fine to pass them through here.

pip/src/pip/_internal/req/constructors.py

Lines 116 to 119 in baaf66f

raise InstallationError(
f'{editable_req} is not a valid editable requirement. '
f'It should either be a path to a local project or a VCS URL '
f'(beginning with {backends}).'


url = editable_req

Expand Down
17 changes: 17 additions & 0 deletions tests/unit/test_req.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,23 @@ def test_parse_editable_vcs_extras():
)


@pytest.mark.parametrize(
"req_str, expected",
[
(
'foo[extra] @ svn+http://foo ; os_name == "nt"',
('foo ; os_name == "nt"', 'svn+http://foo', {'extra'}),
),
(
'foo @ svn+http://foo',
('foo', 'svn+http://foo', set()),
),
],
)
def test_parse_editable_pep508(req_str, expected):
assert parse_editable(req_str) == expected


@patch('pip._internal.req.req_install.os.path.abspath')
@patch('pip._internal.req.req_install.os.path.exists')
@patch('pip._internal.req.req_install.os.path.isdir')
Expand Down