Add cli commands, restructure data and update code and tests by johnmhoran · Pull Request #281 · aboutcode-org/purldb

johnmhoran · 2024-01-31T01:36:09Z

Reference: #247

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

JonoYang

@johnmhoran I've left some comments on the meta command and related functions.

purldb-toolkit/src/purldb_toolkit/purlcli.py

johnmhoran · 2024-02-01T02:01:41Z

Thanks for your helpful comments @JonoYang AND for taking a look at this nascent PR so quickly.

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

pombredanne · 2024-02-15T15:03:34Z

purldb-toolkit/tests/data/purlcli/expected_metadata_output_unique.json

+            ],
+            "errors": [],
+            "warnings": [
+                "input PURL: 'pkg:pypi/fetchcode@0.3.0' normalized to 'pkg:pypi/fetchcode'",


Please do NOT normalize unless someone requested it... and we surely should never drop a version when request for one.

I get that. You made the point over the weekend and I acknowleged and explained what I'm doing as a result. Please look at the name of this file and the context -- this is the result when the --unique flag is included.

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

johnmhoran · 2024-02-16T03:33:56Z

@JonoYang Puzzling -- all 76 of my test_purlcli.py tests pass locally but 3 of my new urls tests fail via GH checks -- all failures concern pkg:gem/bundler-sass. No idea why the failures. I reran the GH tests several times but got the same failures.

urls calls purl2url from packageurl.contrib, which handles both pkg:gem/ and pkg:rubygems/. https://github.com/package-url/packageurl-python/blob/main/src/packageurl/contrib/purl2url.py#L173-L186

An example of the diff for purldb-toolkit/tests/test_purlcli.py::TestPURLCLI_urls::test_urls_details:


E         Full diff:
E           {
E            'headers': [{'errors': [],
E                         'options': {'--file': None,
E                                     '--output': '',
E                                     '--purl': ['pkg:gem/bundler-sass'],
E                                     'command': 'urls'},
E                         'purls': ['pkg:gem/bundler-sass'],
E                         'tool_name': 'purlcli',
E                         'tool_version': '0.0.1',
E                         'warnings': []}],
E            'packages': [{'download_url': {'url': None},
E         -                'inferred_urls': [{'url': 'https://rubygems.org/gems/bundler-sass'}],
E         +                'inferred_urls': [],
E                          'purl': 'pkg:gem/bundler-sass',
E                          'repo_download_url': {'url': None},
E                          'repo_download_url_by_package_type': {'url': None},
E         -                'repo_url': {'url': 'https://rubygems.org/gems/bundler-sass'},
E         -                'url': {'url': 'https://rubygems.org/gems/bundler-sass'}}],
E         +                'repo_url': {'url': None},
E         +                'url': {'url': None}}],
E           }

But that test passes locally and if I check that PURL with the urls command I get the same result I expect in my test (but GH rejects):

(venv) Thu Feb 15, 2024 07:10 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ python -m purldb_toolkit.purlcli urls --purl pkg:gem/bundler-sass --output -
{
    "headers": [
        {
            "tool_name": "purlcli",
            "tool_version": "0.0.1",
            "options": {
                "command": "urls",
                "--purl": [
                    "pkg:gem/bundler-sass"
                ],
                "--file": null,
                "--output": "<stdout>"
            },
            "purls": [
                "pkg:gem/bundler-sass"
            ],
            "errors": [],
            "warnings": []
        }
    ],
    "packages": [
        {
            "purl": "pkg:gem/bundler-sass",
            "download_url": {
                "url": null
            },
            "inferred_urls": [
                {
                    "url": "https://rubygems.org/gems/bundler-sass"
                }
            ],
            "repo_download_url": {
                "url": null
            },
            "repo_download_url_by_package_type": {
                "url": null
            },
            "repo_url": {
                "url": "https://rubygems.org/gems/bundler-sass"
            },
            "url": {
                "url": "https://rubygems.org/gems/bundler-sass"
            }
        }
    ]
}
(venv) Thu Feb 15, 2024 07:12 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$

johnmhoran · 2024-02-16T06:25:02Z

@JonoYang I think I might have found the cause of these erroneous GitHub test failures: the GitHub tests have an old version of packageurl-python installed.

The current version of https://github.com/package-url/packageurl-python/tree/main is 0.13.4. This is what pip list tells me I have installed locally.
0.13.4 has two routes in purl2url.py that support both pkg:gem/ and pkg:rubygems/ See https://github.com/package-url/packageurl-python/blob/main/src/packageurl/contrib/purl2url.py#L173 (@repo_router.route("pkg:(gem|rubygems)/.*")) and https://github.com/package-url/packageurl-python/blob/main/src/packageurl/contrib/purl2url.py#L294 (@download_router.route("pkg:(gem|rubygems)/.*")).
My code and test code is based on that version of packageurl-python.
But: GitHub's log of the failed tests indicates that the version of packageurl-python it has installed is 0.11.2. See https://github.com/nexB/purldb/actions/runs/7925276260/job/21640725273 at lines 40-41 https://github.com/nexB/purldb/actions/runs/7925276260/job/21640725273#step:5:41
In 0.11.2, the relevant routes in route in purl2url.py support rubygems BUT NOT gem. See https://github.com/package-url/packageurl-python/blob/v0.11.2/src/packageurl/contrib/purl2url.py#L174 (@repo_router.route("pkg:rubygems/.*")) and https://github.com/package-url/packageurl-python/blob/v0.11.2/src/packageurl/contrib/purl2url.py#L295 (@download_router.route("pkg:rubygems/.*")).

If I'm correct, could we update the GitHub tests to use the current version of packageurl-python? And is there some sort of automated process that could enable us to avoid this sort of hard-to-track problem?

Update: I see that setup.cfg has packageurl-python == 0.11.2. https://github.com/nexB/purldb/blob/main/setup.cfg#L54 I'll change that locally to packageurl-python == 0.13.4, commit and push and see what happens.

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

johnmhoran · 2024-02-16T06:54:41Z

@JonoYang Close but no cigar. There are other problems, apparently related to an effort to pin 0.11.2. See https://github.com/nexB/purldb/actions/runs/7927038107/job/21642750969?pr=281#step:5:60 et seq:

ERROR: Cannot install purldb-toolkit==0.0.1 and purldb[testing]==2.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    purldb-toolkit 0.0.1 depends on packageurl-python
    purldb[testing] 2.0.0 depends on packageurl-python==0.13.4
    The user requested (constraint) packageurl-python==0.11.2

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Notice:  A new release of pip available: 22.2.2 -> 24.0
Notice:  To update, run: python -m pip install --upgrade pip
make: *** [Makefile:41: dev] Error 1
Error: Process completed with exit code 2.

Not sure how to translate this into language I can understand and use to enable my simple tests to pass using the current version of packageurl-python. Can you suggest how I can approach my troubleshooting on this? ;-)

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Signed-off-by: John M. Horan <johnmhoran@gmail.com>

johnmhoran · 2024-02-16T21:53:17Z

Thank you @TG1999 for helping me identify the source of the 0.11.2 vs 0.13.4 packageurl-python conflict. To address the remaining conflicts referred to in the GH PR page. I've updated main, merged main into my branch, resolved the conflicts (those also involved packageurl-python's version as well as versions of some packages in the incoming code from main, which I think is work @JonoYang has been doing).

When I ran my local PURL CLI tests before pushing, I got an error ModuleNotFoundError: No module named 'drf_spectacular'. It occurred to me that I needed to run make test and did so but got another error -- one that mentions scancode.io and is puzzling to me. Perhaps we can discuss on Monday?

(venv) Fri Feb 16, 2024 01:37 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ make dev
-> Configure and install development dependencies
Obtaining file:///home/jmh/dev/nexb/purldb/matchcode-toolkit
  Checking if build backend supports build_editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Obtaining file:///home/jmh/dev/nexb/purldb/purldb-toolkit
  Checking if build backend supports build_editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Obtaining file:///home/jmh/dev/nexb/purldb
  Checking if build backend supports build_editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: bitarray in ./venv/lib/python3.8/site-packages (from matchcode-toolkit==2.0.1) (2.6.0)
Requirement already satisfied: plugincode in ./venv/lib/python3.8/site-packages (from matchcode-toolkit==2.0.1) (32.0.0)
Requirement already satisfied: commoncode in ./venv/lib/python3.8/site-packages (from matchcode-toolkit==2.0.1) (31.0.3)
Requirement already satisfied: fetchcode in ./venv/lib/python3.8/site-packages (from purldb-toolkit==0.0.1) (0.3.0)
Requirement already satisfied: requests in ./venv/lib/python3.8/site-packages (from purldb-toolkit==0.0.1) (2.31.0)
Requirement already satisfied: click in ./venv/lib/python3.8/site-packages (from purldb-toolkit==0.0.1) (8.1.7)
Requirement already satisfied: packageurl-python in /home/jmh/dev/nexb/packageurl-python/src (from purldb-toolkit==0.0.1) (0.13.4)
Requirement already satisfied: rubymarshal==1.0.3 in ./venv/lib/python3.8/site-packages (from purldb==2.0.0) (1.0.3)
Requirement already satisfied: markdown==3.5.1 in ./venv/lib/python3.8/site-packages (from purldb==2.0.0) (3.5.1)
Requirement already satisfied: univers==30.11.0 in ./venv/lib/python3.8/site-packages (from purldb==2.0.0) (30.11.0)
Collecting psycopg[binary]==3.1.17
  Downloading psycopg-3.1.17-py3-none-any.whl (178 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 178.0/178.0 kB 1.3 MB/s eta 0:00:00
Collecting django-filter==23.5
  Downloading django_filter-23.5-py3-none-any.whl (94 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94.4/94.4 kB 2.1 MB/s eta 0:00:00
Requirement already satisfied: reppy2==0.3.6 in ./venv/lib/python3.8/site-packages (from purldb==2.0.0) (0.3.6)
Collecting rq-scheduler==0.13.1
  Downloading rq_scheduler-0.13.1-py2.py3-none-any.whl (13 kB)
Requirement already satisfied: natsort==8.2.0 in ./venv/lib/python3.8/site-packages (from purldb==2.0.0) (8.2.0)
Requirement already satisfied: ftputil==5.0.4 in ./venv/lib/python3.8/site-packages (from purldb==2.0.0) (5.0.4)
Requirement already satisfied: urlpy==0.5 in ./venv/lib/python3.8/site-packages (from purldb==2.0.0) (0.5)
Requirement already satisfied: PyGithub==1.56 in ./venv/lib/python3.8/site-packages (from purldb==2.0.0) (1.56)
Collecting drf-spectacular==0.26.5
  Downloading drf_spectacular-0.26.5-py3-none-any.whl (94 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94.1/94.1 kB 2.6 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of purldb-toolkit to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of matchcode-toolkit to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install purldb[testing]==2.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    purldb[testing] 2.0.0 depends on scancodeio==33.0.0
    The user requested (constraint) scancodeio==33.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[notice] A new release of pip available: 22.2.2 -> 24.0
[notice] To update, run: pip install --upgrade pip
make: *** [Makefile:43: dev] Error 1

(venv) Fri Feb 16, 2024 01:39 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$

johnmhoran · 2024-02-16T22:58:48Z

Is this some sort of conflict re drf_spectacular-0.26.5-py3-none-any.whl ?

JonoYang · 2024-02-16T23:19:20Z

@johnmhoran I think what happened is that when I updated the dependencies and the pinned requirements in the requirements.txt and requirements-dev.txt files, that the versions of those packages chosen work on python 3.10+. I remember encountering this issue when the github tests would fail on the test instances running python 3.8. I ended up just only having the github tests run on python 3.10 and 3.11.

johnmhoran · 2024-02-16T23:31:11Z

@JonoYang But when I try to run test_purlcli.py locally now, I get

    import_module(entry)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'drf_spectacular'

As recently as late last night, all 76 of my tests passed. How am I supposed to run my tests locally? The venv Python is 3.8.10. I can no longer run my tests right now and cannot run make dev either without errors. That seems to bring my dev work to a complete stop until I can fix these errors introduced by merging the latest main earlier today.

(venv) Fri Feb 16, 2024 03:28 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ python --version
Python 3.8.10

(venv) Fri Feb 16, 2024 03:28 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$

JonoYang · 2024-02-16T23:38:12Z

@johnmhoran I'll try to redo the pinned dependencies using Python 3.8, otherwise, I'd suggest using https://github.com/pyenv/pyenv to install a more recent version of Python.

johnmhoran · 2024-02-16T23:41:36Z

@JonoYang I already use Python 3.10.13 in my "sandbox` by installing it in the venv. But the purldb venv is 3.8.10. Couldn't I just update Python in that venv? How can I help you do what you need? If purldb relies now on 3.10 because of your changes, we should just update the venv, no?

Maybe you need not pin for 3.8 if there's something I can do instead....

JonoYang · 2024-02-16T23:44:55Z

@johnmhoran Please use Python 3.10+ in your purldb venv if you can

johnmhoran · 2024-02-16T23:47:30Z

@JonoYang How can I update the venv on my branch to use 3.10+ ? I'd love to do that!

johnmhoran · 2024-02-16T23:55:30Z

or do I just need to install 3.10.13 or whatever in my WSL2 Ubuntu-20.04 distro and the purldb venv will automatically use that? I assume our venv chose the 3.8.10 that I use in my purldb branch.

johnmhoran · 2024-02-16T23:57:14Z

I'm going to try to install 3.10+ in my WSL2 Ubuntu distro and see if the purldb venv uses that instead of 3.8.10.

johnmhoran · 2024-02-17T00:04:29Z

@JonoYang I now have Python 3.10.13 but my purldb venv has not changed. How do I use 3.10.13 in that venv? I'll try to select 3.10.13 in my VSCode instance, though I don't see why that would affect what happens in my Windows Terminal instance running WSL2.

JonoYang · 2024-02-17T00:05:24Z

@johnmhoran

I use https://github.com/pyenv/pyenv to manage python on my systems. pyenv has code that allows you to have different python installs that do not interfere with the system installed python. Ive run into a problem where I updated python3 on Ubuntu and something went wrong with one of the system tools.

I followed the instructions for the basic github checkout:
https://github.com/pyenv/pyenv?tab=readme-ov-file#basic-github-checkout

then I followed the instructions for the bash shell environment configuration:
https://github.com/pyenv/pyenv?tab=readme-ov-file#set-up-your-shell-environment-for-pyenv

then I installed the Python build dependencies for Ubuntu/Debian/Mint:
https://github.com/pyenv/pyenv/wiki#suggested-build-environment

then I restart the shell:
https://github.com/pyenv/pyenv?tab=readme-ov-file#restart-your-shell

Then you can install Python 3.10.12 by running:
pyenv install 3.10.12

You can set Python 3.10.12 to be the default python/python3 run by running:
pyenv global 3.10.12

Afterwards, run make clean && make dev in your purldb directory to clean and recreate the venv

johnmhoran · 2024-02-17T00:06:27Z

VSCode wants me to stay with 3.8.10.

johnmhoran · 2024-02-17T01:00:37Z

@JonoYang Will do. To confirm: in that order, right?

make clean
python3.10 -m venv venv from the root of the purldb directory
source venv/bin/activate
pip install -e .
pip install -e purldb-toolkit

(Then what? In any event, starting right now.)

johnmhoran · 2024-02-17T01:04:25Z

@JonoYang It all ran without errors. What did I just do, and what does this tell us about the cause of all the errors I've encountered?

johnmhoran · 2024-02-17T01:07:46Z

I realize I have an editable installation, but wonder whether I only temporarily updated the venv to Python 3.10, and not sure what the purldb-toolkit install -e did. That is not what I've been doing when I worked on this repo.

JonoYang · 2024-02-17T01:09:27Z

@johnmhoran

It all ran without errors. What did I just do, and what does this tell us about the cause of all the errors I've encountered?

I think I need to update the Makefile/configure script so it really uses the python3 set in PYTHON_EXE from the Makefile

johnmhoran · 2024-02-17T01:12:22Z

Is that something I could help with? I hate to see you spend any of your time on these errors, though I'm not sure what actually caused them. Did I possibly make an error of my own when I merged main and resolved the few simple-looking conflicts? Or did some of the incoming code from main already conflict somehow with something elsewhere in purldb? I really would like to understand this experience.

johnmhoran · 2024-02-17T01:18:20Z

I think the conflicts in requirements.txt and setup.cfg involved packageurl-python (my update from 0.11.2 to 0.13.4), and several of your updates, e.g., psycopg[binary]==3.1.17 from 3.1.12.

johnmhoran · 2024-02-17T01:23:28Z

@JonoYang I tried to run some of my tests -- new error: No module named click.

(venv) Fri Feb 16, 2024 05:21 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ DJANGO_SETTINGS_MODULE=purldb_project.settings pytest -vvs purldb-toolkit/tests/test_purlcli.py::TestPURLCLI_versions
=========================================================================================================== test session starts ============================================================================================================
platform linux2 -- Python 2.7.18, pytest-4.6.9, py-1.8.1, pluggy-0.13.0 -- /usr/bin/python2
cachedir: .pytest_cache
rootdir: /home/jmh/dev/nexb/purldb/purldb-toolkit
collected 0 items / 1 errors

================================================================================================================== ERRORS ==================================================================================================================
__________________________________________________________________________________________________ ERROR collecting tests/test_purlcli.py __________________________________________________________________________________________________
ImportError while importing test module '/home/jmh/dev/nexb/purldb/purldb-toolkit/tests/test_purlcli.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python2.7/dist-packages/_pytest/python.py:507: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
/usr/lib/python2.7/dist-packages/py/_path/local.py:701: in pyimport
    __import__(modname)
/usr/lib/python2.7/dist-packages/_pytest/assertion/rewrite.py:304: in load_module
    exec(co, mod.__dict__)
purldb-toolkit/tests/test_purlcli.py:5: in <module>
    import click
E   ImportError: No module named click
========================================================================================================= 1 error in 0.15 seconds ==========================================================================================================
ERROR: not found: /home/jmh/dev/nexb/purldb/purldb-toolkit/tests/test_purlcli.py::TestPURLCLI_versions
(no name '/home/jmh/dev/nexb/purldb/purldb-toolkit/tests/test_purlcli.py::TestPURLCLI_versions' in any of [<Module tests/test_purlcli.py>])


(venv) Fri Feb 16, 2024 05:22 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$

JonoYang · 2024-02-17T01:28:31Z

@johnmhoran

Is that something I could help with? I hate to see you spend any of your time on these errors, though I'm not sure what actually caused them. Did I possibly make an error of my own when I merged main and resolved the few simple-looking conflicts? Or did some of the incoming code from main already conflict somehow with something elsewhere in purldb? I really would like to understand this experience.

The issue is that the versions of dependencies in the requirements.txt and requirements-dev.txt files do not work on Python 3.8. In your shell, python3 points to python3.8. When you updated PYTHON_EXE in the Makefile to python3.10, that was not passed to configure when the Makefile called it. In configure, it defaulted to using python3, which on your system is Python 3.8.

You can help me try out a possible fix:

Deactivate the venv and run make clean
Update the Makefile:
- https://github.com/nexB/purldb/blob/main/Makefile#L11, ensure this is PYTHON_EXE?=python3.10
- https://github.com/nexB/purldb/blob/main/Makefile#L39, ensure this is @PYTHON_EXECUTABLE=${PYTHON_EXE} ./configure
- https://github.com/nexB/purldb/blob/main/Makefile#L43, ensure this is @PYTHON_EXECUTABLE=${PYTHON_EXE} ./configure --dev
- https://github.com/nexB/purldb/blob/main/Makefile#L82, ensure this is @PYTHON_EXECUTABLE=${PYTHON_EXE} ./configure --clean
Run make dev

JonoYang · 2024-02-17T01:35:46Z

@johnmhoran I've updated the commands in my comment above. Let me know if that works or not

johnmhoran · 2024-02-17T01:42:44Z

@JonoYang I got this error when I ran make dev -- as if some other script is still looking for the old @./configure command.

Fri Feb 16, 2024 05:39 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ make dev
-> Configure and install development dependencies
PYTHON_EXECUTABLE=python3.10 @./configure --dev
/bin/sh: 1: @./configure: not found
make: *** [Makefile:43: dev] Error 127

Fri Feb 16, 2024 05:41 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$

JonoYang · 2024-02-17T01:43:58Z

@johnmhoran The @ needs to be before PYTHON_EXECUTABLE. I've made a mistake when I first wrote the instructions

johnmhoran · 2024-02-17T01:44:20Z

The error refers to Makefile 43, which now reads PYTHON_EXECUTABLE=${PYTHON_EXE} @./configure --dev

JonoYang · 2024-02-17T01:48:58Z

@johnmhoran

The error refers to Makefile 43, which now reads PYTHON_EXECUTABLE=${PYTHON_EXE} @./configure --dev

That line should be @PYTHON_EXECUTABLE=${PYTHON_EXE} ./configure --dev, and the other lines I've marked for update should lead with @ and not have @ in front of ./configure

johnmhoran · 2024-02-17T01:50:50Z

@JonoYang Yes, moved the @ to the start of each line and the make dev just succeeded. No errors. Should I try one of my purlcli commands, or one of the test suites, or wait to hear back from you for further troubleshooting/updating?

JonoYang · 2024-02-17T01:51:13Z

@johnmhoran Feel free to try anything out

johnmhoran · 2024-02-17T01:55:15Z

@JonoYang This is encouraging. ;-) About to try a regular purlcli command, then one of my pytest command groups.

Fri Feb 16, 2024 05:51 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ python --version

Command 'python' not found, did you mean:

  command 'python3' from deb python3
  command 'python' from deb python-is-python3


Fri Feb 16, 2024 05:51 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ python3 --version
Python 3.8.10

Fri Feb 16, 2024 05:52 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ python3.10 --version
Python 3.10.13

Fri Feb 16, 2024 05:52 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ source venv/bin/activate

(venv) Fri Feb 16, 2024 05:52 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$ python --version
Python 3.10.13

(venv) Fri Feb 16, 2024 05:52 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$

johnmhoran · 2024-02-17T02:00:04Z

@JonoYang My command worked and my first sample test worked. But the test also output many many lines like this. This is new behavior -- maybe the deprecations come from 3.8 >> 3.10?

purldb-toolkit/tests/test_purlcli.py::TestPURLCLI_versions::test_check_versions_purl[test_input7-None] PASSED
purldb-toolkit/tests/test_purlcli.py::TestPURLCLI_versions::test_check_versions_purl[test_input8-valid_but_not_supported] PASSED

============================================================================================================= warnings summary =============================================================================================================
minecode/mappers/apache.py:170
  /home/jmh/dev/nexb/purldb/minecode/mappers/apache.py:170: DeprecationWarning: invalid escape sequence '\.'
    @map_router.route('http?s://(archive\.)?apache\.org/dist/.*')

venv/lib/python3.10/site-packages/packagedcode/gemfile_lock.py:344
  /home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/packagedcode/gemfile_lock.py:344: DeprecationWarning: invalid escape sequence '\s'
    BUNDLED_WITH = re.compile('^\s+(?P<version>(?:\d+.)+\d+)\s*$').match

johnmhoran · 2024-02-17T02:01:33Z

@JonoYang What's next? Commit my Makefile changes, push, and hope all the GH tests pass and the warning This branch has conflicts that must be resolved goes away?

JonoYang · 2024-02-17T02:05:16Z

@johnmhoran I will merge https://github.com/nexB/purldb/tree/update-makefile into main, and then you pull main into your branch

johnmhoran · 2024-02-17T02:08:52Z

Will do @JonoYang . I'm running my full set of purlcli tests now. (I don't think these run when I run make test.) Ah good:

76 passed, 1 warning in 199.97s (0:03:19)

johnmhoran · 2024-02-17T02:13:46Z

@JonoYang When you ping me I'll pull main then git merge into my branch and resolve any conflicts. Then what?

Just commit and push and watch the GH tests run? Or before that do I need to do another deactivate then make clean then make dev then source venv/bin/activate?

JonoYang · 2024-02-17T02:19:48Z

@johnmhoran I've merged in the changes into main. Please merge them into your branch and then push it to github.

Or before that do I need to do another deactivate then make clean then make dev then source venv/bin/activate?

I dont think you need to do this again

Reference: #247 Signed-off-by: John M. Horan johnmhoran@gmail.com

johnmhoran · 2024-02-17T02:33:05Z

@JonoYang Just pushed.

johnmhoran · 2024-02-17T02:38:49Z

@JonoYang All GH checks passed and This branch has no conflicts with the base branch.

johnmhoran · 2024-02-17T02:46:54Z

@JonoYang I ran make test and just the one expected failure.

FAILED minecode/tests/test_maven.py::MavenEnd2EndTest::test_visit_and_map_with_index - AssertionError: Lists differ: [{'ur[31 chars]ven2/cnuernber/dtype-next/0.4.2/dtype-next-0.4[49087 chars]one}] != [{'ur[31 chars]ven2/.index/nexus-maven-repository-index.532.g[49087 chars]one}]
XFAIL minecode/tests/test_debian.py::DebutilsTest::test_debcon_get_paragraph_data_from_file_control_basic - reason:
XFAIL minecode/tests/test_debian.py::DebutilsTest::test_debcon_get_paragraph_data_from_file_control_invalid - reason:
XFAIL minecode/tests/test_debian.py::DebutilsTest::test_debcon_get_paragraph_data_from_file_with_non_existing_path - reason:
XFAIL minecode/tests/test_debian.py::DebianReleaseTest::test_parse_release - reason:
XFAIL minecode/tests/test_debian.py::DebianReleaseTest::test_parse_release_with_md5 - reason:
XFAIL minecode/tests/test_debian.py::DebianReleaseTest::test_visit_debian_release - reason:
XFAIL minecode/tests/test_debian.py::DebianCopyrightTest::test_parse_copyright_allinfo_basic - reason:
XFAIL minecode/tests/test_debian.py::DebianCopyrightTest::test_parse_copyright_allinfo_with_incorrect_path - reason:
XFAIL minecode/tests/test_debian.py::DebianCopyrightTest::test_parse_copyright_allinfo_with_invalid_file - reason:
XFAIL minecode/tests/test_debian.py::DebianCopyrightTest::test_parse_copyright_only_basic - reason:
XFAIL minecode/tests/test_debian.py::DebianCopyrightTest::test_parse_copyright_only_with_incorrect_file - reason:
XFAIL minecode/tests/test_debian.py::DebianCopyrightTest::test_parse_copyright_only_with_incorrect_path - reason:
XFAIL minecode/tests/test_debian.py::DebianCopyrightTest::test_parse_license_basic - reason:
XFAIL minecode/tests/test_debian.py::DebianCopyrightTest::test_parse_license_with_incorrect_path - reason:
XFAIL minecode/tests/test_debian.py::DebianCopyrightTest::test_parse_license_with_invalid_file - reason:
XFAIL minecode/tests/test_debian.py::DebianSourcesTest::test_DebianSourcesVisitor - reason:
XFAIL minecode/tests/test_debian.py::DebianSourcesTest::test_DebianSourcesVisitor_with_invalid_file - reason:
XFAIL minecode/tests/test_debian.py::DebianSourcesTest::test_build_source_file_packages - reason:
XFAIL minecode/tests/test_debian.py::DebianPackagesTest::test_parse_packages_from_debian_Packages - reason:
XFAIL minecode/tests/test_debian.py::DebianPackagesTest::test_parse_packages_from_installed_status - reason:
XFAIL minecode/tests/test_debian.py::DebianPackagesTest::test_parse_packages_from_ubuntu_Packages - reason:
XFAIL minecode/tests/test_debian.py::DebianDescriptionTest::test_DebianDescriptionVisitor - reason:
XFAIL minecode/tests/test_debian.py::DebianDescriptionTest::test_parse_description - reason:
XFAIL minecode/tests/test_debian.py::DebianMapperTest::test_get_dependencies - reason:
XFAIL minecode/tests/test_openwrt.py::OpenWRTVistorTest::test_visitor_openwrt_ipk - reason:
XFAIL minecode/tests/test_openwrt.py::OpenWRTVistorTest::test_visitor_openwrt_ipk2 - reason:
XFAIL minecode/tests/test_openwrt.py::OpenWRTVistorTest::test_visitor_openwrt_packages_gz - reason:
XFAIL minecode/tests/test_openwrt.py::OpenWRTMapperTest::test_build_packages_1 - reason:
XFAIL minecode/tests/test_openwrt.py::OpenWRTMapperTest::test_build_packages_2 - reason:
XFAIL minecode/tests/test_version.py::VersionHintTestCase::test_versions_corner_cases2 - reason:
=============================================================================== 1 failed, 592 passed, 2 skipped, 30 xfailed, 80 warnings in 91.46s (0:01:31) ===============================================================================
make: *** [Makefile:133: test] Error 1

(venv) Fri Feb 16, 2024 06:45 PM  /home/jmh/dev/nexb/purldb jmh (247-purl-cli-add-urls)
$

pombredanne

LGTM!
I think this is time to merge this.

Add cli commands, restructure data and update code and tests

johnmhoran added 2 commits January 30, 2024 17:29

Update 'meta' code and tests including data structure #247

11e0db9

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Add local .txt test purl input file #247

4da30a7

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

johnmhoran mentioned this pull request Jan 31, 2024

Create PURL services CLI tool and library #247

Closed

JonoYang requested changes Feb 1, 2024

View reviewed changes

johnmhoran added 4 commits January 31, 2024 21:38

Create construct_headers() #247

0608044

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Add initial meta normalization code #247

51ade91

Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Add --unique option for metadata command #247

ab4c9b6

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Replace repeated function calls with variables #247

2b4e442

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

pombredanne reviewed Feb 15, 2024

View reviewed changes

Update data vetting and warnings, add urls tests #247

8aa6fcb

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Update setup.cfg to packageurl-python == 0.13.4 #247

4261dd5

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

johnmhoran added 2 commits February 16, 2024 13:10

Update requirements.txt packageurl-python #247

eeadc63

Reference: #247 Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Merge branch 'main' into 247-purl-cli-add-urls

db9b01c

Signed-off-by: John M. Horan <johnmhoran@gmail.com>

Merge branch 'main' into 247-purl-cli-add-urls

745b24f

Reference: #247 Signed-off-by: John M. Horan johnmhoran@gmail.com

pombredanne approved these changes Feb 18, 2024

View reviewed changes

pombredanne merged commit e317b56 into main Feb 18, 2024

pombredanne deleted the 247-purl-cli-add-urls branch February 18, 2024 17:23

JonoYang pushed a commit that referenced this pull request Apr 14, 2025

Merge pull request #281 from nexB/247-purl-cli-add-urls

f266ee9

Add cli commands, restructure data and update code and tests

Uh oh!

Conversation

johnmhoran commented Jan 31, 2024

Uh oh!

JonoYang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johnmhoran commented Feb 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pombredanne Feb 15, 2024

Choose a reason for hiding this comment

Uh oh!

johnmhoran Feb 15, 2024

Choose a reason for hiding this comment

Uh oh!

johnmhoran commented Feb 16, 2024

Uh oh!

johnmhoran commented Feb 16, 2024

Uh oh!

johnmhoran commented Feb 16, 2024

Uh oh!

johnmhoran commented Feb 16, 2024

Uh oh!

johnmhoran commented Feb 16, 2024

Uh oh!

JonoYang commented Feb 16, 2024

Uh oh!

johnmhoran commented Feb 16, 2024

Uh oh!

JonoYang commented Feb 16, 2024

Uh oh!

johnmhoran commented Feb 16, 2024

Uh oh!

JonoYang commented Feb 16, 2024

Uh oh!

johnmhoran commented Feb 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnmhoran commented Feb 16, 2024

Uh oh!

johnmhoran commented Feb 16, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

JonoYang commented Feb 17, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

JonoYang commented Feb 17, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

JonoYang commented Feb 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonoYang commented Feb 17, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

JonoYang commented Feb 17, 2024

Uh oh!

johnmhoran commented Feb 17, 2024

Uh oh!

JonoYang commented Feb 17, 2024

johnmhoran commented Feb 1, 2024 •

edited

Loading

johnmhoran commented Feb 16, 2024 •

edited

Loading

JonoYang commented Feb 17, 2024 •

edited

Loading