feat: find Python files in Rust

[mkniewallner]

PR Checklist

• A description of the changes is added to the description of this PR.
• If there is a related issue, make sure it is linked to this PR.
• If you've fixed a bug or added code that should be tested, add tests!
• Documentation in docs is updated

Description of changes

Leverage Rust to find Python files by using https://crates.io/crates/ignore. The crate provides similar options to https://pypi.org/project/pathspec/ in order to handle .gitignore while also handling files walking. For the exclusion, the regex crate that we already depend on is used, to match the behaviour we have with the Python implementation. At some point, although this would be a breaking change, I would really like to move to globsets instead of regexes.

In terms of performance, testing the changes on a repository with ~7k files shows a 27% speedup, dropping from 1.43s to 1.04s.

[codecov]

Codecov Report

Attention: Patch coverage is 50.00000% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 90.0%. Comparing base (4f697a1) to head (8fd8a07).

❗ Current head 8fd8a07 differs from pull request most recent head 920fa9c. Consider uploading reports for the commit 920fa9c to get more accurate results

Files	Patch %	Lines
python/deptry/core.py	28.5%	5 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #591     +/-   ##
=======================================
- Coverage   91.3%   90.0%   -1.4%     
=======================================
  Files         34      33      -1     
  Lines        996     951     -45     
  Branches     202     191     -11     
=======================================
- Hits         910     856     -54     
- Misses        69      78      +9     
  Partials      17      17             

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fpgmaas]

Looks good! I see that the verbose logging has become a bit more verbose:

Collecting Python files to scan...
built glob set; 0 literals, 0 basenames, 0 extensions, 0 prefixes, 0 suffixes, 2 required extensions, 0 regexes
glob converted to regex: Glob { glob: "**/*.py[cod]", re: "(?-u)^(?:/?|.*/)[^/]*\\.py[cod]$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('.'), Literal('p'), Literal('y'), Class { negated: false, ranges: [('c', 'c'), ('o', 'o'), ('d', 'd')] }]) }
glob converted to regex: Glob { glob: "**/.coverage.*", re: "(?-u)^(?:/?|.*/)\\.coverage\\.[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('.'), Literal('c'), Literal('o'), Literal('v'), Literal('e'), Literal('r'), Literal('a'), Literal('g'), Literal('e'), Literal('.'), ZeroOrMore]) }
built glob set; 5 literals, 63 basenames, 10 extensions, 0 prefixes, 0 suffixes, 2 required extensions, 2 regexes
whitelisting python/deptry/config.py: Whitelist(IgnoreMatch(Types(Glob(Matched { def: FileTypeDef { name: "python", globs: ["*.py"] } }))))
whitelisting python/deptry/dependency_getter/pdm.py: Whitelist(IgnoreMatch(Types(Glob(Matched { def: FileTypeDef { name: "python", globs: ["*.py"] } }))))
...
Python files to scan for imports:
python/deptry/config.py
python/deptry/dependency_getter/pdm.py
python/deptry/dependency_getter/pep_621.py

Do we want that level of detail exposed to the user in the verbose logging?

[mkniewallner]

Looks good! I see that the verbose logging has become a bit more verbose:

Collecting Python files to scan...
built glob set; 0 literals, 0 basenames, 0 extensions, 0 prefixes, 0 suffixes, 2 required extensions, 0 regexes
glob converted to regex: Glob { glob: "**/*.py[cod]", re: "(?-u)^(?:/?|.*/)[^/]*\\.py[cod]$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('.'), Literal('p'), Literal('y'), Class { negated: false, ranges: [('c', 'c'), ('o', 'o'), ('d', 'd')] }]) }
glob converted to regex: Glob { glob: "**/.coverage.*", re: "(?-u)^(?:/?|.*/)\\.coverage\\.[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('.'), Literal('c'), Literal('o'), Literal('v'), Literal('e'), Literal('r'), Literal('a'), Literal('g'), Literal('e'), Literal('.'), ZeroOrMore]) }
built glob set; 5 literals, 63 basenames, 10 extensions, 0 prefixes, 0 suffixes, 2 required extensions, 2 regexes
whitelisting python/deptry/config.py: Whitelist(IgnoreMatch(Types(Glob(Matched { def: FileTypeDef { name: "python", globs: ["*.py"] } }))))
whitelisting python/deptry/dependency_getter/pdm.py: Whitelist(IgnoreMatch(Types(Glob(Matched { def: FileTypeDef { name: "python", globs: ["*.py"] } }))))
...
Python files to scan for imports:
python/deptry/config.py
python/deptry/dependency_getter/pdm.py
python/deptry/dependency_getter/pep_621.py

Do we want that level of detail exposed to the user in the verbose logging?

Unfortunately I don't think there's a way to control that, at least not easily. Setting debug log level mode in Python will also set the same level in Rust through pyo3, so both languages will output debug logs when enabling debug mode.