Automatically use requirements.in if the project uses a requirements.txt + requirements.in-setup

[fpgmaas]

PR Checklist

• A description of the changes is added to the description of this PR.
• If there is a related issue, make sure it is linked to this PR.
• If you've fixed a bug or added code that should be tested, add tests!
• Documentation in docs is updated

Description of changes

This implements a solution that was proposed by @mkniewallner for the issue described in #608. That is; use a same approach as we use for the exclude parameter to determine if a user specified the argument requirement-files. If no input is provided by the user for this argument and the project contains both a requirements.txt and a requirements.in, use the requirements.in.

Considerations:

• When do we perform the check for a requirements.in file?
  • Before DependencySpecificationDetector().detect(): This would make sure that deptry also works if the project only contains a requirements.inand no requirements.txt. But this will log to the user that the requirements.in will be used before theDependencySpecificationDetector has run. This will be a lie if the project detects that Poetry is used for dependency management in the DependencySpecificationDetector.
  • In the DependencySpecificationDetector -> this seems to violate SRP; now it not only detects, but also overwrites the requirements_files argument if it decides that is necessary.
  • After DependencySpecificationDetector().detect(): This means it does not work automatically if the project only contains a requirements.inand no requirements.txt. This should be a rare case, so this solution seems like the best solution to me.

---

If the user does not explicitly set the requirements-files argument, the dependency format is detected as requirements-file because the project contains a requirements.txt, and requirements.in is detected, the user will see the following message:

Detected a 'requirements.in' file in the project and no 'requirements-files' were explicitly specified. Automatically using 'requirements.in' as the source for the project's dependencies. To specify a different source for the project's dependencies, use the '--requirements-files' option.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.4%. Comparing base (714596c) to head (3a5edec).

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #641   +/-   ##
=====================================
  Coverage   92.4%   92.4%           
=====================================
  Files         35      35           
  Lines        950     958    +8     
  Branches     175     176    +1     
=====================================
+ Hits         878     886    +8     
  Misses        55      55           
  Partials      17      17           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mkniewallner]

Reviewing this made me think that we could simplify a bit the dependency specification detector by getting rid of the intermediate DependencyFormat enum: #654.

Aside from this refactoring, I'm wondering about 2 things:

• In case requirements_files is not set by the user, should we first try to parse a requirements.in even if there is no requirements.txt, as it's possible to manually set a different name for the resulting .txt file (for instance with pip-compile -o requirements-prod.txt)?
• Should we apply the same logic of trying to find .in first for requirements_files_dev in case the default value is left untouched by the user?

[fpgmaas]

In case requirements_files is not set by the user, should we first try to parse a requirements.in even if there is no requirements.txt, as it's possible to manually set a different name for the resulting .txt file (for instance with pip-compile -o requirements-prod.txt)?

Yes, definitely! In the earlier approach I did not do that because of the way we had the DependencySpecificationDetector set-up, see also the considerations in the PR description about when the check should run. With your addition of the DependencyGetterBuilder, this is now solved and we can just do a check for a requirements.in file.

Should we apply the same logic of trying to find .in first for requirements_files_dev in case the default value is left untouched by the user?

I think we should not do that, for two reasons:

• I think it is quite uncommon to pin the development requirements files like this; For example, Google finds 133,000,000 hits for requirements.in but only 1,840 hits for requirements-dev.in. I think with the current implementation we tackle >90% of cases by default, if people really want to have their development dependencies read from a different file they can of course always still use the CLI arguments.
• The impact of using the incorrect requirements file is much larger than using the incorrect requirements-dev file; in the first case, deptry will raise a DEP002 error for all transitive dependencies. In the latter case, there will only be an incorrect violation raise if a transitive development dependency is used as a regular import in the project.

[mkniewallner]

Nice work 🙂