Meta-ticket: Adopt mainstream Python testing/linting infrastructure: tox, pytest, ..., describe in Developer's Guide

[mkoeppe]

This metaticket proposes to use mainstream Python testing and linting infrastructure for Sage -- by adding configuration files and documentation.

tox:

• Add src/tox.ini as a standard entrypoint for doctesting #28987 Add src/tox.ini: As an alternative interface to running the sage doctester.
• src/tox.ini: Check patchbot plugin patterns and common typo patterns #30467 src/tox.ini: Check patchbot plugin patterns
• Command "sage -tox" #30410 (Command "sage -tox")
• Make tox a standard package #30416 (Make tox a standard package)
• Document "sage -tox" #30453 Document "sage -tox"
• src/tox.ini: Add environment pyright #30411 (src/tox.ini: Add environment pyright)
• src/tox.ini: Add validation of docstrings using flake8-rst-docstrings #30448 (src/tox.ini: Add validation of rst files and docstrings via a quick docbuild)

pytest:

• Add minimal pytest configuration #31003 Add minimal pytest configuration
• sage.numerical.backends: Replace use of TestSuite by pytest #31103 sage.numerical.backends: Replace use of TestSuite by pytest
• Add symplectic structures #30362 Add symplectic structures
• Improve doctest interaction with pytest #32975 Improve doctest interaction with pytest
• Rename test_... functions to avoid the naming scheme reserved by pytest #33549 Rename test_... functions to avoid the naming scheme reserved by pytest
• sage -t: Do not run pytest on individual Python files unless they match the pytest file pattern #31924 sage -t: Do not run pytest on individual Python files unless they match the pytest file pattern
• sage -pytest #33572 sage -pytest
• pytest: ignore cython files #33560 pytest: ignore cython files
• Rename test_... classes to avoid the naming scheme reserved by pytest #33561 Rename test_... classes to avoid the naming scheme reserved by pytest
• Preparse sage files before passing them to pytest #33550 Preparse sage files before passing them to pytest
• Document pytest #31123 Document pytest
• doctesting with pytest fails on system install of sage #33521 doctesting with pytest fails on system install of sage
• pytest in sage is broken, please make it really optional #33531 pytest in sage is broken, please make it really optional
• Make pytest a standard package #31110 Make pytest a standard package
• Use pytest to run doctests #33546 Replace custom Sage doctest discovery by pytest (using pytest_collect_file)
• Replace custom Sage unit test discovery by pytest #30738 Replace custom Sage unit test discovery by pytest
• sage.geometry.polyhedron: Replace use of TestSuite by pytest #31104 sage.geometry.polyhedron: Replace use of TestSuite by pytest

Other tickets:

• "sage -startuptime" should accept file names too #30452 ("sage -startuptime" should accept file names too)

• Add pyright config and linting documentation #30361 (Add pyright config and linting documentation; dup: sage developer's guide should mention pyflakes #17992)

• sage -t : Write report in JUnitXML format #33232 (sage -t: Write report in JUnitXML format)

• Remove pyflakes? #30415 (Remove pyflakes?)

Documentation tickets:

• Update section "Sage Development Process" in Developer's Guide #29784 Update section "Sage Development Process" in Developer's Guide

• Mention .lgtm.yml in the Developer's Guide #29520: Mention .lgtm.yml in the Developer's Guide

Infrastructure tickets:

• Meta-ticket: Migration from Trac to GitHub #30363 Replace patchbot by github actions (part of https://trac.sagemath.org/ticket/)

• Use pre-commit https://pre-commit.com/

• Use pipenv in place of venv https://pipenv.pypa.io/en/latest/#

See also:

• Meta-ticket Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages #29060: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

• Generalize sage.numerical.backends.logging_backend.LoggingBackend to help generate _test methods

• Add script-type _spkg_check package that installs tox, mock, pytest etc. #28998: Add script-type check_dependencies package that installs tox, mock, pytest etc.

• Extensive description of different linters: https://github.com/vintasoftware/python-linters-and-code-analysis

CC: @jdemeyer @dimpase @embray @videlec @fchapoton @tscrim @jm58660 @tobiasdiez @kliem @nthiery @tornaria

Component: doctest framework

Issue created by migration from https://trac.sagemath.org/ticket/28936

[embray]

Replying to @mkoeppe:

tox seems to be the preferred interface for running tests - see for example pypa/setuptools#931 (deprecation of setup.py test).

I don't know what you mean by "preferred interface" here. There isn't really any such thing as a preferred interface. It's just being proposed here as a recommendation over setup.py test. Sage doesn't use setup.py test anyways so there's nothing that needs to be "replaced" with a different interface if/when setup.py goes away.

The main advantage to tox is the ability to run a package's tests across multiple runtime environments (esp. different Python versions). This isn't really possible currently with Sage; it would actually be very difficult to use tox in any sensible manner with Sage.

If someone wants to do this anyways I don't see the harm really, but I'm not sure what problem it would be solving.

[mkoeppe]

comment:4

A first step: #28987

[mkoeppe]

comment:5

Replying to @embray:

Replying to @mkoeppe:
The main advantage to tox is the ability to run a package's tests across multiple runtime environments (esp. different Python versions). This isn't really possible currently with Sage; it would actually be very difficult to use tox in any sensible manner with Sage.

FWIW, I managed to use tox on top of the sage installation, see https://github.com/mkoeppe/sage-numerical-backends-gurobi/blob/master/tox.ini: The trick is to run tox with sitepackages = true and an install_command that installs into the user scheme. With this trick, after all, no new plugin for tox is needed to handle the sage environment.

[mkoeppe]

comment:6

I have broadened the ticket description.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,6 +1,18 @@
-`tox` seems to be the preferred interface for running tests - see for example https://github.com/pypa/setuptools/issues/931 (deprecation of `setup.py test`).
+`tox` is a popular package for advertising the testing/linting infrastructure of a Python project by listing them in one configuration file `tox.ini`; it supports running tests in isolated virtual environments.
 
-We should create a `tox` plugin (https://tox.readthedocs.io/en/latest/plugins.html)
-- for testing `sagelib` (as an alternative to `sage -t`)
-- for users' packages that depend on sage. 
+A typical example of `tox.ini`: https://opendev.org/openstack/neutron/raw/branch/master/tox.ini
 
+This metaticket proposes to use `tox` and other mainstream Python testing and linting infrastructure for Sage.
+
+Tickets:
+
+- #28987 Add `src/tox.ini`: As an alternative interface to running the sage doctester.
+
+- Add linters equivalent to those reported by patchbot: https://github.com/sagemath/sage-patchbot/blob/master/sage_patchbot/plugins.py:
+
+  So that users can check style locally instead of going through endless cycles with the patchbot on trac.
+
+- Use pre-commit https://pre-commit.com/
+
+- #28998: Add script-type check_dependencies package that installs tox, mock, pytest etc.
+

[mkoeppe]

Description changed:

--- 
+++ 
@@ -5,6 +5,8 @@
 This metaticket proposes to use `tox` and other mainstream Python testing and linting infrastructure for Sage.
 
 Tickets:
+
+- #29053: Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add `SAGE_ROOT/tox.ini`
 
 - #28987 Add `src/tox.ini`: As an alternative interface to running the sage doctester.
 

[mkoeppe]

comment:8

pushing these forward to 9.2

[mkoeppe]

Description changed:

--- 
+++ 
@@ -7,6 +7,8 @@
 Tickets:
 
 - #29053: Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add `SAGE_ROOT/tox.ini`
+
+- #29520: Mention .lgtm.yml in the Developer's Guide
 
 - #28987 Add `src/tox.ini`: As an alternative interface to running the sage doctester.
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -6,7 +6,7 @@
 
 Tickets:
 
-- #29053: Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add `SAGE_ROOT/tox.ini`
+- #29053: Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add `SAGE_ROOT/tox.ini`; see also meta-tickets #29146, #29060. 
 
 - #29520: Mention .lgtm.yml in the Developer's Guide
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -20,3 +20,4 @@
 
 - #28998: Add script-type check_dependencies package that installs tox, mock, pytest etc.
 
+- Document pyrightconfig.json ... as added by #29775

[mkoeppe]

Description changed:

--- 
+++ 
@@ -21,3 +21,5 @@
 - #28998: Add script-type check_dependencies package that installs tox, mock, pytest etc.
 
 - Document pyrightconfig.json ... as added by #29775
+
+- Generalize `sage.numerical.backends.logging_backend.LoggingBackend` to help generate `_test` methods

[tobiasdiez]

Description changed:

--- 
+++ 
@@ -20,6 +20,6 @@
 
 - #28998: Add script-type check_dependencies package that installs tox, mock, pytest etc.
 
-- Document pyrightconfig.json ... as added by #29775
+- Document pyrightconfig.json ... as added by #30361
 
 - Generalize `sage.numerical.backends.logging_backend.LoggingBackend` to help generate `_test` methods

[tobiasdiez]

Description changed:

--- 
+++ 
@@ -23,3 +23,5 @@
 - Document pyrightconfig.json ... as added by #30361
 
 - Generalize `sage.numerical.backends.logging_backend.LoggingBackend` to help generate `_test` methods
+
+- Replace patchbot by github actions (part of https://github.com/sagemath/sage-prod/issues/30363)

[mkoeppe]

Description changed:

--- 
+++ 
@@ -12,6 +12,8 @@
 
 - #28987 Add `src/tox.ini`: As an alternative interface to running the sage doctester.
 
+- #29784 Update section "Sage Development Process" in Developer's Guide
+
 - Add linters equivalent to those reported by patchbot: https://github.com/sagemath/sage-patchbot/blob/master/sage_patchbot/plugins.py:
 
   So that users can check style locally instead of going through endless cycles with the patchbot on trac.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,8 +1,4 @@
-`tox` is a popular package for advertising the testing/linting infrastructure of a Python project by listing them in one configuration file `tox.ini`; it supports running tests in isolated virtual environments.
-
-A typical example of `tox.ini`: https://opendev.org/openstack/neutron/raw/branch/master/tox.ini
-
-This metaticket proposes to use `tox` and other mainstream Python testing and linting infrastructure for Sage.
+This metaticket proposes to use mainstream Python testing and linting infrastructure for Sage -- by adding configuration files and documentation.
 
 Tickets: