Repository navigation
test(packaging): bind CONTRIBUTING.md's suite section to the shared list - #1103
Conversation
The "Running the tests" section sent readers to `.github/workflows/ci.yml` for "the full set" -- ci.yml names 21 suite paths while `scripts/test_suites.py` holds 86, and the two checks beside this one assert the workflows invoke that module instead of naming suites. It also transcribed three of ten `PYTEST_SUITES` members as if that were the set, and named three of the seven exempt suites. Rewrite the section around the two lists it now derives, and add the derivation: the pointer, the existence of every path the page names, each bash example against the list its runner belongs to (NandhaKishorM#374's trap is that `python tests/x.py` on a pytest suite asserts nothing and exits 0), and set equality with `EXEMPT_TEST_SUITES` so an exemption cannot be added or dropped quietly.
|
Merged in 0.4.3. Pointing contributors at Transcribing three of ten pytest suites and three of seven hardware and weights suites is the drift that deriving the section from the two lists prevents. 138 checks to 145 in Leaving Verified on the merged tree: all 87 suites pass and |
What
CONTRIBUTING.md's "Running the tests" section said the full suite set lives in.github/workflows/ci.yml. It does not:scripts/test_suites.pyis the one list (SCRIPT_SUITES76 +
PYTEST_SUITES10 = 86 suites), and the check immediately above the new ones intests/test_packaging.pyalready fails if either workflow stops invoking that module. The section alsotranscribed the pytest-based suites as three names (of ten) and the hardware/weights suites as three
names (of seven).
The page now points at the module and describes the two lists, and
tests/test_packaging.pyderivesthe section from them instead of trusting the prose -- 138 checks to 145. No behavior changes, no
suite added or removed, and
scripts/test_suites.pyis untouched on purpose (ten open PRs edit it).Why
Three concrete drifts, all measured on
6880462:tests/test_*.pypaths. Five of them (test_evals.py,test_evals_api.py,test_evals_onnx.py,test_evals_shortlist.py,test_onnx_quantize.py) are not in the shared listat all, and the shared list holds 86 -- so a contributor following the page's link reads a partial
lane list and believes they have the set.
tests/test_serve.py,tests/test_onnx.pyandtests/test_truncation_direction.py" named 3 of 10, andtests/test_onnx.pyis not even inPYTEST_SUITES-- it is run under pytest byevals.yml:71, which the sentence does not say.EXEMPT_TEST_SUITESexempts seven suites; the page named three of them under "need hardware orlocal weights". The four it never mentioned are
tests/test_compile_cuda.pyandtests/test_fast_cpu.py, which no workflow runs at all, andtests/test_onnx.pyandtests/test_server_example.py, which run only in theevals.ymllane (python -m pytest tests/test_onnx.py -qat line 71,python tests/test_server_example.pyat line 78) rather than inthe PR
testjob.The gate is the contribution; the prose is the symptom. Two mechanisms keep it from drifting back:
#374's trap, applied to the contributor page:python tests/<name>.pyon a pytest suite definesthe test functions, asserts nothing and exits 0. So each
python tests/x.pyline on the page mustname a
SCRIPT_SUITESmember, and eachpython -m pytest tests/x.pymust name something that ispytest-based --
PYTEST_SUITESor pytest-invoked in any lane (workflow comments mention pytesttoo, so only run lines count; that distinction is load-bearing,
evals.ymlhas two comment linesthat name suites).
without gaining a line on the page is red, and a page line whose exemption was removed is red. That
is what M5 below tests by editing the dict rather than the page.
How it was verified
The battery (
laya-bench/mutate_contrib_gate.py; every Python mutantast.parse-guarded, every filerestored from HEAD,
git status --porcelainclean at the end):python tests/test_serve.pyadded to the script examplescontrib/every script example is a script suitetests/test_router.py)contrib/every pytest example is a pytest-based suitecontrib/the page lists every exempt suite, and only thosecontrib/every suite the page names exists(plus the pytest-example check, since the same string is in that command)EXEMPT_TEST_SUITESgains an exemption the page never mentionscontrib/the page lists every exempt suite, and only thosescripts/test_suites.py->scripts/test_suite.py)md/no link to a file that does not existcontrib/the two shared lists parse: 0 script, 10 pytest, and the script-example check goes red with itTwo guards are there because the set-difference checks cannot report their own input: if either
SCRIPT_SUITES/PYTEST_SUITESregex ever stops matching, the differences are computed against anempty set and the failures name the page rather than the parse.
contrib/the two shared lists parsesays it directly -- M7 shows the pair it produces -- and the page's new
[scripts/test_suites.py](scripts/test_suites.py)link is covered by this suite's existing markdownlink check, which walks every
*.mdin the tree (M6).Deconfliction: of the 64 open PRs, none touch
CONTRIBUTING.md.tests/test_packaging.pyis touchedby #1073 at one hunk,
@@ -594,7 +594,8 @@ def option_type(opt); this diff ends at line 338, so the twonever meet.
scripts/test_suites.pyis deliberately not edited (#1095, #977, #970, #923, #922, #921,#908, #729, #687 all add entries to it).
Diff scanned for credentials: 143 lines, 0 hits.
Checklist
mainFixes: no open issue.
AGENTS.md's suite table names three of the seven exempt suites and is accurateas far as it goes, so it is left alone rather than half-rewritten here.