Skip to content

test(packaging): bind CONTRIBUTING.md's suite section to the shared list - #1103

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-contrib-suite-list-source
Oct 11, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-contrib-suite-list-source

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

CONTRIBUTING.md's "Running the tests" section said the full suite set lives in
.github/workflows/ci.yml. It does not: scripts/test_suites.py is the one list (SCRIPT_SUITES
76 + PYTEST_SUITES 10 = 86 suites), and the check immediately above the new ones in
tests/test_packaging.py already fails if either workflow stops invoking that module. The section also
transcribed the pytest-based suites as three names (of ten) and the hardware/weights suites as three
names (of seven).

The page now points at the module and describes the two lists, and tests/test_packaging.py derives
the section from them instead of trusting the prose -- 138 checks to 145. No behavior changes, no
suite added or removed, and scripts/test_suites.py is untouched on purpose (ten open PRs edit it).

Why

Three concrete drifts, all measured on 6880462:

  • ci.yml names 21 tests/test_*.py paths. Five of them (test_evals.py, test_evals_api.py,
    test_evals_onnx.py, test_evals_shortlist.py, test_onnx_quantize.py) are not in the shared list
    at all, and the shared list holds 86 -- so a contributor following the page's link reads a partial
    lane list and believes they have the set.
  • "A few are pytest based and run with tests/test_serve.py, tests/test_onnx.py and
    tests/test_truncation_direction.py" named 3 of 10, and tests/test_onnx.py is not even in
    PYTEST_SUITES -- it is run under pytest by evals.yml:71, which the sentence does not say.
  • EXEMPT_TEST_SUITES exempts seven suites; the page named three of them under "need hardware or
    local weights". The four it never mentioned are tests/test_compile_cuda.py and
    tests/test_fast_cpu.py, which no workflow runs at all, and tests/test_onnx.py and
    tests/test_server_example.py, which run only in the evals.yml lane (python -m pytest tests/test_onnx.py -q at line 71, python tests/test_server_example.py at line 78) rather than in
    the PR test job.

The gate is the contribution; the prose is the symptom. Two mechanisms keep it from drifting back:

  • #374's trap, applied to the contributor page: python tests/<name>.py on a pytest suite defines
    the test functions, asserts nothing and exits 0. So each python tests/x.py line on the page must
    name a SCRIPT_SUITES member, and each python -m pytest tests/x.py must name something that is
    pytest-based -- PYTEST_SUITES or pytest-invoked in any lane (workflow comments mention pytest
    too, so only run lines count; that distinction is load-bearing, evals.yml has two comment lines
    that name suites).
  • Exemptions are checked as a set difference, both directions: a suite that gains an exemption
    without gaining a line on the page is red, and a page line whose exemption was removed is red. That
    is what M5 below tests by editing the dict rather than the page.

How it was verified

python tests/test_packaging.py     # 145 passed, 0 failed   (138 -> 145: the 7 new contrib/ checks)
python scripts/test_suites.py      # all 86 suites passed; pytest leg 469 passed, 1 skipped in 16.51s
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120    # All checks passed!
python -m compileall -q laya/ tests/

# the reproduction: the new gate against main's page, with only the page swapped back
git show 6880462:CONTRIBUTING.md > CONTRIBUTING.md && python tests/test_packaging.py
#   142 passed, 3 failed
#     FAIL contrib/the page points at the one shared suite list: CONTRIBUTING.md never names it
#     FAIL contrib/no sentence sends readers to a workflow for the full set
#     FAIL contrib/the page lists every exempt suite, and only those

The battery (laya-bench/mutate_contrib_gate.py; every Python mutant ast.parse-guarded, every file
restored from HEAD, git status --porcelain clean at the end):

mutant result
M1 -- python tests/test_serve.py added to the script examples KILLED -- contrib/every script example is a script suite
M2 -- the pytest example line switched to a script suite (tests/test_router.py) KILLED -- contrib/every pytest example is a pytest-based suite
M3 -- one exempt suite dropped from the page's list KILLED -- contrib/the page lists every exempt suite, and only those
M4 -- a path the page names renamed to something not on disk KILLED -- contrib/every suite the page names exists (plus the pytest-example check, since the same string is in that command)
M5 -- EXEMPT_TEST_SUITES gains an exemption the page never mentions KILLED -- contrib/the page lists every exempt suite, and only those
M6 -- the page's link target broken (scripts/test_suites.py -> scripts/test_suite.py) KILLED by this suite's existing md/no link to a file that does not exist
M7 -- the gate's list regex made unable to match KILLED -- contrib/the two shared lists parse: 0 script, 10 pytest, and the script-example check goes red with it

Two guards are there because the set-difference checks cannot report their own input: if either
SCRIPT_SUITES/PYTEST_SUITES regex ever stops matching, the differences are computed against an
empty set and the failures name the page rather than the parse. contrib/the two shared lists parse
says it directly -- M7 shows the pair it produces -- and the page's new
[scripts/test_suites.py](scripts/test_suites.py) link is covered by this suite's existing markdown
link check, which walks every *.md in the tree (M6).

Deconfliction: of the 64 open PRs, none touch CONTRIBUTING.md. tests/test_packaging.py is touched
by #1073 at one hunk, @@ -594,7 +594,8 @@ def option_type(opt); this diff ends at line 338, so the two
never meet. scripts/test_suites.py is deliberately not edited (#1095, #977, #970, #923, #922, #921,
#908, #729, #687 all add entries to it).

Diff scanned for credentials: 143 lines, 0 hits.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Fixes: no open issue. AGENTS.md's suite table names three of the seven exempt suites and is accurate
as far as it goes, so it is left alone rather than half-rewritten here.

The "Running the tests" section sent readers to `.github/workflows/ci.yml` for "the full set" --
ci.yml names 21 suite paths while `scripts/test_suites.py` holds 86, and the two checks beside this
one assert the workflows invoke that module instead of naming suites. It also transcribed three of
ten `PYTEST_SUITES` members as if that were the set, and named three of the seven exempt suites.
Rewrite the section around the two lists it now derives, and add the derivation: the pointer, the
existence of every path the page names, each bash example against the list its runner belongs to
(NandhaKishorM#374's trap is that `python tests/x.py` on a pytest suite asserts nothing and exits 0), and set
equality with `EXEMPT_TEST_SUITES` so an exemption cannot be added or dropped quietly.
@NandhaKishorM
NandhaKishorM merged commit c149cbf into NandhaKishorM:main Oct 11, 2026
36 of 38 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.3. Pointing contributors at .github/workflows/ci.yml for the suite list sends them to a file that names 21 test paths when the real list is 86, so the first thing a new contributor learns about the test setup is wrong by a factor of four. Pointing at scripts/test_suites.py instead is correct and, more usefully, is the file they then have to edit when they add a suite.

Transcribing three of ten pytest suites and three of seven hardware and weights suites is the drift that deriving the section from the two lists prevents. 138 checks to 145 in tests/test_packaging.py, in the file whose neighbouring check already fails if either workflow stops invoking that module, so the two halves of the claim are now pinned in the same place.

Leaving scripts/test_suites.py untouched on purpose, because ten open PRs edit it, is the right call and it paid off: #1095 added tests/test_doc_citations.py to SCRIPT_SUITES in this same release and the two merged without touching each other. The merged tree reports 87 suites, and your derived section counts them rather than naming them, so it was correct the moment #1095 landed.

Verified on the merged tree: all 87 suites pass and tests/test_packaging.py passes at its new count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants