[PHP Tests] Increase parity with WordPress Core

[hellofromtonya]

This is an epic ticket to track ongoing roadmap to increase the PHP test suite parity with WordPress Core.

Updates:

• 17 Aug 2022: Initially created.

Scope

WordPress Core' PHP tests:

• Run against all supported PHP versions (currently 5.6 to 8.1)
• Use PHPUnit Polyfills to match the PHPUnit and PHP versions
• Use the WordPress Core test coding standards including:
  • the standards for file structures, file naming, and test class naming
  • preferred assertions such as assertSame() instead of assertEquals()
  • renaming of the shared fixtures
  • use of data providers
  • annotations including @covers for code coverage
• With PHP notices and deprecations turned on for awareness and testing

The scope of this roadmap is to upgrade the Gutenberg PHP unit tests for parity to WordPress Core.

Why? Benefits

• Reduce backporting effort by getting tests Core-ready long before backporting starts.
• To promote discovery of potential issues and incompatibilities earlier and continuously through the development and testing cycles.
• To increase commonality for Core contributors to more easily contribute to Gutenberg.

Ongoing Work

• PHPUnit: turns on PHP notices and deprecations #43102
• PHPUnit: Let PHPUnit Polyfills match PHPUnit version #43334
• Run PHPUnit jobs across multiple PHP versions #46510
• GitHub workflows: parity with Core #48603
• Reorganize and rename test files and classes to Core's test coding standards
• Audit shared fixtures and resolve any mismatches
• Audit tests and convert into data providers where appropriate
• Add failure messages when a test method has more than 1 assertion or expectation
• Enforce checks against redeclaration for functions and classes #52696
• Document the naming convention for block-library PHP functions #53777
• Enforce valid function names in the packages/block-library/src/*/*.php files #53438
• Refactor unit tests in the gutenberg/gutenberg-coding-standards package

[ockham]

Run test CI jobs across multiple PHP versions

I wonder if we can leverage reusable workflows to basically, well, re-use wordpress-develop's phpunit-tests.yml 🤔

cc/ @desrosj 👋

[desrosj]

I wonder if we can leverage reusable workflows to basically, well, re-use wordpress-develop's phpunit-tests.yml 🤔

I have been thinking about this for a while, but I'm not sure the best way to approach it. Callable workflows execute as a job, and as far as I know, you can't attach additional steps in the calling workflow. However, the callable workflow could potentially be dynamic and look for the presence of a JSON file, or something similar to provide test conditions.

For example, instead of manually defining the matrix of values in the phpunit-tests.yml file, it would look for a JSON file with the desired matrix instead. This should allow Gutenberg to have a workflow file that calls the one in Core@trunk while still defining its own desired test conditions.

Where this potentially breaks down is when multiple versions of WordPress are tested. As the workflow file within trunk is updated over time, it's likely it will break for older branches at some point.

[ockham]

I have been thinking about this for a while, but I'm not sure the best way to approach it. Callable workflows execute as a job, and as far as I know, you can't attach additional steps in the calling workflow.

If I'm reading the real-world example correctly, then it should be possible to run the re-usable workflow at job level 🤔

But even if that wasn't the case: I know GB currently has one workflow for unit tests that includes both JS and PHP ones, but I don't think that's a hard requirement -- we could create a separate workflow file for PHP unit tests if that's required to re-use wordpress-develop's PHPUnit test workflow...

However, the callable workflow could potentially be dynamic and look for the presence of a JSON file, or something similar to provide test conditions.

Yeah, TBH, I haven't worked with re-usable workflows myself yet, so I'm still not totally sure about how they relate to their respective repositories -- i.e. if a re-usable workflow could be configured through any files in the "consuming" repository... Anyway, it seems that re-usable workflows should allow for some degree of parametrization, so maybe we can leverage that.

For example, instead of manually defining the matrix of values in the phpunit-tests.yml file, it would look for a JSON file with the desired matrix instead. This should allow Gutenberg to have a workflow file that calls the one in Core@trunk while still defining its own desired test conditions.

TBH I wouldn't mind avoiding this degree of configurability -- at least for starters (since it introduces some extra complexity) 😅

Maybe I'll try filing a PR against wordpress-develop to add the workflow_call trigger.

If this isn't going to work, we might consider composite actions; or maybe just exporting a PHPUnit test action from wordpress-develop...

[jrfnl]

I fully support the efforts being made here.

The Gutenberg ports to WP Core have in recent merges introduced quite some new PHP 8.1 and 8.2 incompatibilities, which could have been avoided if the tests were run on all appropriate PHP versions.

These kind of new incompatibilities being introduced via new code is highly discouraging for the people trying to make WordPress Core compatible with more recent PHP versions.

Also see #44536, #44537 and #44538

While I understand that it would be a "nice-to-have" to use some form of re-usable workflow, getting the tests running on multiple PHP versions shouldn't need to wait on that and is actionable now.

If anyone is working on this and needs guidance, give me a shout. If nobody is working on this, let me know and I'll get this sorted ASAP.

[talldan]

Run test CI jobs across multiple PHP versions

I had a look at this in #44566. Have left some comments about the issues. I welcome commits to the PR if anyone has ideas for improving things.

One thing I'm unsure about is how many jobs we can add or if there's any kind of limit to our github actions. (I'm guessing there is)