Tests: Move e2e tests to the npm package

[gziolo]

Description

Part of #12313.

This PR moves all e2e test files to the new npm package called @wordpress/tests-e2e. This is the first step which should allow us to use those tests to verify WordPress core. Next step will require to extract expose all tests plugins and utils to their own package and make sure that they can be run from the node_modules folder. Jest by default ignore all files located in the folder which contains installed npm packages.

How has this been tested?

npm run test-e2e

[talldan]

this path seems like it'd be problematic when test-e2e is installed as a node module. Is this something you're planning on tackling in a later PR?

[gziolo]

Yes, good point. The same issue applies to the root directory. I think it's easier to revisit once both #12437 and #12465 are merged. I feel like having a follow-up PR with changes which try to make it reusable outside is going to be easier. For the time being, I feel like we only need to make this package private to ensure it isn't exposed to npm as it clearly isn't fully ready. However, I want to move forward so don't have to deal with the folder structure and rebases forever.

[gziolo]

This file was removed so it's sort of a non-issue at the moment. However, I want to explore whether this is something we want to include in the default e2e config. We would need to expose e2e utils first, to be able to reference those two though.

[oandregal]

If the goal is to let others (core, plugin/theme authors, etc) reuse and execute these tests, they will also need a setup. Gutenberg setup is being extracted at #12437 as a different package, and I wonder: what's the benefit of having two different packages if the two are required (and dependant)? Do we need to run these e2e tests with a different setup? Would a single package for tests and setup be enough?

[gziolo]

what's the benefit of having two different packages if the two are required (and dependant)? Do we need to run these e2e tests with a different setup? Would a single package for tests and setup be enough?

wp-scripts test-e2e is a general purpose script which is meant to run any Jest + Puppeteer tests. It's targeting plugin developers who want to add tests for their code. That's why we have to offer our test utils so they don't need to copy them. You could argue whether they should be in their own package or can be bundled with core tests. I assumed it's nice to have utils as a separate package. I don't feel strongly about that though.

I also anticipate that there might be 3 different config related setups required to make it all work as intended. We might need 2 or 3 files:

1. Gutenberg configuration to run all e2e tests.
2. Config to run all e2e tests directly from the npm package.
3. A default config used by plugins if they don't provide anything.
   Hopefully, we can consolidate 1 & 2 into one thing.

[talldan]

@gziolo Instead of running tests directly from node_modules and needing extra configuration to make that happen, I wondered whether an option could be to instead import the tests from the package into a file that's not in node modules. e.g in our tests/e2e/index.js file we'd just have import '@wordpress/tests-e2e';.

The index.js in the tests-e2e package would need to import each of the test files for it to work.

[talldan]

One other thought on this, I was wondering if tests-e2e might be too much of a generic name. I'm not sure if we'd consider having more than just gutenberg/block-editor tests in this package.

[youknowriad]

Agree that we should have more granular control on what tests can be run. I expect this to grow past the editor only. At some point we'd be able to run phase2 tests, core editor tests, ... separately. It could be just . import { editorTests } from "@wordpress/tests-e2e".

I don't have a strong opinion or a perfect solution at the moment though, this is probably something we'd have to iterate on.

[gziolo]

Jest support projects, so we can split it into smaller groups using this feature:

https://jestjs.io/docs/en/configuration#projects-array-string-projectconfig

[oandregal]

@gziolo I'm still on the fence regarding how to distribute utils and core tests (bundled with the wp-scripts package or separate utilities). I'm admittedly less exposed to other use cases (WordPress core, 3rd party WordPress setups, plugins, etc), so I'll let people with that kind of experience chime in and offer advice.

[youknowriad]

It's looking good for me as a start. We should test it in Core and see from there.
The only concern I have is about the name
Is this package going to include all WordPress e2e tests? If it's the case, the name is good
otherwise, we might need to be specific

[gziolo]

The only concern I have is about the name
Is this package going to include all WordPress e2e tests? If it's the case, the name is good
otherwise, we might need to be specific

I would prefer to keep all test suites in one package, but find a way to make it easy to run the subset of them using a specific folder structure or projects option built-in in Jest.