Scripts: Add test-e2e script to wp-scripts

[gziolo]

Description

Part of #12313.

I mostly moved code between files and extracted new utils hasBabelConfig and hasJestConfig to reuse in the newly introduced script.

• Added wp-scripts test-e2e script to @wordpress/scripts
• Added documentation
• Added changlog entry
• Bumped versions for puppeteer, jest-puppeteer and lint-staged

Follow-up task

• Improve the default config: https://github.com/WordPress/gutenberg/blob/0761d2e311ec06b8b1e9f27d630dee83641a13d0/packages/scripts/config/jest-e2e.config.js - it requires moving utils to their own package

How has this been tested?

Both:

• npm run test-e2e
• npm run test-unit

should still work the same.

To test zero-config stuff, it's possible to run:

• ./node_modules/.bin/wp-scripts test-e2e --showConfig
• ./node_modules/.bin/wp-scripts test-unit-js --showConfig

Types of changes

Checklist:

• My code is tested.
• My code follows the WordPress code style.
• My code has proper inline documentation.

[gziolo]

I had to bump it to ensure it uses the latest version of jest-validate. Otherwise, it shows some warnings for the configuration which is invalid.

[gziolo]

When working on docs (132a1e5) I realized how much config options we offer which are not documented. I can't add it to the docs in the current shape. I hope it's going to be possible as soon as I start working on the follow-up tasks. I think that the next step should be to put all tests into their own package and starting from there we should be able to extract another package with tests helper and utils to reuse with test-e2e script.

[oandregal]

This is missing the fromScriptsRoot export.

[gziolo]

Good catch, I think only spawnScript needs it at the moment so we might keep it internal. I will double check.

[gziolo]

It's ony used by spawnScript so we can leave it unexposed. Unless you think we should keep it exported?

[oandregal]

I don't feel strongly either way, I'd perhaps expose it for coherency, but I'm fine the other way round. Just wanted to bring our attention in case it was important.

[gziolo]

Can expose, it's fine 👍

[gziolo]

I'm seeing the following response for Vimeo embed in demo test:

Object {
   \"code\": \"rest_cookie_invalid_nonce\",
   \"data\": Object {
     \"status\": 403,
   },
   \"message\": \"Cookie nonce is invalid\",
}

I have no idea where it comes from to be honest.

[oandregal]

@gziolo This is great work, bravo!

I followed these testing instructions that use a test plugin I've created. Calling wp-scripts test-unit-js and wp-scripts test-e2e with and without configuration works. The no config option is particularly helpful! ❤️

I've noticed two gotchas, though:

• I needed to install @babel/core in my plugin package although I don't use it. It looks like some wp-scripts dependencies require @babel/core to be installed as they have it as a peer dep. I don't consider this a blocker for this PR, but I wonder whether there is something we can do to avoid this?
• I've noticed that without configuration the tests found by the unit and e2e commands are different and I'd love homogenizing both test matchers (I'm less opinionated about which matcher to use, although the one used by the unit tests is more comprehensive):
  • e2e only finds test files that contain test in their name using the plugin dir as the root.
  • unit find those and the ones stored in any test folder using the plugin dir as the root.

[gziolo]

• I needed to install @babel/core in my plugin package although I don't use it. It looks like some wp-scripts dependencies require @babel/core to be installed as they have it as a peer dep. I don't consider this a blocker for this PR, but I wonder whether there is something we can do to avoid this?

It sounds like something which needs to be fixed before we merge this PR. We have it in the root package.json. I need to double check why it had to be configured this way in the first place. I don't see any reasons why we couldn't put it inside @wordpress/scripts @wordpress/babel-preset-default also.

[gziolo]

• I've noticed that without configuration the tests found by the unit and e2e commands are different and I'd love homogenizing both test matchers (I'm less opinionated about which matcher to use, although the one used by the unit tests is more comprehensive)

Yes, we can fix that now. I will just copy matcher for unit tests which extends default with what we effectively use in Gutenberg.

[gziolo]

@nosolosw - f2482e6 includes updates based on your comments 👍

[oandregal]

It works now on a plugin, with and without config. Bravo!

[youknowriad]

This is risky I think :P. I understand the reasoning but the way npm works (we often do rm -rf locally) can create issues.

[gziolo]

I will remove, just wanted to speed up the things, see the number of commits added 😅

[ntwb]

Core still caches node_modules, might be worth considering pulling it from there also https://github.com/WordPress/wordpress-develop/blob/master/.travis.yml#L7

[gziolo]

Oh interesting, Riad might be right that it can cause troubles for PRs where the lock file gets frequently updated

[aduth]

For end-to-end tests, should this be checking for jest.config.js (or package.json jest)? Or only jest-puppeteer.config.js? How should we expect to give people control over whether to override configuration for either unit tests or end-to-end tests?

I'm running into this in my Dones project, where I do choose to override the default configuration for unit tests, but unfortunately it seems to bleed over into end-to-end test scripts as well, where I would prefer to simply use the default. Or at least, it's not clear how I can configure for both unit tests and end-to-end tests.

It kinda feels like this should be limited to checking for the presence of jest-e2e.config.js perhaps?

[gziolo]

It kinda feels like this should be limited to checking for the presence of jest-e2e.config.js perhaps?

An interesting challenge and I'm surprised it didn't come up earlier :)

How about we add higher priority for non-standard:

• jest-e2e.config.js
• jest-unit.config.js

To give folks a way to target only one or both of those tools without the need for using --config flag.

As a temporary workaround, you can use the non-standard name of the file and pass it as a config option to the test script. In fact, this is how it works in Gutenberg:

gutenberg/package.json

Line 227 in 6c80cc3

"test-e2e": "wp-scripts test-e2e --config packages/e2e-tests/jest.config.js",

gutenberg/package.json

Line 232 in 6c80cc3

"test-unit": "wp-scripts test-unit-js --config test/unit/jest.config.js",

[aduth]

In retrospect, it seems a bit more obvious in retrospect in in observing that the default configurations folder includes variations of these two files (jest-e2e.config.js, jest-unit.config.js), yet only one is supported for end-users. It's an interesting exercise to consider for other potential issues. Is there a similar problem with .eslintrc-md.js and .eslintrc.js ?

How about we add higher priority for non-standard:

I think this could be a good solution. I was worried if it might have to be a "Breaking Change" if we only start considering one of these two, but (a) IIRC jest.config.js is a "standard" file supported by Jest and (b) it could be viewed as a rare, advanced use-case where one opts to configure both.

As a temporary workaround, you can use the non-standard name of the file and pass it as a config option to the test script. In fact, this is how it works in Gutenberg:

Yeah, I ended up using something like this, or at least observing that higher priority is given to --config, also observing that the E2E configuration is quite small and I wasn't entirely on board with the specs forced folder convention of the default configuration 😅

wp-scripts test-e2e --config {\"preset\":\"jest-puppeteer\"} e2e

[aduth]

Pull request: #22477