e2e tests: update setup to match updates in the Gutenberg repo

[jeherve]

Fixes #33626

Proposed changes:

Update our infrastructure following changes in Gutenberg. See #33626 (comment) for more info

Other information:

• Have you written new tests for your changes, if applicable?
• Have you checked the E2E test CI results, and verified that your changes do not break them?
• Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

• 8586-gh-jpop-issues

Does this pull request change what data or activity we track or use?

• No

Testing instructions:

• I will temporarily run the tests on this PR:
  https://github.com/Automattic/jetpack/actions/runs/6944694826/job/18892395483?pr=33644

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Team Review, ...).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.

---

Once your PR is ready for review, check one last time that all required checks appearing at the bottom of this PR are passing or skipped.
Then, add the "[Status] Needs Team Review" label and ask someone from your team review the code. Once reviewed, it can then be merged.
If you need an extra review from someone familiar with the codebase, you can update the labels from "[Status] Needs Team Review" to "[Status] Needs Review", and in that case Jetpack Approvers will do a final review of your PR.

[WunderBart]

@jeherve, I've noticed that this is now failing with the following error:

Error: ENOENT: no such file or directory, open 'results/base.0.test.results.json'

If the RESULTS_ID env variable is not specified, the curated results will be saved under <artifactsFolder>/<testSuite>.performance-results.json, which would be ./results/post-editor.performance-results.json in this case. The .performance-results.json suffix is fixed, so if we want a custom name, like base.0.performance-results.json, the RESULTS_ID should be set to base.0.

[jeherve]

@WunderBart Thanks a lot for the help, I really appreciate it! 🙇

Let's give it a try with RESULTS_ID 🤞

[WunderBart]

Looks like the tests are OK now, but we still need the following bit to account for the new results filename suffix (*.test.results.json → *.performance-results.json):

jetpack/tools/e2e-commons/bin/performance.js

Lines 54 to 68 in 7c37a31

	function mergeResults( type ) {
	const objs = [];
	for ( let i = 0; i < numAttempts; i++ ) {
	const file = `results/${ type }.${ i }.test.results.json`;
	objs.push( JSON.parse( readFileSync( file ) ) );
	}
	const out = _.mergeWith( {}, ...objs, ( objValue, srcValue ) => {
	if ( _.isArray( objValue ) ) {
	return objValue.concat( srcValue );
	}
	} );
	writeFileSync( `results/${ type }.test.results.json`, JSON.stringify( out ) );
	}

[WunderBart]

Also, the following part is no longer true, so we should be OK without the disable-animations.php plugin.

jetpack/.github/workflows/block-performance.yml

Line 55 in 7c37a31

# Gutenberg Performance tests require disable-animations.php test plugin to be available on a site. Below we inject the mapping for that plugin into a Docker config file

[WunderBart]

Running Gutenberg's performance tests in a different environment was an eye-opening challenge. It's also a great chance to suggest enhancements that could improve the flexibility of this setup. 💪

cc: @swissspidy as there's a related ongoing effort in making the perf tools shareable.