E2E tests: run tests on updates in mirror repos

[adimoldovan]

Changes proposed in this Pull Request:

Trigger e2e tests on changes in mirror repos.
This PR only adds the trigger workflow in the mirror repos. The actual execution of relevant tests will be added with a follow-up PR.
In the mirror repos, on push event a repository_dispatch call will be sent back to the monorepo. Based on the event_type of the dispatch the e2e tests workflow in the monorepo will execute the relevant tests.

Other information:

• Have you written new tests for your changes, if applicable?
• Have you checked the E2E test CI results, and verified that your changes do not break them?

Jetpack product discussion

pd5faL-i5-p2

Does this pull request change what data or activity we track or use?

no

Testing instructions:

• Cannot be fully tests until merged.
• After merge, check that e2e-tests workflow automattic/jetpack-production ran successfully.

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ All commits were linted before commit.
  
• ✅ Add a "[Status]" label (In Progress, Needs Team Review, ...).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.

---

Once your PR is ready for review, check one last time that all required checks (other than "Required review") appearing at the bottom of this PR are passing or skipped.
Then, add the "[Status] Needs Team review" label and ask someone from your team review the code.
Once you’ve done so, switch to the "[Status] Needs Review" label; someone from Jetpack Crew will then review this PR and merge it to be included in the next Jetpack release.

[anomiex]

This seems a bit convoluted. Why are you having the mirror repos do a repository dispatch back to the monorepo to trigger an action in response to the monorepo pushing to them? Keeping in mind that fairly often one merge in the monorepo will trigger several mirror repo pushes, which would then trigger several repository dispatches.

[adimoldovan]

This seems a bit convoluted. Why are you having the mirror repos do a repository dispatch back to the monorepo to trigger an action in response to the monorepo pushing to them? Keeping in mind that fairly often one merge in the monorepo will trigger several mirror repo pushes, which would then trigger several repository dispatches.

I agree it looks convoluted, but there are events in the mirror repos like a tag being created that we want to trigger tests.

[anomiex]

"Tag being created" most of the time is still going to be because of a push from the monorepo, with the autotagger workflow in between. The only exceptions I can think of are (1) something breaks in the autotagger workflow so someone has to tag it manually and (2) hard-way package releases.

I can't think of any other events that would trigger your added workflow. If tags are the only thing you could probably limit the added workflow to only running on tag events, like the autorelease workflow does. That would still mean dozens of dispatches in a flood when the Jetpack weekly releases all the packages and js-packages Jetpack depends on though.

[adimoldovan]

The dispatch is conditioned:
if: github.repository == 'automattic/jetpack-production'.
This will be extended to include all mirror repos of standalones that have e2e tests. I don't see a risk of flooding here, unless I'm missing something.
Another approach I initially though of was to only copy this workflow in repos that should trigger e2e test, but I couldn't see a fast and straightforward way to update the copy behaviour. Do you think that would be better?
What I like about the current approach is that you can easily see in the workflow what repos are sending the dispatch.

The idea here is to test closer to what gets released, and the code from mirros is released, from tags in the mirrors. There are other options, sure, but convoluted as this option seems, I think it might be the cleanest.
I'm even thinking this can replace the "push" one in the monorepo. Then we'll only have PRs tested with code from monorepo, using the dev build, and all other updates will be tested using the built code from mirrors, triggered from the release source, the mirror repo.

[brbrr]

I'm fine with this approach, and we can iterate on it later on.

[anomiex]

This will be extended to include all mirror repos of standalones that have e2e tests. I don't see a risk of flooding here, unless I'm missing something.

No, you're right on that part, I missed that there was the check.

Another approach I initially though of was to only copy this workflow in repos that should trigger e2e test, but I couldn't see a fast and straightforward way to update the copy behaviour. Do you think that would be better?

I do think that would be cleaner, as then the mirrors that don't need it wouldn't even contain the workflow and wouldn't have do-nothing runs.

The way I'd do it would be to make a separate .github/files/gh-e2e-something/ folder and add logic near

jetpack/tools/cli/commands/build.js

Lines 681 to 699 in 51ca350

	// Copy standard .github.
	await copyDirectory( '.github/files/mirror-.github', npath.join( buildDir, '.github' ) );
	// Copy autotagger, autorelease, wp-svn-autopublish, and/or npmjs-autopublisher if enabled.
	if ( composerJson.extra?.autotagger ) {
	await copyDirectory( '.github/files/gh-autotagger', npath.join( buildDir, '.github' ) );
	}
	if ( composerJson.extra?.autorelease ) {
	await copyDirectory( '.github/files/gh-autorelease', npath.join( buildDir, '.github' ) );
	}
	if ( composerJson.extra?.[ 'wp-svn-autopublish' ] ) {
	await copyDirectory( '.github/files/gh-wp-svn-autopublish', npath.join( buildDir, '.github' ) );
	}
	if ( composerJson.extra?.[ 'npmjs-autopublish' ] ) {
	await copyDirectory(
	'.github/files/gh-npmjs-autopublisher',
	npath.join( buildDir, '.github' )
	);
	}

to copy it when needed. It needn't be done with a composer.json check, you could look for the presence of E2E tests in the filesystem instead if that would make sense.

The idea here is to test closer to what gets released, and the code from mirros is released, from tags in the mirrors. There are other options, sure, but convoluted as this option seems, I think it might be the cleanest.

I still think it would be cleaner to trigger it directly in the monorepo, from the end of the Build job or from the Post-Build job I really need to re-introduce one of these days (see #25892, reverted in #25894). You could still fetch from the mirror repo if you really want to, just saving the back-and-forth dispatches.