E2E Tests: Migrate to github actions

[brbrr]

Move E2E tests from Travis into GH actions

With GH Actions, we could customize our runs, run different suites on different events (e.g. on new tags, merges to master, etc)

Jetpack product discussion

n/a

Does this pull request change what data or activity we track or use?

n/a

Testing instructions:

• Take a look at "Checks" section
• Make sure the tests have passed.

Proposed changelog entry for your changes:

• n/a

[jetpackbot]

Scheduled Jetpack release: November 10, 2020.
Scheduled code freeze: November 3, 2020

E2E results is available here (for debugging purposes): https://jetpack-e2e-dashboard.herokuapp.com/pr-17094

Thank you for the great PR description!

When this PR is ready for review, please apply the [Status] Needs Review label. If you are an a11n, please have someone from your team review the code if possible. The Jetpack team will also review this PR and merge it to be included in the next Jetpack release.

Generated by 🚫 dangerJS against ba31235

[jeherve]

The tests seem to pass.

I was a bit worried about potentially having secrets slip out in the builds, but couldn't find anything bad.

I only have 2 questions below.

[kraftbj]

I really dig the approach here. The action is erroring, though, ERROR: WP_SITE_URL is not set after launching an ngrok.

I think it'll give us a lot of future flexibility, so I'm good to roll through it once it passes.

[fullofcaffeine]

Hi @brbrr 👋🏻

We're currently working to better understand what would it take to have E2E tests in other repos use a specific/latest version of Gutenberg*. @ockham pointed me to the work you've done here, and also mentioned this PR as we also want to understand what (mostly E2E) tooling/infrastructure is available for each project, and Jetpack seems to have have the edge for now as a template/inspiration we could use for other projects.

We also would like to wait for this PR to be merged before adding any new tests (as part of this other related task).

That being said, how can we help you testing this PR? I see the testing instructions just mention checking if all tests pass in the list of GH jobs below, as some are failing. Is there anything we can do to help you?

Thanks! 🙇🏻

---

*See pc9Uba-8b-p2

[kraftbj]

This looks good to me. Added a minor commit to remove the allowed-failure in travis for the e2e-guten test.

[brbrr]

Hey @fullofcaffeine! Sorry for the late response.

The PR is merged, so nothing to test here :) There is this PR in the works (it is ready for review, but I still want to double-check it if works fine) #17523

It is infrastructure related, but with that PR merged, we would be able to run multiple E2E containers in parallel. I would appreciate a review :)