Automated Testing: Re-enable storybook smoke test

[t-hamano]

What?

This PR re-enables the storybook smoke test and makes some enhancements to prevent CI failures in the feature.

Why?

To prevent Storybook stories from breaking, #69679 added a smoke test. However, it was subsequently skipped by #69943 due to CI failures caused by a new release of Playwright.

We need to re-enable this CI to ensure that storybook stories don't break.

How?

I used the --no-install flag to avoid any impact from future Playwright upgrades. This should install the local installation of Playwright. This change should not be necessary as #70503 has updated the local playwright to the latest version, but it is a precaution for the future.

Additionally, I have pinned the test runner version to 0.22.1, as the latest version of the test runner causes CI to fail with the following error:

[TEST] Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './internal/csf' is not defined by "exports" in /home/runner/work/gutenberg/gutenberg/node_modules/storybook/package.json

Example: https://github.com/WordPress/gutenberg/actions/runs/16819117756/job/47642444066

My guess is that the following change made in the test runner version 0.23.0 is the cause:

storybookjs/test-runner#556

In the future, if we update Storybook to the latest version, we will be able to remove this pinned version.

Testing Instructions

Check the CI and make sure it's properly smoke tested.

https://github.com/WordPress/gutenberg/actions/runs/16819975585/job/47644834408?pr=71126

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: t-hamano <[email protected]>
Co-authored-by: tyxla <[email protected]>
Co-authored-by: aduth <[email protected]>
Co-authored-by: im3dabasia <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[github-actions]

Flaky tests detected in f609ead.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/16929683265
📝 Reported issues:

• [Flaky Test] Redirection after template creation #60678 in /test/e2e/specs/site-editor/site-editor-url-navigation.spec.js

[tyxla]

Thanks @t-hamano! 🚀

As a follow-up, it would be nice to try and see what we can do to make the check faster. Right now it appears to take too long time for a smoke test.

[im3dabasia]

As a follow-up, it would be nice to try and see what we can do to make the check faster. Right now it appears to take too long time for a smoke test.

Thanks for the feedback! Before smoke tests, Storybook checks took about 7 minutes See, now it’s around 9.5 minutes, just a reasonable increase for the added value.

[t-hamano]

Thanks everyone for the reviews!

@im3dabasia If there's a way to make the check faster, let's address that in a follow-up.

[t-hamano]

Somehow, the CI started to fail again with the following error:

Details

Run npx concurrently -k -s first -n "SB,TEST" -c "magenta,blue" \
  npx concurrently -k -s first -n "SB,TEST" -c "magenta,blue" \
  "npx http-server ./storybook/build --port 50240 --silent" \
  "npx wait-on tcp:127.0.0.1:50240 && \
  NODE_PATH=./node_modules \
  npx --package=@storybook/[email protected] -- \
  test-storybook --url http://localhost:50240 --config-dir ./storybook"
  shell: /usr/bin/bash -e {0}
[SB] npm warn exec The following package was not found and will be installed: [email protected]
[TEST] npm warn exec The following package was not found and will be installed: @storybook/[email protected]
[TEST] npm warn deprecated [email protected]: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
[TEST] npm warn deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
[TEST] npm warn deprecated [email protected]: Glob versions prior to v9 are no longer supported
[SB] (node:6758) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
[SB] (Use `node --trace-deprecation ...` to show where the warning was created)
[TEST] Test Suites: 0 of 157 total
[TEST] Tests:       0 total
[TEST] Snapshots:   0 total
[TEST] Time:        0.216 s
[TEST] Ran all test suites.
[TEST] Error: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium_headless_shell-1187/chrome-linux/headless_shell
[TEST] ╔═════════════════════════════════════════════════════════════════════════╗
[TEST] ║ Looks like Playwright Test or Playwright was just installed or updated. ║
[TEST] ║ Please run the following command to download new browsers:              ║
[TEST] ║                                                                         ║
[TEST] ║     npx playwright install                                              ║
[TEST] ║                                                                         ║
[TEST] ║ <3 Playwright Team                                                      ║
[TEST] ╚═════════════════════════════════════════════════════════════════════════╝ Failed to launch browser.
[TEST]     at executablePathOrDie (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/registry/index.js:517:15)
[TEST]     at Object.executablePathOrDie (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/registry/index.js:548:45)
[TEST]     at Chromium._prepareToLaunch (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/browserType.js:184:39)
[TEST]     at async Chromium._launchProcess (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/browserType.js:196:22)
[TEST]     at async Chromium._innerLaunch (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/browserType.js:112:70)
[TEST]     at async Chromium._innerLaunchWithRetries (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/browserType.js:99:14)
[TEST]     at async /home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/browserServerImpl.js:77:18
[TEST]     at async ProgressController.run (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/server/progress.js:78:22)
[TEST]     at async BrowserServerLauncherImpl.launchServer (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/browserServerImpl.js:68:17)
[TEST]     at async BrowserType.launchServer (/home/runner/.npm/_npx/8cbbe221a9f534f3/node_modules/playwright-core/lib/client/browserType.js:73:12)
[TEST] npx wait-on tcp:127.0.0.1:50240 && NODE_PATH=./node_modules npx --package=@storybook/[email protected] -- test-storybook --url http://localhost:50240 --config-dir ./storybook exited with code 1
--> Sending SIGTERM to other processes..
[SB] npx http-server ./storybook/build --port 50240 --silent exited with code null
Error: Process completed with exit code 1.

CI example: https://github.com/WordPress/gutenberg/actions/runs/17112981527/job/48538253515
Actions: https://github.com/WordPress/gutenberg/actions/workflows/storybook-check.yml

The CI failures started around the same time as the new Playwright v1.55.0 was released, so I'm sure that the Playwright update is affecting us.

I believe we pinned all libraries to local versions with this PR, but maybe I missed something. Updating the Playwright version may solve the problem, but it won't be a fundamental solution.

Any ideas would be appreciated 🙏

[t-hamano]

Testing what happens if I explicitly install Chromium: #71284

Update: Installing the test runner first seems to resolve the problem.