Tweak a11y e2e tests #3507
Tweak a11y e2e tests #3507
Conversation
|
✅ Deploy Preview for astro-starlight ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
A leftover from when we removed Pa11y CI.
| # Uninstall man-db to prevent man page updates taking a long time after package installations | ||
| # on Ubuntu 24.04. | ||
| # https://github.com/actions/runner/issues/4030 | ||
| # https://github.com/actions/runner-images/issues/10977 | ||
| - name: Uninstall man-db | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get remove man-db | ||
There was a problem hiding this comment.
This is something I discovered and investigated recently, but when Playwright installs browsers, on Ubuntu 24.04, it can randomly take a very long time to update the manual page index caches.
For example, in this run on this PR before this change, it took more than 1m30s (you need to scroll to the top, click on the settings icon, select Show timestamps and then go back to the Processing triggers for man-db line).
Considering we definitely don't need man pages, we uninstall the command ahead of the package installations.
If we agree to keep this workaround until this is fixed (I personally think we should), I'll backport it to our Linux e2e tests too that also installs some packages and can suffer from the same issue.
There was a problem hiding this comment.
I also spotted this just now: https://stevefenton.co.uk/blog/2025/09/playwright-insteall-github-actions/#fewer-dependencies
Seems a bit fragile but could be another optimization to test.
There was a problem hiding this comment.
Interesting, definitely fragile (I would guess mostly around majors), but interesting nonetheless. With the current setup of this PR, dependencies takes 5 seconds, so I'm tempted to say it may not be worth it right now but something to keep in mind if it ever becomes more of a bottleneck?
There was a problem hiding this comment.
Ah yeah, could be that your fix has the same end result as theirs but with a much nicer (and I’d guess more stable) approach.
* main: (47 commits) feat: add some of trueberryless' new Starlight blogs to community content (withastro#3525) i18n(fr): update `resources/themes` (withastro#3526) Fix file filters added in withastro#3520 (withastro#3522) i18n(fr): update `guides/site-search.mdx` (withastro#3524) i18n(fr): update `resources/plugins.mdx` (withastro#3523) docs: add starlight-themes link (withastro#3510) i18n(de): update translation resources/plugins.mdx (withastro#3519) i18n(de): update translation guides/site-search.mdx (withastro#3518) Skip a11y CI checks for docs changes that aren’t tested anyway (withastro#3520) Speed up Linux e2e tests (withastro#3517) i18n(ru): update translations (withastro#3512) Tweak a11y e2e tests (withastro#3507) [ci] format docs(plugin): starlight-docsearch-typesense (withastro#3504) Add starlight-page-actions (withastro#3515) ci: update file icons (withastro#3511) [i18nIgnore] Match title case convention for components/using-components (withastro#3508) Disable Prettier on GitHub Actions workflow files (withastro#3509) [ci] release (withastro#3490) Update release workflow for OIDC (withastro#3500) ...
2 participants
Description
As we discussed, this PR tweaks the a11y e2e tests. At the moment, we process the sitemap after building the docs site, filter out some slugs, and run the a11y tests against the remaining URLs for 2 locales (English and Japanese).
We don't get a lot of value from testing all the same pages in both English and Japanese, even more so many of them. This PR refactors the test utils to be more i18n-aware:
Here are a comparison of the test runs before and after this change:
List of pages tested before the change
//components/asides//components/badges//components/card-grids//components/cards//components/code//components/file-tree//components/icons//components/link-buttons//components/link-cards//components/steps//components/tabs//environmental-impact//guides/authoring-content//guides/css-and-tailwind//guides/route-data//guides/sidebar//ja//ja/components/asides//ja/components/badges//ja/components/card-grids//ja/components/cards//ja/components/code//ja/components/file-tree//ja/components/icons//ja/components/link-buttons//ja/components/link-cards//ja/components/steps//ja/components/tabs//ja/environmental-impact//ja/guides/authoring-content//ja/guides/css-and-tailwind//ja/guides/route-data//ja/guides/sidebar//ja/reference/configuration//ja/reference/icons//ja/reference/route-data//ja/resources/community-content//ja/resources/plugins//ja/resources/showcase//ja/resources/themes//reference/configuration//reference/icons//reference/route-data//resources/community-content//resources/plugins//resources/showcase//resources/themes/List of pages tested after the change
//components/asides//components/badges//components/card-grids//components/cards//components/code//components/file-tree//components/icons//components/link-buttons//components/link-cards//components/steps//components/tabs//environmental-impact//guides/authoring-content//guides/css-and-tailwind//guides/sidebar//ja/guides/route-data//ja/reference/frontmatter//reference/configuration//reference/icons//resources/community-content//resources/plugins//resources/showcase//resources/themes/We can obviously tweak the included/excluded slugs further if we want as we should have more control now.