Fix Playwright PR commenter false-positives when tests are skipped #5341
+114
−45
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR fixes the issue where the PR comment bot incorrectly reports "All tests passed" when Playwright tests are skipped or don't run. Changes: 1. Modified test-ui.yaml to capture and upload test status as deployment-info artifacts 2. Modified pr-playwright-deploy.yaml to properly handle missing test results The fix ensures that: - Test exit codes are properly captured and passed between workflows - When tests are skipped, the comment clearly states "Tests were skipped or not run" - Each browser test result is tracked individually with proper status reporting Fixes #5338 🤖 Generated with Claude Code Co-Authored-By: Claude <[email protected]>
🎭 Playwright Test Results✅ All tests passed across all browsers! ⏰ Completed at: 09/07/2025, 01:59:01 AM UTC 📊 Test Reports by Browser
🎉 Your tests are passing across all browsers! |
dosubot
bot
added
the
size:L
Member
Author
|
Just reviewed and |
snomiao
commented
Sep 5, 2025
The sharded chromium tests were generating deployment info only from shard 1, which could miss failures from other shards. This change moves the deployment info generation to the merge-reports job where we have visibility into the overall result of all shards. Fixes #5338
Member
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes the issue where the PR comment bot incorrectly reports "All tests passed" when Playwright tests are skipped or don't run.
Fixes #5338
Problem
The PR comment workflow (
pr-playwright-deploy.yaml) was checking for test failures by looking fordeployment-info-*artifacts. However:test-ui.yamlworkflow wasn't creating these artifactsALL_PASSED=trueSolution
1. Modified
test-ui.yamlsteps.playwright.outcomedeployment-info-*artifacts for each browser with format:browser|exit_code|url2. Modified
pr-playwright-deploy.yamlTest Plan
🤖 Generated with Claude Code
┆Issue is synchronized with this Notion page by Unito