feat: add test count display to Playwright PR comments #5458
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add extract-playwright-counts.mjs script to parse test results from Playwright reports - Update pr-playwright-deploy-and-comment.sh to extract and display test counts - Show overall summary with passed/failed/flaky/skipped counts - Display per-browser test counts inline with report links - Use dynamic status icons based on test results (✅/❌/⚠️ ) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
🎭 Playwright Test Results✅ All tests passed! ⏰ Completed at: 09/13/2025, 04:28:45 AM UTC 📈 Summary
📊 Test Reports by Browser
🎉 Click on the links above to view detailed test results for each browser configuration. |
- Add skipped test extraction for individual browser reports - Update per-browser display format to show all four counts: (✅ passed / ❌ failed /⚠️ flaky / ⏭️ skipped) - Provides complete test result visibility at a glance 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Use absolute paths for script and report directories - Add debug logging to help diagnose extraction issues - Move counts display after View Report link as requested - Format: [View Report](url) • ✅ passed / ❌ failed /⚠️ flaky / ⏭️ skipped 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add JSON reporter to Playwright test runs - Generate report.json alongside HTML reports - Store JSON report in playwright-report directory - This enables accurate test count extraction from CI artifacts The HTML reports alone don't contain easily extractable test statistics as they use a React app with dynamically loaded data. JSON reports provide direct access to test counts. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Use proper syntax for JSON reporter with outputFile option - Run separate commands for HTML and JSON report merging - Specify output path directly in reporter configuration - Ensures report.json is created in playwright-report directory This fixes the "No such file or directory" error when trying to move report.json file, as it wasn't being created in the first place. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
This reverts commit 605d7cc.
- Use --reporter=html,json syntax (comma-separated, not space) - Move test-results.json to playwright-report/report.json after generation - Remove incorrect PLAYWRIGHT_JSON_OUTPUT_NAME env variable - Add || true to prevent failure if JSON file doesn't exist The JSON reporter outputs to test-results.json by default when using the comma-separated reporter list syntax. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Use separate --reporter flags for list, html, and json - Set PLAYWRIGHT_JSON_OUTPUT_NAME env var to specify JSON output path - Run HTML and JSON report generation separately for merged reports - Ensures report.json is created in playwright-report directory The combined reporter syntax wasn't creating the JSON file properly. Using separate reporter flags with env var ensures JSON is generated. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
dosubot
bot
added
the
size:L
DrJKL
requested changes
Sep 10, 2025
Contributor
DrJKL
left a comment
DrJKL
left a comment
There was a problem hiding this comment.
I like the emoji and more granular feedback.
I'm a little wary of this much structural repetition, feels like it's asking for typos.
Overall, 👍🏻, but could you do it in Typescript for uniformity?
Co-authored-by: Alexander Brown <[email protected]>
- Convert extract-playwright-counts.mjs to TypeScript (.ts) - Add proper TypeScript types for better type safety - Use tsx for execution instead of node - Auto-install tsx in CI if not available - Better alignment with the TypeScript codebase This provides better type safety and consistency with the rest of the codebase while maintaining the same functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…ck to the beginning of the script for better organization and efficiency
🔧 Auto-fixes AppliedThis PR has been automatically updated to fix linting and formatting issues.
Changes made:
|
Myestery
pushed a commit
that referenced
this pull request
Sep 17, 2025
* feat: add test count display to Playwright PR comments - Add extract-playwright-counts.mjs script to parse test results from Playwright reports - Update pr-playwright-deploy-and-comment.sh to extract and display test counts - Show overall summary with passed/failed/flaky/skipped counts - Display per-browser test counts inline with report links - Use dynamic status icons based on test results (✅/❌/⚠️ ) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * feat: include skipped test count in per-browser display - Add skipped test extraction for individual browser reports - Update per-browser display format to show all four counts: (✅ passed / ❌ failed /⚠️ flaky / ⏭️ skipped) - Provides complete test result visibility at a glance 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * fix: improve test count extraction reliability in CI - Use absolute paths for script and report directories - Add debug logging to help diagnose extraction issues - Move counts display after View Report link as requested - Format: [View Report](url) • ✅ passed / ❌ failed /⚠️ flaky / ⏭️ skipped 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * fix: generate JSON reports alongside HTML for test count extraction - Add JSON reporter to Playwright test runs - Generate report.json alongside HTML reports - Store JSON report in playwright-report directory - This enables accurate test count extraction from CI artifacts The HTML reports alone don't contain easily extractable test statistics as they use a React app with dynamically loaded data. JSON reports provide direct access to test counts. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * fix: correct JSON reporter syntax for Playwright tests - Use proper syntax for JSON reporter with outputFile option - Run separate commands for HTML and JSON report merging - Specify output path directly in reporter configuration - Ensures report.json is created in playwright-report directory This fixes the "No such file or directory" error when trying to move report.json file, as it wasn't being created in the first place. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Revert "fix: correct JSON reporter syntax for Playwright tests" This reverts commit 605d7cc. * fix: use correct Playwright reporter syntax with comma-separated list - Use --reporter=html,json syntax (comma-separated, not space) - Move test-results.json to playwright-report/report.json after generation - Remove incorrect PLAYWRIGHT_JSON_OUTPUT_NAME env variable - Add || true to prevent failure if JSON file doesn't exist The JSON reporter outputs to test-results.json by default when using the comma-separated reporter list syntax. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * fix: improve test count extraction reliability in CI - Use separate --reporter flags for list, html, and json - Set PLAYWRIGHT_JSON_OUTPUT_NAME env var to specify JSON output path - Run HTML and JSON report generation separately for merged reports - Ensures report.json is created in playwright-report directory The combined reporter syntax wasn't creating the JSON file properly. Using separate reporter flags with env var ensures JSON is generated. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Update scripts/cicd/pr-playwright-deploy-and-comment.sh Co-authored-by: Alexander Brown <[email protected]> * refactor: convert extraction script to TypeScript and use tsx - Convert extract-playwright-counts.mjs to TypeScript (.ts) - Add proper TypeScript types for better type safety - Use tsx for execution instead of node - Auto-install tsx in CI if not available - Better alignment with the TypeScript codebase This provides better type safety and consistency with the rest of the codebase while maintaining the same functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * chore(pr-playwright-deploy-and-comment.sh): move tsx installation check to the beginning of the script for better organization and efficiency * [auto-fix] Apply ESLint and Prettier fixes --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Alexander Brown <[email protected]> Co-authored-by: GitHub Action <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Changes
extract-playwright-counts.mjsscript to parse test statistics from Playwright HTML reportspr-playwright-deploy-and-comment.shto extract and display test counts in PR commentsTest plan
🤖 Generated with Claude Code
┆Issue is synchronized with this Notion page by Unito