Skip to content

Switch performance tests to Playwright #52022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WunderBart merged 46 commits into from
Aug 16, 2023
Merged

Switch performance tests to Playwright #52022

Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
a88918c
Run with Playwright in CI
WunderBart Jun 28, 2023
4ce6571
Print stdout on error
WunderBart Jun 28, 2023
f6d9851
Run build:packages to use e2e utils
WunderBart Jun 28, 2023
6fde09d
Extend timeout for the canvas spinner
WunderBart Jun 28, 2023
6f28c60
Add missing front end perf report
WunderBart Jun 28, 2023
b9b0440
Remove obsolete step
WunderBart Jun 29, 2023
ea6891f
Make the typing testing consistent
WunderBart Jun 29, 2023
aeae2ec
Improve post editor perf testing consistency
WunderBart Jul 4, 2023
309b9a2
Do not report slow tests
WunderBart Jul 5, 2023
b47d2b7
Use the same methodology for measuring post and site editors metrics
WunderBart Jul 5, 2023
0427a37
Make the results table transformer more readable
WunderBart Jul 5, 2023
542456e
Streamline the results processing p.1
WunderBart Jul 5, 2023
c06ab19
Improve visual step separation a tad :)
WunderBart Jul 5, 2023
df1216e
First step to separate metrics from the runner
WunderBart Jul 6, 2023
2e13186
Define the results file suffix
WunderBart Jul 6, 2023
ed1e0cc
Merge remote-tracking branch 'origin' into refactor/playwright-perfor…
WunderBart Jul 6, 2023
fecc1a0
cosmetic
WunderBart Jul 6, 2023
4a394d8
Fix matching pattern typo
WunderBart Jul 6, 2023
647ea57
Make performance.js comparison runner results-agnostic
WunderBart Jul 10, 2023
1a18006
Fix results handling
WunderBart Jul 10, 2023
f03e671
Use Playwright's attachments API to handle results
WunderBart Jul 10, 2023
5db75c3
Upload raw results as well
WunderBart Jul 10, 2023
da43808
Be consistent with calc utils
WunderBart Jul 11, 2023
00f641d
Streamline results handling
WunderBart Jul 11, 2023
2b9a5ac
Use the same sampling convention for front-end metrics
WunderBart Jul 11, 2023
1f8ecd1
Fix the metrics output in perf runner
WunderBart Jul 11, 2023
d649aac
tmp: compare performance with base branch
WunderBart Jul 12, 2023
ae40542
tmp: compare performance with base branch (run 2)
WunderBart Jul 12, 2023
18fe132
tmp: compare performance with base branch (run 3)
WunderBart Jul 12, 2023
8f15066
Revert "tmp: compare performance with base branch"
WunderBart Jul 12, 2023
b1d7f83
Remove unused code and do the switch
WunderBart Jul 12, 2023
e37cbc2
Merge remote-tracking branch 'origin' into refactor/playwright-perfor…
WunderBart Jul 12, 2023
2c305da
Fix site editor typing spec
WunderBart Jul 26, 2023
c9c625e
Merge remote-tracking branch 'origin' into refactor/playwright-perfor…
WunderBart Jul 26, 2023
bbae249
Add inline comment
WunderBart Jul 27, 2023
894896b
Expain why we need larger timeout for site editor visit util
WunderBart Jul 27, 2023
7ac254e
Use dirent for getting files from dir
WunderBart Jul 27, 2023
84bb262
Do not use the browser.newPage shortcut
WunderBart Jul 27, 2023
182fbc1
Merge remote-tracking branch 'origin' into refactor/playwright-perfor…
WunderBart Jul 27, 2023
1cf298a
Use trace buffer instead of tmp file
WunderBart Jul 27, 2023
f787f1e
Merge remote-tracking branch 'origin' into refactor/playwright-perfor…
WunderBart Aug 14, 2023
5a68bbc
Fix 'unterminated string' error
WunderBart Aug 14, 2023
d0d81fb
Fix 'unterminated string' error p.2
WunderBart Aug 14, 2023
37ef946
Make sure sanitized branch name is used for reading results json
WunderBart Aug 16, 2023
127b21e
Merge remote-tracking branch 'origin' into refactor/playwright-perfor…
WunderBart Aug 16, 2023
568d9d7
Prevent logging in via test.use() instead of creating new context
WunderBart Aug 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Expain why we need larger timeout for site editor visit util
  • Loading branch information
@WunderBart
WunderBart committed Jul 27, 2023
commit 894896bb3ac81ac4ed0f43b762708e0b87b69a4d
3 changes: 3 additions & 0 deletions packages/e2e-test-utils-playwright/src/admin/visit-site-editor.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,5 +66,8 @@ export async function visitSiteEditor(
// TODO: Ideally the content underneath the spinner should be marked inert until it's ready.
await this.page
.locator( '.edit-site-canvas-spinner' )
// Bigger timeout is needed for larger entities, for example the large
// post html fixture that we load for performance tests, which often
// doesn't make it under the default 10 seconds.
.waitFor( { state: 'hidden', timeout: 60_000 } );
Copy link
Member

@kevin940726 kevin940726 Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this here? If it applies to all the tests then maybe we should add a comment above explaining why 😅 .

Copy link
Member Author

@WunderBart WunderBart Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was needed because, for performance tests, we load an enormous post fixture to test against, which often takes more than the default 10 seconds to load.

I'm not sure the rationale for using a post big enough to freeze the browser for a bit, but it's probably something to be discussed in a separate PR? /cc @youknowriad

Copy link
Member

@dmsnell dmsnell Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when I've tested with the large post in my browser it opens reasonably fast, but if I open the profiling tools I've seen it freeze for more than a minute.

I'm inferring that Chrome is spending a lot of time instrumenting the thousands of listeners and hooks when we turn of tracing, which probably happens here as well.

so this may not be realistic from a user perspective, but the wait may still occur when testing and profiling

Copy link
Member

@kevin940726 kevin940726 Jul 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, profiling adds a lot of overhead. We could assume that the overhead added shares averagely across all code and can be seen as a fair emulated slowdown of the CPU, but I don't know if that's true 😅 .

Copy link
Member

@dmsnell dmsnell Jul 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's weird. I don't understand it, but it doesn't seem to appear in the metrics. I suspect that the profiling mechanisms don't start until after this initialization. the page isn't interactive while it's starting up, but then once it finishes things are suddenly fast and fine.

so it may not be the case that Chrome reports this big delay, but it still delays the tests. based on some recent profiling work I'm wondering if we might have code paths that are to blame. there are so many listeners involved and it doesn't seem reasonable that we should have so many. maybe Chrome's profiler simply chokes at tens of thousands of listeners (or maybe this is a normal number 🤷‍♂️ I really don't know).

}