Is time-based auto-chunking useful?

[kainino0x]

In #2770 we added a mechanism to automatically subdivide WPT tests into chunks that don't individually take too long. This was to make the tests runnable under the WPT harness-adapter in runtimes that have per-test limits (like Chromium's 5-second default limit, though Chromium does not use the WPT adapter).

@mwyrzykowski @ErichDonGubler do you know if you're relying on this mechanism at all? It takes some work to maintain listing_meta.json and we have not been keeping it up to date. So I'd like to remove it, if it has turned out not to be useful.

(Apologies if I asked this before. I can't find it.)

[mwyrzykowski]

We are not using it @kainino0x , though we have been running into timeouts for some of the slower tests, though I think that is a separate issue.

[ErichDonGubler]

CC @jimblandy.

Firefox is not currently using this for its WPT executions, partly because we have some fundamental work that distorts runtimes pretty significantly from what our "base" performance could be (see bug 1870699). I will experience no sadness if you eliminate this maintenance burden, and I believe I can speak for Firefox's team on that. Also, this seems like something that can be reverted, if we suspect it does provide value down the line.

[kainino0x]

Firefox is not currently using this for its WPT executions, partly because we have some fundamental work that distorts runtimes pretty significantly from what our "base" performance could be (see bug 1870699).

If you do want to use this, the solution to this would be to "merge" Firefox runtimes in with the ones we generated using Chromium, so that the chunking assumes the worst of the two runtimes. It'll make more chunks, but that's fine.

It's not too hard but we could probably do that for you - I don't think I ran Firefox when I generated the initial data, but just because Firefox's implementation was in the middle of rapidly becoming more complete.

OTOH if you aren't running into major issues right now with getting tests to run, you probably have a better timeout system than we did already, and don't need this at all. Chromium's problem was its 5-second global timeout on each WPT test, which we avoided by setting up a heartbeat-based timeout system. (We hook CTS in a bunch of places to send a heartbeat signal, to reset the timeout timer in the test driver.)

[jimblandy]

Some of Firefox's runtimes are catastrophic; see the bugzilla link if you are curious about the details. Incorporating that data into any dataset would make it pretty weird. It will get much better soon.