Support test splitting in rspec when given individual specs

[jonathanyeong]

We're implementing test selection internally and our test selection algorithm spits out a file with individual rspec specs. E.g.

./spec/models/user.rb[1:1]
./spec/models/user.rb[1:1:1:1]
./spec/models/user.rb[1:1:1:3]
./spec/models/user.rb[1:1:1:4]
./spec/models/user.rb[1:1:1:5]
./spec/models/user.rb[1:1:1:6]

If we ran the ./spec/models/user.rb we would test more than is needed. As far as I know, bktec doesn't handle splitting across individual specs but only on file globs. Is that correct?

Feel free to close this issue if this is not the right place for it! Thanks for all your help in advance.

[nprizal]

Hi @jonathanyeong, we’re planning to add support for bktec to accept a list of tests as input for splitting. However, this requires the input to be correctly formatted so we can calculate each test’s duration. We haven’t finalised the format yet but it’s likely to include a scope and name for each test to uniquely identify them in the Test Engine. You can find more information about Scope in our documentation and how we generate the scope and name for our Rspec collector in our source code.

I'm curious about your team’s current test selection algorithm. Is it simply an rspec --tag filtering or a more sophisticated approach like commit diff analysis?

[jonathanyeong]

Hey @nprizal, sounds good! We're changing our implementation slightly to use bktec plan which I think can already handle individual specs.

Our test selection algorithm uses the Gitlabs fork of the Crystalball gem which under the hood uses commit diff analysis. We also have an in-house predictor which will spit out test files rather than individual specs. Happy to hop on a call as well if you want to chat more!

[nprizal]

I assume you’re using bktec plan with the --files flag. We added --files flag so customers can do their own test selection and pass the output to bktec. However, this feature is only intended for test files. Passing a list of individual rspec specs like ./spec/models/user.rb[1:1:1:1] will technically work but the split won’t be optimal because we can’t estimate the duration of each test with that format. If you’re happy with the splitting time I think this is a good approach for selecting tests at the individual spec level and distributing them across jobs.

Our plan is to implement something similar to --files that takes a list of individual specs, so we can create an optimum test splitting plan based on the duration of each spec.