Allow for waiting for helix to use agentless tasks

[ChadNedzlek]

In order to reduce the amount of wasted compute, we can to use server jobs for waiting for helix jobs to complete, so we don't tie a build machine up just waiting for a response.

I'm not entirely sure of the mechanics here, because server jobs are incredibly limited (I don't think it's possible to, for example, access the source code, or any of the artifacts published by the build).

Related write up here: https://gist.github.com/RikkiGibson/4423b6470881ea4111a807e8db3a47a1

[RikkiGibson]

I'm not entirely sure of the mechanics here, because server jobs are incredibly limited (I don't think it's possible to, for example, access the source code, or any of the artifacts published by the build).

It sounds like this depends on what changes are more or less practical to make in Helix. The write-up that I created suggests having the build job initiate the Helix job, then having the agentless task essentially "subscribe" to the Helix job in-progress, having Helix stream back any logs, test results, etc. as they come in.

This means that further work would be needed to enable retry, though. If this were combined with the plan to automatically retry failing tests across the board dotnet/core-eng#12078, this might not be too bad of a place to land. But it is probably desirable to be able to retry the Helix job again if the automatic retries end in failure--for example, if there is a service outage of some kind.

Given that, it may make sense to instead have the build job upload the artifacts somewhere, write the URL to an AzDO pipeline variable or some other appropriate place, then have the agentless task make a request which both initiates a Helix job and subscribes to the results, including the URL to the artifacts in the request body.

[ChadNedzlek]

As long as it's only a separate job (and not a stage), we should be able to use an "output variable" to communicate between stages.

I'm hesitant to try and wire up a bunch of retry stuff, because that's a much larger and less well understood change, and the benefits are less clear. (In particular, we'd need some way to define which jobs to retry, which isn't something intrinsic to helix, so a lot of heuristics would need to be defined there, and presumably, be customizable to each repo, since no two repos use helix the same way).

[ChadNedzlek]

I think we are also going to have to make the Helix servers a lot more involved with Azure DevOps here. Right now the SDK creates and finalized the "test run", but that isn't possible in a serverless job, so the Helix server is going to have to reach into AzDO and muck about (and then somehow pass those values into the jobs, which means a fairly substantial rewrite of all the python logic as well).

[ChadNedzlek]

It's looking like this probably won't fit inside the dev workflow epic. It's growing a lot, and a lot of other priorities are showing up for our team. We'll leave it here for now, but I just wanted to add some context.

[jaredpar]

Can we get a better explanation of what happened here?

When we met with core-eng this seemed like a pretty clear win. It saves a significant amount of compute, lowers the burden for adopting Helix in medium / large repositories and reduces the level of complication that we have to add to our own infrastructure. The reduction in pressure on the Ubuntu queues is also likely to positively benefit other items like reducing the queue time for non-Helix jobs.

Everyone seemed pretty on board when we were meeting about this. The conclusion doesn't seem to match up with where the meeting ended.

[ChadNedzlek]

It's just a resourcing issue at the moment. Our team is stretched thin, and there is a lot of very high priority work that we are dealing with right now. This plan is a solid one, and definitely is something we want to do.

[markwilkie]

This is not cut for sure, but it is in a precarious state. Yes, we can (and sounds like we should) chat about this more so that we're all on the same page. We all want to do this, and we're also really struggling right now making things fit. We thought this one would be cheap, but upon more investigation, it appears less so now.

Regardless, we'd not be getting to this until after the PR stuff (including retries) - so there's definitely still time to figure the right thing.