Slow solution load times for massive solutions

[cartermp]

This is a continuation and up to date version of #2107.

There are two project systems used, depending on what you target:

• .NET SDK (Framework or Core or Standard) ---> CPS via the .NET Project system
• .NET Framework (big project files) ---> Legacy F# project system

The latter is used by most solutions in the wild today, but many people are migrating to the .NET SDK for a number of reasons. Both suffer solution load time woes.

SDK style project behavior

From @saul on SDK projects:

I could see that the projects were moving around/refreshing in the Solution Explorer (some projects had 'Dependencies' nodes, some didn't). After about 5 seconds, the UI completely locked up and 5 minutes later I still can't use it (it's 'Not Responding').

The following are upon testing huge/Dense.sln

Legacy project behavior

From @saul for Legacy projects:

Opening Huge/Dense.sln in VS 2015 Update 3 takes 02:00 minutes to get to a fully working editor. After the "Preparing Solution" progress dialog closes, it's basically ready to use.

In Visual Studio 15.7.1 with the nightly tools - same machine, same solution:

00:36 minutes in, the Solution Explorer appears (and is populated). The 'Preparing Solution' dialog stays open.
08:16 minutes in, the 'Preparing Solution' dialog closes so I can now edit the source files. The UI is semi-responsive, syntax colourisation is not updating.
08:53 minutes in, the colourisation is updating in real-time as you'd expect.

I've got a feeling that we've regressed on the 'compute sources and flags' again, and that we're calling this too often on load again.

This is on an 8 core, 32GB RAM server with SSD - I dread to think what this is like on my dual core laptop.

[zpodlovics]

Where are the flamegraphs to see when and what and where it happens during the project loading and/or editing? Take a look at this issue report as an inspiration example: https://github.com/dotnet/coreclr/issues/18465

[TIHan]

As far as the SDK project-system is concerned, we have the same problem with C# SDK style projects.

For legacy, it is odd for it to be slower.

[cartermp]

Just as an update, with VS 2019, solution load times for large .NET SDK-style projects will be significantly improved due to the Roslyn workspace being changed to free-threaded initialization.

[effyteva]

Looking forward for this fix on 16.1, as 16.0 seems to be even slower than VS2017.

[cartermp]

@effyteva Can you share some more details? By our own measurements solution load times for solutions using the .NET SDK are dramatically better. It's made my own usage of VisualFSharp.sln much more sane. If you're not using the .NET SDK, the old project system is effectively frozen - any performance characteristics you observed in VS 2017 would be the same in VS 2019 (but perhaps better due to platform-level memory and CPU usage reductions).

[effyteva]

@cartermp
Sorry, you're actually right - solution load time is better (.NET SDK - .NET Core projects).

I'm a bit frustrated from VS2019 (first version since the initial VS2002 release), VS2017 become much slower than VS2015, and now VS2019 feels even slower.
Compilation and publish times became much longer, I believe this happens mostly due the 32 bit architecture and some old memory leaks which still exists.
I've already restarted VS2019 4 times today (1 time it restarted itself due to low memory), as VS2019 doesn't benefit enough from the current hardware (my development machine is i7-7800X, 32GB RAM, Samsung PRO SSD).
I'm using one solution with ~70 .NET core projects (mostly class libraries), wish there were any improvement on that end...

[cartermp]

Compilation/publish can feel slower for a relatively short while due to delayed NGEN - NGEN runs asynchronously and will get to things when it gets to them. Typically within a few hours everything should be fully NGEN'd, but it could be longer. Memory leaks could be at play here, but we'd need diagnostic information to tell. #6420 fixes an issue if you are changing solutions with VS open, but the characteristic you'd observe here is an OOM rather than slow compile times. 32-bit is a red herring here. Unless you're regularly reaching 3GB in your working set (or OOMing), then VS being a 32-bit process is unrelated. Additionally, builds shell out to MSBuild, which compiles in its own process.