Unacceptably slow solution load times

[saul]

F# solutions that contain a relatively large number of projects (e.g. 100) that have a relatively large number of project references (approx 50 on average) take an age to load, even with Lightweight Solution Load enabled.

On my machine, "Preparing solution" takes over 6 minutes (I killed VS after this so I don't know how long it takes in full). Furthermore each project load takes several seconds. This is in stark contrast to C# solutions of the same size and with the same number of project references, which takes under a minute to load completely.

This issue is not specific to VS 2017, as the same issue occurs in VS 2015 (and most likely 2013 too). I tried to use dotTrace to analyse where all the time is being spent, but 96% of it is in "Stack traces without user methods". I'm not sure whether this means I don't have symbols for some modules, or whether the majority of the time is spent in native code.

Repro steps

I have attached a script to reproduce this issue.

1. Download and extract the attached ZIP (see end of issue)
2. Run gen.fsx
3. Open dense/Dense.sln in Visual Studio. This will load a large F# solution.
4. Compare the load times to denseCSharp/Dense.sln.

Expected behavior

Similar performance to C#, as the projects contain no code or public interface to be analysed/type inferred.

Actual behavior

Painfully slow performance :(

Known workarounds

Use VS Code.

Related information

Provide any related information

• Operating system: Windows 10
• Branch: 2017 RC2
• Editing Tools (e.g. Visual Studio Version): 2017 RC2
• Links to performance testing scripts: Perf.zip

[vasily-kirichenko]

Try to profile with dotTrace in its "Timeline" mode.

[saul]

That's what I was using - there was a lot of Visual Studio stack traces but they amounted to <5% of the total time. I saw no FSharp stack traces so I assume they aren't being loaded for whatever reason.

[vasily-kirichenko]

I can recommend to build VFT from current master, then run an experimental VS instance and profile it, you'll be able to see the sources in dotTrace.

[saul]

Thanks - it looks like 99.5% of the time is being spent in FSharpProjectNode.ComputeSourcesAndFlags as a result of OnActiveProjectCfgChange.

This in turn invokes MSBuild to do a compile of the project (with actuallyBuild set to false - I'm not sure what this entails).

Do any of the F# developers mind shedding some light onto why this is necessary during startup, and whether there are any quick wins? This would drastically reduce load times for F# solutions. /cc @dsyme, @KevinRansom

[vasily-kirichenko]

@saul this is the same results I found when I was investigating huge pause after Ctrl+F5 on VFT solution, see the exact line and a big comment from @dsyme here #2046 (comment)

[saul]

Thanks - I'd already read that but I still have a few questions for @dsyme

1. Why can't we use ComputeCompileItems to determine sources?
2. Why are the flags necessary to compute? Can't we use the same project file parsing logic instead of forking out to MSBuild (that does the same thing for us)?
3. If we really have to do it this way, can we at least show a message box whilst it's in progress? There's many situations where this locks up the UI thread for a significant amount of time (e.g. changing solution configuration).

[smoothdeveloper]

@saul, I'm not sure what is ComputeCompileItems you are mentioning, one thing though is that the files and assembly references (I guess the two main things in fsproj files) can be wrapped around conditionals that need to be evaluated to give accurate result; I'm sure you know about this.

That being said, if we could have a "dumb" but quick way of getting this info (just with a xpath query?), even with inaccuracies, and catch-up in the background for the accurate results without locking the UI, I think this would really help a lot.

[vasily-kirichenko]

Please, read the @dsyme comment https://github.com/Microsoft/visualfsharp/blob/master/vsintegration/src/FSharp.ProjectSystem.FSharp/Project.fs#L1350-L1360

[saul]

There's no need to use naive logic to get around this @smoothdeveloper. We already have access to everything to calculate the sources and flags ourselves without going via MSBuild. Not to mention MSBuild has already evaluated our project file so there's no need to guess at conditionals.

Here's my current understanding of what's happening:

1. At project load (and many other times such as changing solution configuration or adding files to the project), ComputeSourcesAndFlags is called.
2. This function invokes an MSBuild for the project with target Compile.
3. The Compile target is executed, which then executes the CoreCompile target which can be found in the Microsoft.FSharp.targets file.
4. This instantiates an Fsc (fsharp\FSharp.Build\Fsc.fs) task that calculates the compile flags and file sources, as defined in the project file.
5. MSBuild executes this Fsc task, which reflectively calls the Compile method on FSharpProjectNode (vsintegration\src\FSharp.ProjectSystem\Project.fs)
6. This Compile method then checks whether the compilation is as a result of ComputeSourcesAndFlags, in which case it simply stores the sources and flags and then bails without actually invoking FSC.

Here's my proposal for an alternative:
We are in full control of the sources array and the flags array - they are both calculated in FSharp.Build/Fsc.fs in ExecuteTool. There is no reason why we can't interrogate the BuildProject member in ComputeSourcesAndFlags: this.BuildProject.Targets["CoreCompile"].Tasks. We can then cast the ProjectTaskInstance to an Fsc instance and pull the sources and flags out then.

This completely removes the need to invoke a fake compilation with MSBuild, and we still have access to the evaluated values and so should function identically.

[dsyme]

@vasily-kirichenko @saul Just to mention that's not my comment. The approach used dates from Visual Studio 2010, blame traces back to the initial import of the code into github

Not sure why "Compile" is used, it certainly looks wrong.

[Pilchie]

Compile is the target that C# and VB use for their design-time builds to find out the options and references to the compiler as well. The new project system aims to make this asynchronous and non-blocking. In the meantime, you might running a diagnostic build and looking at the timings of the various tasks and targets that run to see if you can optimize them at all.

[dsyme]

@saul THe plan of attack you mention above sounds quite reasonable. Invoking MSBuild has always seemed spurious

[saul]

Thanks @dsyme. Do you also have any idea as to why ComputeSourcesAndFlags is called twice for every project on configuration change (including at solution load)? There's a review comment that has the same concerns - this second call to ComputeSourcesAndFlags is the reason why "Preparing solution..." takes so long. I'd like to get that fix in for RTM if possible as it's a 50% solution load time win.

Also @Pilchie - have you any idea where this is done in the C# project system? C# projects seem to load instantly which is in stark contrast to the F# project system.

[dsyme]

@saul

Thanks @dsyme. Do you also have any idea as to why ComputeSourcesAndFlags is called twice for every project on configuration change (including at solution load)?

No, I don't, sorry

[Pilchie]

@saul the C# invocations are in the closed source C++ code for csproj.dll and msvbprj.dll for most projects. Note that this is skipped for C#/VB with Lightweight solution load enabled though.

@davkean might be able point out where it happens in the new project system.