Add question flag to "question" the build if it is incremental

[yuehuang010]

Fixes #7348

This PR adds "question" switch to msbuild.exe that will error out if a target or a task fails incremental check. Targets will fail if both Inputs and Outputs are present and not skip. Tasks changes are individually modified to support the interface IIncrementalTask, which sets the question boolean. Each task will need to be updated this interface take part.

I have started with the following tasks and fixed some of the issues within MSBuild enlistment. And there are more, see the notes below. Tasks updated: ToolTask, Copy, MakeDir, Touch, WriteLinesTofile, RemoveDir, DownloadFile, Move, ZipDirectory, Unzip, GenerateResource, GenerateBindingRedirects.

Using question investigate incremental issues is orders of magnitude easier. Reading the logs is simpler and repros are more consistent. In this PR, it includes a few fixes to the common targets which address some issues.

Example of a Target I/O error.

External Packages Incremental Notes

dotnet\sdk\7.0.100-rc.1.22431.12\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets Target Name="_IncrementalCleanPublishDirectory" DependsOnTargets="_GetCurrentAndPriorPublishFileWrites" WriteLinesToFile File="$(IntermediateOutputPath)$(_PublishCleanFile)" Lines="@(_CurrentPublishFileWrites)" Overwrite="true" WriteOnlyWhenDifferent="true"

.nuget\packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.22458.2\tools\VisualStudio.VsixBuild.targets
Some Copy task are missing SkipUnchangedFiles="true"

BootStrapMSBuild.targets, copy is overwriting files, thus it would never be incremental. A small refactor is required.

RAR sometimes touches the cache and cause GenerateBindingRedirects to run.
The FileState .dependencies are missing from the cache so it dirty the cache to fetched again.
See SystemState.GetAssemblyMetadata().

microsoft.sourcelink.github\1.0.0\build\Microsoft.SourceLink.GitHub.targets
Target Name="_InitializeAzureReposGitSourceLinkUrl" Inputs="@(SourceRoot)" Outputs="|%(Identity)|"

[rainersigwald]

I am very skeptical of this approach. In general, incremental build problems are hard to debug because so many things are out-of-date--if there's only one thing that's broken, searching for the incremental build key phrases (is newer than) will quickly point to what's wrong. But usually there are both many things that are expected to run on every build, and an unpleasant cascade of builds from a single source of problems.

Can you give an example of a concrete real-world incremental build problem where this would help investigate? That might help understand the value.

[yuehuang010]

The analogy is maintaining code in hard without tests. Once tests are in place and reliable, then confidence grows and the product is more manageable.

The goal of this feature is to 1) provide a fast way to investigate the issue by stopping on first incremental issue. This will shorten investigation time and more fixing time. 2) is to establish a way to validate incremental (aka testing). At the moment, there is too existing issues for 2 to be in place, but hopefully, this will lay the foundation going forward.

Here is an example of an issue fixed in this PR. With the target "GenerateBindingRedirects" can hit an state where the IO will run indefinitely. Touch the project file without changing anything. This will trigger $(MSBuildAllProjects) on the input. The GenerateBindingRedirects task will run but won't write a new file because the content are same. The output wasn't modified and the target will repeat. While this task won't lead to disk write, it causes "noise" in the (is newer than) search.

[yuehuang010]

@rainersigwald Ping.

[rainersigwald]

This is on my to-do list, hopefully early this week.

[yuehuang010]

Ping.

[Forgind]

@rainersigwald, looks like resource files are unhappy; may need to build + push again

[Forgind]

I think I screwed up 😢

[Forgind]

I have a meeting but will try to fix afterwards