Add support for running XUnit and MSTest tests with the VSTest runner #5347
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| <?xml version="1.0" encoding="utf-8"?> | ||
| <!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. See the LICENSE file in the project root for more information. --> | ||
| <Project> | ||
| <ItemGroup> | ||
| <PackageReference Include="Microsoft.TestPlatform" Version="$(MicrosoftTestPlatformVersion)" IsImplicitlyDefined="true" PrivateAssets="all" Publish="true"/> | ||
| <PackageReference Include="MSTest.TestFramework" Version="$(MSTestTestFrameworkVersion)" IsImplicitlyDefined="true" PrivateAssets="all" Publish="true"/> | ||
| <PackageReference Include="MSTest.TestAdapter" Version="$(MSTestTestAdapterVersion)" IsImplicitlyDefined="true" PrivateAssets="all" Publish="true"/> | ||
| </ItemGroup> | ||
|
|
||
| <Import Project="..\VSTest.targets"/> | ||
| </Project> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| <?xml version="1.0" encoding="utf-8"?> | ||
| <!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. See the LICENSE file in the project root for more information. --> | ||
| <Project> | ||
| <ItemGroup> | ||
| <PackageReference Include="Microsoft.TestPlatform" Version="$(MicrosoftTestPlatformVersion)" IsImplicitlyDefined="true" PrivateAssets="all" Publish="true"/> | ||
| <PackageReference Include="NUnit" Version="$(NUnitVersion)" IsImplicitlyDefined="true" PrivateAssets="all" Publish="true"/> | ||
| <PackageReference Include="NUnit3TestAdapter" Version="$(NUnit3TestAdapterVersion)" IsImplicitlyDefined="true" PrivateAssets="all" Publish="true"/> | ||
| </ItemGroup> | ||
|
|
||
| <Import Project="..\VSTest.targets"/> | ||
| </Project> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| <Project> | ||
| <Target Name="RunTests" | ||
| Outputs="%(TestToRun.ResultsStdOutPath)"> | ||
| <PropertyGroup> | ||
| <_TestEnvironment>%(TestToRun.EnvironmentDisplay)</_TestEnvironment> | ||
| <_TestAssembly>%(TestToRun.Identity)</_TestAssembly> | ||
| <_TestRuntime>%(TestToRun.TestRuntime)</_TestRuntime> | ||
| <_TestTimeout>%(TestToRun.TestTimeout)</_TestTimeout> | ||
| <_TestRunnerAdditionalArguments>%(TestToRun.TestRunnerAdditionalArguments)</_TestRunnerAdditionalArguments> | ||
|
|
||
| <!-- Always use net472 for desktop to enable displaying source location from Portable PDBs in stack traces --> | ||
| <_TestRunnerTargetFramework>net472</_TestRunnerTargetFramework> | ||
| <_TestRunnerTargetFramework Condition="'$(_TestRuntime)' == 'Core'">netcoreapp2.0</_TestRunnerTargetFramework> | ||
| <_TestRunnerTargetFramework Condition="%(TestToRun.TargetFramework) == 'netcoreapp1.1' or %(TestToRun.TargetFramework) == 'netcoreapp1.0'">netcoreapp1.0</_TestRunnerTargetFramework> | ||
|
|
||
| <_TargetDir>$([System.IO.Path]::GetDirectoryName('$(_TestAssembly)'))\</_TargetDir> | ||
|
|
||
| <_TestResultDirectory>$([System.IO.Path]::GetDirectoryName('%(TestToRun.ResultsXmlPath)'))</_TestResultDirectory> | ||
| <_TestResultFileName>$([System.IO.Path]::GetFileName('%(TestToRun.ResultsXmlPath)'))</_TestResultFileName> | ||
|
|
||
| <_TestRunnerCommand>$(DotNetTool) test $(_TestAssembly) --logger:"console%3Bverbosity=normal" --logger:"trx%3BLogFileName=$(_TestResultFileName)" "--ResultsDirectory:$(_TestResultDirectory)" "--Framework:%(TestToRun.TargetFrameworkIdentifier),Version=%(TestToRun.TargetFrameworkVersion)"</_TestRunnerCommand> | ||
| <_TestRunnerCommand Condition="'$(VSTestRunSettingsFile)' != ''">$(_TestRunnerCommand) "--settings:$(VSTestRunSettingsFile)"</_TestRunnerCommand> | ||
| <_TestRunnerCommand Condition="'$(_TestRunnerAdditionalArguments)' != ''">$(_TestRunnerCommand) $(_TestRunnerAdditionalArguments)</_TestRunnerCommand> | ||
|
|
||
| <!-- | ||
| Redirect std output of the runner. | ||
| --> | ||
| <_TestRunnerCommand Condition="'$(TestCaptureOutput)' != 'false'">$(_TestRunnerCommand) > "%(TestToRun.ResultsStdOutPath)" 2>&1</_TestRunnerCommand> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <_OutputFiles Include="%(TestToRun.ResultsXmlPath)" /> | ||
| <_OutputFiles Include="%(TestToRun.ResultsHtmlPath)" /> | ||
| <_OutputFiles Include="%(TestToRun.ResultsStdOutPath)" /> | ||
| </ItemGroup> | ||
|
|
||
| <MakeDir Directories="@(_OutputFiles->'%(RootDir)%(Directory)')"/> | ||
| <Delete Files="@(_OutputFiles)" /> | ||
|
|
||
| <Message Text="Running tests: $(_TestAssembly) [$(_TestEnvironment)]" Importance="high"/> | ||
| <Exec Command='$(_TestRunnerCommand)' | ||
| LogStandardErrorAsError="false" | ||
| WorkingDirectory="$(_TargetDir)" | ||
| IgnoreExitCode="true" | ||
| Timeout="$(_TestTimeout)" | ||
| ContinueOnError="WarnAndContinue"> | ||
| <Output TaskParameter="ExitCode" PropertyName="_TestErrorCode" /> | ||
| </Exec> | ||
|
|
||
| <!-- | ||
| Add command line to the log. | ||
| --> | ||
| <WriteLinesToFile File="%(TestToRun.ResultsStdOutPath)" | ||
| Overwrite="false" | ||
| Lines=";=== COMMAND LINE ===;$(_TestRunnerCommand)" | ||
| Condition="'$(TestCaptureOutput)' != 'false'" /> | ||
|
|
||
| <!-- | ||
| Report test status. | ||
| --> | ||
| <Message Text="Tests succeeded: $(_TestAssembly) [$(_TestEnvironment)]" Condition="'$(_TestErrorCode)' == '0'" Importance="high" /> | ||
|
Member
There was a problem hiding this comment. VSTest already does all the test logging.
Member
Author
There was a problem hiding this comment. VSTest output doesn't get placed in the build log.
Member
There was a problem hiding this comment. Right, that's because of the indirection that I mentioned. I don't we should continue doing this with VSTest and just directly invoke it. |
||
|
|
||
| <PropertyGroup> | ||
| <_ResultsFileToDisplay>%(TestToRun.ResultsHtmlPath)</_ResultsFileToDisplay> | ||
| <_ResultsFileToDisplay Condition="!Exists('$(_ResultsFileToDisplay)')">%(TestToRun.ResultsStdOutPath)</_ResultsFileToDisplay> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- | ||
| Ideally we would set ContinueOnError="ErrorAndContinue" so that when a test fails in multi-targeted test project | ||
| we'll still run tests for all target frameworks. ErrorAndContinue doesn't work well on Linux though: https://github.com/Microsoft/msbuild/issues/3961. | ||
| --> | ||
| <Error Text="Tests failed: $(_ResultsFileToDisplay) [$(_TestEnvironment)]" Condition="'$(_TestErrorCode)' != '0'" File="VSTest" /> | ||
|
Member
There was a problem hiding this comment. VSTest already logs the generated log files like trx and html to stdout. Why are we duplicating that logic?
Member
Author
There was a problem hiding this comment. Same reason as above, the output from VSTest is written into a txt file, this shows up in the build log as an error and causes the build to fail when tests fail. |
||
|
|
||
| <ItemGroup> | ||
| <FileWrites Include="@(_OutputFiles)"/> | ||
| </ItemGroup> | ||
| </Target> | ||
| </Project> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| <Project> | ||
|
Contributor
There was a problem hiding this comment. What do we expect to use this runner type? Is there some benefit to using this rather than vstest?
Member
Author
There was a problem hiding this comment. With the current implementation xunit tests default to using this runner. This is the runner it was using before I added vstest as an option. The xunit console runner works correctly for a few cases where vstest doesn't do the right thing. @tmat can elaborate, but I believe it was tests running on desktop don't get the correct runtime framework version so the compat switches are not set how the test expects.
Contributor
There was a problem hiding this comment. Why are we defaulting to the thing we don't want people using? Is the plan to change the default later and then remove this implementation, so that vstest is the only option? Is the desktop scenario something that's important for arcade in master to be supporting?
Member
Author
There was a problem hiding this comment. I think the desktop scenario is still important because VS is desktop and some of the repos run inside VS.
Contributor
There was a problem hiding this comment. Does that imply some repos are staying on netstandard2.0 forever? That's certainly an expensive proposition from a support perspective. |
||
| <PropertyGroup> | ||
| <XUnitDesktopSettingsFile Condition="'$(XUnitDesktopSettingsFile)' == ''">$(MSBuildThisFileDirectory)xunit.runner.json</XUnitDesktopSettingsFile> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- | ||
| Include settings file (xunit.runner.json) if specified. | ||
| --> | ||
| <ItemGroup> | ||
| <None Include="$(XUnitDesktopSettingsFile)" | ||
| CopyToOutputDirectory="PreserveNewest" | ||
| Visible="false" | ||
| Condition="'$(XUnitDesktopSettingsFile)' != '' and '$(TargetFrameworkIdentifier)' == '.NETFramework'" /> | ||
|
|
||
| <None Include="$(XUnitCoreSettingsFile)" | ||
| CopyToOutputDirectory="PreserveNewest" | ||
| Visible="false" | ||
| Condition="'$(XUnitCoreSettingsFile)' != '' and '$(TargetFrameworkIdentifier)' == '.NETCoreApp'" /> | ||
| </ItemGroup> | ||
|
|
||
| <Target Name="RunTests" | ||
| Outputs="%(TestToRun.ResultsStdOutPath)"> | ||
| <Telemetry EventName="NETCORE_ENGINEERING_TELEMETRY" EventData="Category=Test" /> | ||
| <PropertyGroup> | ||
| <_TestEnvironment>%(TestToRun.EnvironmentDisplay)</_TestEnvironment> | ||
| <_TestAssembly>%(TestToRun.Identity)</_TestAssembly> | ||
| <_TestRuntime>%(TestToRun.TestRuntime)</_TestRuntime> | ||
| <_TestTimeout>%(TestToRun.TestTimeout)</_TestTimeout> | ||
| <_TestRunnerAdditionalArguments>%(TestToRun.TestRunnerAdditionalArguments)</_TestRunnerAdditionalArguments> | ||
|
|
||
| <!-- Always use net472 for desktop to enable displaying source location from Portable PDBs in stack traces --> | ||
| <_TestRunnerTargetFramework>net472</_TestRunnerTargetFramework> | ||
| <_TestRunnerTargetFramework Condition="'$(_TestRuntime)' == 'Core'">netcoreapp2.0</_TestRunnerTargetFramework> | ||
| <_TestRunnerTargetFramework Condition="%(TestToRun.TargetFramework) == 'netcoreapp1.1' or %(TestToRun.TargetFramework) == 'netcoreapp1.0'">netcoreapp1.0</_TestRunnerTargetFramework> | ||
| </PropertyGroup> | ||
|
|
||
| <PropertyGroup Condition="'$(_TestRuntime)' == 'Core'"> | ||
| <_TargetFileNameNoExt>$([System.IO.Path]::GetFileNameWithoutExtension('$(_TestAssembly)'))</_TargetFileNameNoExt> | ||
| <_TargetDir>$([System.IO.Path]::GetDirectoryName('$(_TestAssembly)'))\</_TargetDir> | ||
| <_CoreRuntimeConfigPath>$(_TargetDir)$(_TargetFileNameNoExt).runtimeconfig.json</_CoreRuntimeConfigPath> | ||
| <_CoreDepsPath>$(_TargetDir)$(_TargetFileNameNoExt).deps.json</_CoreDepsPath> | ||
|
|
||
| <_TestRunner Condition="'%(TestToRun.Architecture)'=='x86' And Exists('$(DotNetRoot)x86\dotnet.exe')">$(DotNetRoot)x86\dotnet.exe</_TestRunner> | ||
| <_TestRunner Condition="'$(_TestRunner)'==''">$(DotNetTool)</_TestRunner> | ||
|
|
||
| <_TestRunnerArgs>exec --depsfile "$(_CoreDepsPath)" --runtimeconfig "$(_CoreRuntimeConfigPath)" $(TestRuntimeAdditionalArguments) "$(NuGetPackageRoot)xunit.runner.console/$(XUnitVersion)/tools/$(_TestRunnerTargetFramework)/xunit.console.dll" "$(_TestAssembly)" -noautoreporters -xml "%(TestToRun.ResultsXmlPath)" -html "%(TestToRun.ResultsHtmlPath)" $(_TestRunnerAdditionalArguments)</_TestRunnerArgs> | ||
| </PropertyGroup> | ||
|
|
||
| <PropertyGroup Condition="'$(_TestRuntime)' != 'Core'"> | ||
| <_XUnitConsoleExe>xunit.console.exe</_XUnitConsoleExe> | ||
| <_XUnitConsoleExe Condition="'%(TestToRun.Architecture)' == 'x86'">xunit.console.x86.exe</_XUnitConsoleExe> | ||
| <_XUnitConsoleExePath>$(NuGetPackageRoot)xunit.runner.console\$(XUnitVersion)\tools\$(_TestRunnerTargetFramework)\$(_XUnitConsoleExe)</_XUnitConsoleExePath> | ||
|
|
||
| <_TestRunnerArgs>"$(_TestAssembly)" -noshadow -xml "%(TestToRun.ResultsXmlPath)" -html "%(TestToRun.ResultsHtmlPath)" $(_TestRunnerAdditionalArguments)</_TestRunnerArgs> | ||
| <_TestRunnerArgs Condition="'$(_TestRuntime)' == 'Mono'">$(TestRuntimeAdditionalArguments) "$(_XUnitConsoleExePath)" $(_TestRunnerArgs)</_TestRunnerArgs> | ||
|
|
||
| <_TestRunner Condition="'$(_TestRuntime)' == 'Mono'">$(MonoTool)</_TestRunner> | ||
| <_TestRunner Condition="'$(_TestRuntime)' != 'Mono'">$(_XUnitConsoleExePath)</_TestRunner> | ||
| </PropertyGroup> | ||
|
|
||
| <PropertyGroup> | ||
| <_TestRunnerCommand>"$(_TestRunner)" $(_TestRunnerArgs)</_TestRunnerCommand> | ||
|
|
||
| <!-- | ||
| Redirect std output of the runner. | ||
| Note that xUnit outputs failure info to both STDOUT (stack trace, message) and STDERR (failed test name) | ||
| --> | ||
| <_TestRunnerCommand Condition="'$(TestCaptureOutput)' != 'false'">$(_TestRunnerCommand) > "%(TestToRun.ResultsStdOutPath)" 2>&1</_TestRunnerCommand> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <_OutputFiles Include="%(TestToRun.ResultsXmlPath)" /> | ||
| <_OutputFiles Include="%(TestToRun.ResultsHtmlPath)" /> | ||
| <_OutputFiles Include="%(TestToRun.ResultsStdOutPath)" /> | ||
| </ItemGroup> | ||
|
|
||
| <MakeDir Directories="@(_OutputFiles->'%(RootDir)%(Directory)')"/> | ||
| <Delete Files="@(_OutputFiles)" /> | ||
|
|
||
| <Message Text="Running tests: $(_TestAssembly) [$(_TestEnvironment)]" Importance="high"/> | ||
| <Exec Command='$(_TestRunnerCommand)' | ||
| LogStandardErrorAsError="false" | ||
| WorkingDirectory="$(_TargetDir)" | ||
| IgnoreExitCode="true" | ||
| Timeout="$(_TestTimeout)" | ||
| ContinueOnError="WarnAndContinue"> | ||
| <Output TaskParameter="ExitCode" PropertyName="_TestErrorCode" /> | ||
| </Exec> | ||
|
|
||
| <!-- | ||
| Add command line to the log. | ||
| --> | ||
| <WriteLinesToFile File="%(TestToRun.ResultsStdOutPath)" | ||
| Overwrite="false" | ||
| Lines=";=== COMMAND LINE ===;$(_TestRunnerCommand)" | ||
| Condition="'$(TestCaptureOutput)' != 'false'" /> | ||
|
|
||
| <!-- | ||
| Report test status. | ||
| --> | ||
| <Message Text="Tests succeeded: $(_TestAssembly) [$(_TestEnvironment)]" Condition="'$(_TestErrorCode)' == '0'" Importance="high" /> | ||
|
|
||
| <PropertyGroup> | ||
| <_ResultsFileToDisplay>%(TestToRun.ResultsHtmlPath)</_ResultsFileToDisplay> | ||
| <_ResultsFileToDisplay Condition="!Exists('$(_ResultsFileToDisplay)')">%(TestToRun.ResultsStdOutPath)</_ResultsFileToDisplay> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- | ||
| Ideally we would set ContinueOnError="ErrorAndContinue" so that when a test fails in multi-targeted test project | ||
| we'll still run tests for all target frameworks. ErrorAndContinue doesn't work well on Linux though: https://github.com/Microsoft/msbuild/issues/3961. | ||
| --> | ||
| <Error Text="Tests failed: $(_ResultsFileToDisplay) [$(_TestEnvironment)]" Condition="'$(_TestErrorCode)' != '0'" File="XUnit" /> | ||
|
|
||
| <ItemGroup> | ||
| <FileWrites Include="@(_OutputFiles)"/> | ||
| </ItemGroup> | ||
| </Target> | ||
| </Project> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems we are still using this property. We also use UsingToolVSTest now. Both should be listed here with their defaults (either opt-in or opt-out).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am only reading UsingToolXUnit as a back-compat for cases where it was set to false for some reason. I will add UsingToolVSTest here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It still should be here, perhaps with a comment. It is useful to see all the values here for reference.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
UsingToolVSTest is a bit more strange too. Its default changes based on what test framework you are using. Its false by default if you are using XUnit, but it is required to be true if using another test framework.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So then I'm not sure naming it
UsingToolXxxis appropriate. These switches are used for repository-wide configuration, not something that should differ project to project.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will name it
UseVSTestRunnerthen.