Unable to run/debug XUnit tests targeting Desktop from Visual Studio on latest Windows 10 machine

[jeremymeng]

Description

After the Windows 10 Creators update, I can no longer run or debug xunit tests
targeting Desktop .NET Framework in Visual Studio.

Steps to reproduce

1. On a machine with .NET 4.7 and VS 2017 15.2. I think .NET 4.7 comes with
   Windows Creators update. After I installed .NET 4.7 on another machine that had .NET 4.6, I
   don't see System.Runtime, Version=4.1.0.0 in GAC. The version
   4.0.0.0 is in GAC.

2. Create an xunit test project with dotnet new xunit.

3. In the generated .csproj file, change the target framework to net46. Also change the package reference versions.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net46</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="xunit" Version="2.2.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0" />
  </ItemGroup>

</Project>

4. Open the project in Visual Studio, rebuild the project, then look at the Test output pane

Expected:

No warnings or errors

Actual:

there's a warning

[xUnit.net 00:00:00.0226725] Skipping: test4 (could not find dependent assembly 'Microsoft.Extensions.DependencyModel, Version=1.1.0')

5. Right click the test in Test Explorer then select Run Selected Tests

Expected:

Test runs and passes.

Actual:

------ Run test started ------
[xUnit.net 00:00:00.0296561] test4: Catastrophic failure: System.TypeInitializationException: The type initializer for 'Xunit.DiaSession' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Reflection.TypeExtensions, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
   at Xunit.DiaSession..cctor()
   --- End of inner exception stack trace ---
   at Xunit.DiaSession..ctor(String assemblyFileName)
   at Xunit.DiaSessionWrapper..ctor(String assemblyFilename)
   at Xunit.XunitFrontController..ctor(AppDomainSupport appDomainSupport, String assemblyFileName, String configFileName, Boolean shadowCopy, String shadowCopyFolder, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink)
   at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo)
========== Run test finished: 0 run (0:00:00.427418) ==========

Environment

Operating system: Windows 10: a Microsoft internal selfhost version)

vstest.executionengine.x86.exe: 15.00.26228.0

My findings

• I am using the SDK-style .csproj to target net46. According to the doc,
  Desktop, UWP & Native unit testing continues to use the test platform (TPv1) located @ "%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow"..
  (I have briefly tried the method of putting a testplatform.config mentioned
  at the same link but it didn't work for me.)

• I have .NET 4.7 on my repro machine. I suspect that
  System.Runtime, Version=4.1.0.0 is in the GAC for latest windows builds.

• When discovering/executing unit tests, VS tries to get additional test
  extensions from both xunit assemblies in the following order

  • %TEMP%\VisualStudioTestExplorerExtensions\2.2.0\build\netcoreapp1.0\xunit.runner.visualstudio.dotnetcore.testadapter.dll
  • %Temp%\VisualStudioTestExplorerExtensions\2.2.0\build\_common\xunit.runner.visualstudio.testadapter.dll

• Because of the GAC version of System.Runtime.dll, Version=4.1.0.0, xunit
  test extension from the first assembly (.NET Core version) is successfully
  added.

• The test extension from the second assembly (Desktop version) is also added.

• Now VS test platform has two test adapters for executor://xunit/VsTestRunner2

• When I rebuild the solution, the VS Test discovery engine uses the first one
  (.NET Core version) and failed to load its dependency
  Microsoft.Extensions.DependencyModel, Version=1.1.0.0 because my test is
  targeting net46.
  Note, I am not clear why this dependency didn't cause problem when VS tries to
  get types via reflection from assembly
  xunit.runner.visualstudio.dotnetcore.testadapter.dll.
  It seems still able to find the test (using the second xunit test adapter?)

------ Discover test started ------
[xUnit.net 00:00:00.0042451] Skipping: test4 (could not find dependent assembly 'Microsoft.Extensions.DependencyModel, Version=1.1.0')
[xUnit.net 00:00:00.2620017]   Discovering: test4
[xUnit.net 00:00:00.3980642]   Discovered:  test4
========== Discover test finished: 1 found (0:00:00.6219404) ==========

• When I run a test from VS Test Explorer, it failed.

• On my non-reproing machine that have .NET 4.6. VS also tries to get test
  extensions from the same two xunit assemblies. But getting types via
  reflection fails for the .NET Core version because of the missing dependency
  on System.Runtime, Version=4.1.0.0. The ReflectionTypeLoadException is
  handled. Thus only the Desktop version of xunit test adapter is added to the
  list of test discoverers.

[jeremymeng]

I have briefly tried the method of putting a testplatform.config mentioned
at the same link but it didn't work for me

I can see that tpv2 is enabled, and vstest.console.exe is started. But at the same time, vstest.discoveryengine.x86.exe is also started

"C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\ENTERPRISE\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\vstest.discoveryengine.x86.exe" /parentProcessId 17480                                                                                                                                                                                                                                                                                                                  
"C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\vstest.console.exe" /parentprocessid:17480 /port:60551   

Is there any other workarounds?

[codito]

@jeremymeng thanks for detailed investigation. Looks like reflection on OM.DiaSession is failing in adapter, need to debug to find what's going on.

[Faizan2304]

@jeremymeng : I tried to create local repro but no luck. I have windows 10 with Creator update (OS build 15063.296). I installed .NET framework 4.7. I too don't see System.Runtime, Version=4.1.0.0 in GAC. The version
4.0.0.0 is in GAC.

Do you have repro machine where I can debug the issue? If not can you please try to provide us crash dump.

To get the dump, attach vstest.executionengine.x86.exe (or vstest.console.exe if you are using TPv2 by enabling it through testplatform.config) and enable first chance exception. When it throws above mentioned exception save the dump.

[jeremymeng]

@Faizan2304 I attache a debugger to Visual Stduio, and got a dump when System.TypeInitializationException is thrown. You can see that the .NET Core version of the xunit test adapter (xunit_runner_visualstudio_dotnetcore_testadapter) is used. I will share the dump with you via email.

Call stack

 # ChildEBP RetAddr  
00 0809e7bc 73725987 KERNELBASE!RaiseException+0x62 [minkernel\kernelbase\xcpt.c @ 904]
01 0809e858 7374a504 clr!RaiseTheExceptionInternalOnly+0x27c [f:\dd\ndp\clr\src\vm\excep.cpp @ 3113]
02 0809e86c 7374a56b clr!RaiseTheException+0x8e [f:\dd\ndp\clr\src\vm\excep.cpp @ 2885]
03 0809e894 7374a59a clr!RealCOMPlusThrowWorker+0x72 [f:\dd\ndp\clr\src\vm\excep.cpp @ 3159]
04 0809e8c0 7374a5a2 clr!RealCOMPlusThrow+0x2e [f:\dd\ndp\clr\src\vm\excep.cpp @ 3196]
05 0809e8cc 7364509e clr!RealCOMPlusThrow+0x7 [f:\dd\ndp\clr\src\vm\excep.cpp @ 3219]
06 0809edac 736edd36 clr!MethodTable::DoRunClassInitThrowing+0x37e [f:\dd\ndp\clr\src\vm\methodtable.cpp @ 3469]
07 0809ee24 735af6a7 clr!MethodDesc::DoPrestub+0x1ad [f:\dd\ndp\clr\src\vm\prestub.cpp @ 1220]
08 0809eea0 7359efb9 clr!PreStubWorker+0xe0 [f:\dd\ndp\clr\src\vm\prestub.cpp @ 1015]
09 0809eec4 08fba090 clr!ThePreStub+0x11 [f:\dd\ndp\clr\src\vm\i386\asmhelpers.asm @ 2104]
0a 0809eed8 08fb9faf xunit_runner_utility_netstandard15_8f40000!Xunit.DiaSessionWrapper..ctor(System.String)+0x20
0b 0809eeec 08fb9982 xunit_runner_utility_netstandard15_8f40000!Xunit.XunitFrontController..ctor(Xunit.AppDomainSupport, System.String, System.String, Boolean, System.String, Xunit.Abstractions.ISourceInformationProvider, Xunit.Abstractions.IMessageSink)+0xaf
0c 0809ef9c 08fb8e03 xunit_runner_visualstudio_dotnetcore_testadapter_8cb0000!Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.RunTestsInAssembly(Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle, LoggerHelper, Xunit.IMessageSinkWithTypes, Xunit.Runner.VisualStudio.AssemblyRunInfo)+0x15a
0d 0809efc0 7252bea6 xunit_runner_visualstudio_dotnetcore_testadapter_8cb0000!Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner+<>c__DisplayClass17_1.<RunTests>b__4(Xunit.Runner.VisualStudio.AssemblyRunInfo)+0x23
0e 0809efd8 08fb6de1 mscorlib_ni!System.Collections.Generic.List`1[[System.__Canon, mscorlib]].ForEach(System.Action`1<System.__Canon>)+0x56 [f:\dd\ndp\clr\src\BCL\system\collections\generic\list.cs @ 553]
0f 0809f020 08fb6457 xunit_runner_visualstudio_dotnetcore_testadapter_8cb0000!Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.RunTests(Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle, LoggerHelper, System.Func`1<System.Collections.Generic.List`1<Xunit.Runner.VisualStudio.AssemblyRunInfo>>)+0x269
10 0809f050 08fb5e41 xunit_runner_visualstudio_dotnetcore_testadapter_8cb0000!Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor.RunTests(System.Collections.Generic.IEnumerable`1<Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase>, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)+0x177
11 0809f0b0 08fb4bef Microsoft_VisualStudio_TestPlatform_Core_3900000!Microsoft.VisualStudio.TestPlatform.Core.TestExecutionManager.RunTestInternalWithExecutors(System.Collections.Generic.IEnumerable`1<System.Uri>, Boolean, System.Collections.Generic.Dictionary`2<System.Uri,System.Collections.Generic.IEnumerable`1<System.String>>, System.Collections.Generic.Dictionary`2<System.Uri,System.Collections.Generic.List`1<Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase>>, Microsoft.VisualStudio.TestPlatform.Core.RunContext, Int64, Microsoft.VisualStudio.TestPlatform.Common.Utilities.MeasurementBlockWrapper)+0x2f9 [F:\dd\src\vset\Agile\testplatform\rocksteady\core\TestRun\TestExecutionManager.cs @ 667]
12 0809f140 08fb3dbc Microsoft_VisualStudio_TestPlatform_Core_3900000!Microsoft.VisualStudio.TestPlatform.Core.TestExecutionManager.RunTestsInternal(System.Collections.Generic.IEnumerable`1<System.Object>, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunSettings, Microsoft.VisualStudio.TestPlatform.Core.TestExecutionContext)+0x36f [F:\dd\src\vset\Agile\testplatform\rocksteady\core\TestRun\TestExecutionManager.cs @ 556]
13 0809f178 08fb3c5c Microsoft_VisualStudio_TestPlatform_Core_3900000!Microsoft.VisualStudio.TestPlatform.Core.TestExecutionManager.RunTests(System.Collections.Generic.IEnumerable`1<System.Object>, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunSettings, Microsoft.VisualStudio.TestPlatform.Core.TestExecutionContext)+0x4c [F:\dd\src\vset\Agile\testplatform\rocksteady\core\TestRun\TestExecutionManager.cs @ 461]
14 0809f1b0 7252c507 Microsoft_VisualStudio_TestPlatform_Core_3900000!Microsoft.VisualStudio.TestPlatform.Core.TestExecutionManager.RunTests(System.Object)+0x84 [F:\dd\src\vset\Agile\testplatform\rocksteady\core\TestRun\TestExecutionManager.cs @ 406]
15 0809f1bc 72487bb5 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart_Context(System.Object)+0x73 [f:\dd\ndp\clr\src\BCL\system\threading\thread.cs @ 74]
16 0809f220 72487ac6 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0xe5 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 954]
17 0809f234 72487a81 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x16 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 902]
18 0809f250 724ec80e mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x41 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 891]
19 0809f268 7359eb36 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart(System.Object)+0x4e [f:\dd\ndp\clr\src\BCL\system\threading\thread.cs @ 93]
1a 0809f274 735a6e50 clr!CallDescrWorkerInternal+0x34 [f:\dd\ndp\clr\src\vm\i386\asmhelpers.asm @ 763]
1b 0809f2c8 735a8134 clr!CallDescrWorkerWithHandler+0x6b [f:\dd\ndp\clr\src\vm\callhelpers.cpp @ 91]
1c 0809f338 7372f2dd clr!MethodDescCallSite::CallTargetWorker+0x16a [f:\dd\ndp\clr\src\vm\callhelpers.cpp @ 655]
1d (Inline) -------- clr!MethodDescCallSite::Call+0xe [f:\dd\ndp\clr\src\vm\callhelpers.h @ 423]
1e 0809f4ac 7371dfea clr!ThreadNative::KickOffThread_Worker+0x173 [f:\dd\ndp\clr\src\vm\comsynchronizable.cpp @ 301]
1f 0809f4c4 7371e054 clr!ManagedThreadBase_DispatchInner+0x71 [f:\dd\ndp\clr\src\vm\threads.cpp @ 10273]
20 0809f568 7371e121 clr!ManagedThreadBase_DispatchMiddle+0x7e [f:\dd\ndp\clr\src\vm\threads.cpp @ 10323]
21 0809f5c4 7371e18f clr!ManagedThreadBase_DispatchOuter+0x5b [f:\dd\ndp\clr\src\vm\threads.cpp @ 10577]
22 0809f5e8 7372f192 clr!ManagedThreadBase_FullTransitionWithAD+0x2f [f:\dd\ndp\clr\src\vm\threads.cpp @ 10641]
23 (Inline) -------- clr!ManagedThreadBase::KickOff+0x15 [f:\dd\ndp\clr\src\vm\threads.cpp @ 10675]
24 0809f664 7371fa91 clr!ThreadNative::KickOffThread+0x256 [f:\dd\ndp\clr\src\vm\comsynchronizable.cpp @ 421]
25 0809fe04 765f86c4 clr!Thread::intermediateThreadProc+0x55 [f:\dd\ndp\clr\src\vm\threads.cpp @ 2872]
26 0809fe18 77305b4d KERNEL32!BaseThreadInitThunk+0x24 [base\win32\client\thread.c @ 64]
27 0809fe60 77305b1d ntdll!__RtlUserThreadStart+0x2f [minkernel\ntdll\rtlstrt.c @ 997]
28 0809fe70 00000000 ntdll!_RtlUserThreadStart+0x1b [minkernel\ntdll\rtlstrt.c @ 914]

[codito]

Workaround:

1. Rename "C:\Users<user>.nuget\packages\xunit.runner.visualstudio\2.2.0\build\netcoreapp1.0\xunit.runner.visualstudio.dotnetcore.testadapter.dll" to "C:\Users<user>.nuget\packages\xunit.runner.visualstudio\2.2.0\build\netcoreapp1.0\xunit.runner.visualstudio.dotnetcore.testadapter.dll.1"
2. Delete everything inside %temp%\VisualStudioTestExplorerExtensions
3. Restart VS, run tests

[JoeBrockhaus]

I'm getting this today as well. @codito 's workaround didn't work for me :(

edit-1: I happened to re-install the latest 2017 preview and it's working again 🤷‍♂️
edit-2: I've still got codito's workaround in place, so presumably that worked.

[craigktreasure]

Not sure if it's the same for 2.2.0, but for 2.3.0-beta3-build3705 i also commented out this section of C:\Users\<user>\AppData\Local\Temp\VisualStudioTestExplorerExtensions\2.3.0-beta3-build3705\build\netcoreapp1.0\xunit.runner.visualstudio.props to get it to build:

    <!-- <None Include="$(MSBuildThisFileDirectory)xunit.runner.visualstudio.dotnetcore.testadapter.dll"> -->
      <!-- <Link>xunit.runner.visualstudio.dotnetcore.testadapter.dll</Link> -->
      <!-- <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory> -->
      <!-- <Visible>False</Visible> -->
    <!-- </None> -->

[BradBarnich]

Seeing this on non-SDK csproj too. The workaround works, but to packages folder in the source tree.

[codito]

Created xunit/xunit#1348 and PR xunit/xunit#1349 in an attempt to fix this. This allows users on older versions of VS to be unblocked.

We need to fix the adapter lookup logic in vstest too. Please see this RFC for details on vstest changes. It will likely get implemented post 15.3 release.