Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataName #59092

steveharter · 2021-09-14T15:07:46Z

Change remaining source generator locations that use Compilation.GetTypeByMetadataName to an extension that addresses conflict cases. The guidance and extension implementation is from the Roslyn team.

See #57349

This is a simple 1:1 method replacement; has been previously tested in System.Text.Json (which has a unit test) and by Roslyn.

This may be ported to 6.0.

ghost · 2021-09-14T15:08:02Z

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

Change remaining source generator locations that use Compilation.GetTypeByMetadataName to an extension that address conflict cases. The guidance and extension implementation is from the Roslyn team.

See #57349

This is a simple 1:1 method replacement; has been previously tested in System.Text.Json (which has a unit test) and by Roslyn.

This may be ported to 6.0.

Author:	steveharter
Assignees:	steveharter
Labels:	`area-System.Text.Json`
Milestone:	6.0.0

...ons/tests/Microsoft.Extensions.Logging.Generators.Tests/LoggerMessageGeneratorParserTests.cs

src/libraries/System.Text.Json/gen/Reflection/RoslynExtensions.cs

steveharter · 2021-09-20T19:26:46Z

WASM issue likely caused by #51961

binlog section (crash on publish/ProxyProjectForAOTOnHelix.proj):

/datadisks/disk1/work/AF0109AA/p/build/wasm/WasmApp.Native.targets(306,5): Failed to compile /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -> /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.o
 "/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpDfyD7v.tmp
clang version 13.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.amrom.workers.dev-llvm-llvm--project 5852582532b3eb3ea8da51a1e272d8d017bd36c9)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin
 (in-process)
 "/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin/clang++" -cc1 -triple wasm32-unknown-emscripten -emit-obj --mrelax-relocations -disable-free -main-file-name Microsoft.CodeAnalysis.CSharp.dll.bc -mrelocation-model static -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -target-cpu generic -fvisibility hidden -debug-info-kind=limited -dwarf-version=4 -debugger-tuning=gdb -v -fcoverage-compilation-dir=/datadisks/disk1/work/AF0109AA/w/AB960941/e/publish -resource-dir /datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/lib/clang/13.0.0 -Oz -Werror=implicit-function-declaration -fdebug-compilation-dir=/datadisks/disk1/work/AF0109AA/w/AB960941/e/publish -ferror-limit 19 -fgnuc-version=4.2.1 -fno-inline-functions -vectorize-slp -iwithsysroot/include/SDL -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/tmpDfyD7v.tmp -x ir /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc
clang -cc1 version 13.0.0 based upon LLVM 13.0.0git default target x86_64-unknown-linux-gnu
emcc: error: '/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpDfyD7v.tmp' failed (received SIGKILL (-9)) [took 356.790s] [/datadisks/disk1/work/AF0109AA/w/AB960941/e/publish/ProxyProjectForAOTOnHelix.proj]

console log:

/datadisks/disk1/work/AAE90973/p/build/wasm/WasmApp.Native.targets(306,5): error : Failed to compile /datadisks/disk1/work/AAE90973/w/B4AD09E0/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -> /datadisks/disk1/work/AAE90973/w/B4AD09E0/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.o [/datadisks/disk1/work/AAE90973/w/B4AD09E0/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/AAE90973/p/build/wasm/WasmApp.Native.targets(306,5): error :  "/datadisks/disk1/work/AAE90973/p/build/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/AAE90973/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/AAE90973/w/B4AD09E0/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpyZM8kY.tmp [/datadisks/disk1/work/AAE90973/w/B4AD09E0/e/publish/ProxyProjectForAOTOnHelix.proj]

lewing · 2021-09-20T19:38:53Z

cc @vargaz

lewing · 2021-09-20T19:41:09Z

Something is killing the processes here, we need to figure out what and why before we can work on solving it.

steveharter · 2021-09-21T18:03:12Z

@lewing unless there is an imminent fix for the crash, can you supply the build condition to avoid the crash? I tried this but the crash is still occurring. Thanks

lewing · 2021-09-21T18:16:47Z

@steveharter have you checked against main after #59352 landed?

lewing · 2021-09-21T18:31:12Z

src/libraries/tests.proj

you shouldn't need a new group for this, the exclusions 4 lines above should be enough but it looks like the paths are incorrect? I see

./Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj ./Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.csproj

The previous section (Condition="'$(TargetsMobile)' == 'true' and '$(TargetOS)' != 'Browser'") also excludes the same tests.

I obtained the conditions from #54833 which is per #51961 (but for System.Text.Json).

The paths appear correct to me and are consistent with other references to them.

shouldn't the path start with Microsoft.Extensions.Logging.Abstractions\tests not Microsoft.Extensions.Logging\tests ?

Yes thanks. Missed that.

@eerhardt it looks like the test exclusion added in 9035d94 has the wrong path. I left it as-is in this PR.

since it's pointing at the wrong path and passing those old exclusions can probably be removed. I'll do it in a separate pr

FWIW Eric fixed that in #59608

steveharter · 2021-09-21T21:45:16Z

@steveharter have you checked against main after #59352 landed?

I just did a rebase and the same crash is occuring.

src/libraries/tests.proj

ericstj · 2021-09-28T21:33:18Z

Looks like test failure is in regex source generator on checked coreclr. May not be related to this change, see #10680. cc @stephentoub

https://dev.azure.com/dnceng/public/_build/results?buildId=1388390&view=ms.vss-test-web.build-test-results-tab&runId=40412074&resultId=192547&paneView=debug

      Timed out at 9/27/2021 8:13:44 PM after 60000ms waiting for remote process.
      Wrote mini dump to: C:\h\w\B3E009D5\w\9C35086E\uploads\3232.igz15kp2.52b.dmp
      	Process ID: 3232
      	Handle: 1756
      	Name: dotnet
      	MainModule: C:\h\w\B3E009D5\p\dotnet.exe
      	StartTime: 9/27/2021 8:12:44 PM
      	TotalProcessorTime: 00:01:02.7812500
      	Threads:
      		Thread #1 (OS 0x16D0)   [MTA]
      			[PrestubMethodFrame] (Microsoft.CodeAnalysis.GeneratorRunResult.get_GeneratedSources)
      			Microsoft.CodeAnalysis.GeneratorDriverRunResult+<>c.<get_GeneratedTrees>b__9_0(Microsoft.CodeAnalysis.GeneratorRunResult)
      			System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[Microsoft.CodeAnalysis.GeneratorRunResult, Microsoft.CodeAnalysis],[System.__Canon, System.Private.CoreLib]].ToArray()
      			System.Linq.Enumerable.ToArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
      			System.Collections.Immutable.ImmutableArray.CreateRange[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
      			System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
      			Microsoft.CodeAnalysis.GeneratorDriverRunResult.get_GeneratedTrees()
      			System.Text.RegularExpressions.Tests.RegexGeneratorHelper+<SourceGenRegexAsync>d__6.MoveNext()
      			System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(System.String, System.Nullable`1<System.Text.RegularExpressions.RegexOptions>, System.Nullable`1<System.TimeSpan>, System.Threading.CancellationToken)
      			System.Text.RegularExpressions.Tests.RegexHelpers+<GetRegexAsync>d__8.MoveNext()
      			System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Text.RegularExpressions.Tests.RegexHelpers.GetRegexAsync(System.Text.RegularExpressions.Tests.RegexEngine, System.String, System.Nullable`1<System.Text.RegularExpressions.RegexOptions>, System.Nullable`1<System.TimeSpan>)
      			System.Text.RegularExpressions.Tests.RegexMatchTests+<>c+<<Match_ExcessPrefix>b__29_0>d.MoveNext()
      			System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Text.RegularExpressions.Tests.RegexMatchTests+<>c.<Match_ExcessPrefix>b__29_0(System.String)
      			[DebuggerU2MCatchHandlerFrame]
      			[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)
      			System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
      			System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
      			Microsoft.DotNet.RemoteExecutor.Program.Main(System.String[])
      		Thread #2 (OS 0x224) [Finalizer] [Background] [MTA]
      			[DebuggerU2MCatchHandlerFrame]
      		Thread #3 (OS 0x640)  [Background] [MTA]
      			[DebuggerU2MCatchHandlerFrame]
      		Thread #5 (OS 0x1434) [Thread pool worker] [Background] [MTA]
      			[HelperMethodFrame] (System.Threading.WaitHandle.WaitOneCore)
      			System.Threading.WaitHandle.WaitOneNoCheck(Int32)
      			System.Threading.WaitHandle.WaitOne()
      			System.Threading.PortableThreadPool+GateThread.GateThreadStart()
      			System.Threading.Thread.StartCallback()
      			[DebuggerU2MCatchHandlerFrame]

ericstj · 2021-09-28T21:34:47Z

I see @stephentoub disabled that test here: 5d71e32

stephentoub · 2021-09-28T21:51:22Z

I see @stephentoub disabled that test here: 5d71e32

Yeah, the actual issue is the test is attributed with a SkipOnCoreClr attribute in order to avoid running on checked runtimes but the test isn't being skipped, as the attribute is being processed in a way different from what was expected. I disabled it while the attribute gets fixed.

ericstj · 2021-09-28T21:53:25Z

@stephentoub while you're here, this PR might be relevant for your RegEx source generator:

runtime/src/libraries/System.Text.RegularExpressions/gen/RegexGenerator.Parser.cs

Lines 49 to 51 in 5bfae1c

    
           // TODO: Use https://github.com/dotnet/runtime/pull/59092 
        
           INamedTypeSymbol? regexSymbol = compilation.GetTypeByMetadataName(RegexName); 
        
           INamedTypeSymbol? regexGeneratorAttributeSymbol = compilation.GetTypeByMetadataName(RegexGeneratorAttributeName);

edit, looks like I should expand my search by a couple lines.

ericstj · 2021-09-29T15:56:39Z

/azp run runtime

azure-pipelines · 2021-09-29T15:57:13Z

Azure Pipelines successfully started running 1 pipeline(s).

steveharter · 2021-09-29T18:41:49Z

Build failure in Build windows x64 Release SingleFile caused by timeout and not related to this PR.

steveharter · 2021-09-29T18:44:49Z

/backport to release/6.0

github-actions · 2021-09-29T18:45:01Z

Started backporting to release/6.0: https://github.com/dotnet/runtime/actions/runs/1288124568

github-actions · 2021-09-29T18:46:34Z

@steveharter backporting to release/6.0 failed, the patch most likely resulted in conflicts:

$ git am --3way --ignore-whitespace --keep-non-patch changes.patch

Applying: Use GetBestTypeByMetadataName instead of GetTypeByMetadataName
Using index info to reconstruct a base tree...
M	src/libraries/Microsoft.Extensions.Logging.Abstractions/gen/LoggerMessageGenerator.Parser.cs
Falling back to patching base and 3-way merge...
Auto-merging src/libraries/Microsoft.Extensions.Logging.Abstractions/gen/LoggerMessageGenerator.Parser.cs
CONFLICT (content): Merge conflict in src/libraries/Microsoft.Extensions.Logging.Abstractions/gen/LoggerMessageGenerator.Parser.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Use GetBestTypeByMetadataName instead of GetTypeByMetadataName
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

…me (dotnet#59092)

…me (#59092) (#59774)

steveharter added this to the 6.0.0 milestone Sep 14, 2021

steveharter requested review from benaadams, layomia, maryamariyan and sharwell September 14, 2021 15:07

steveharter requested a review from eiriktsarpalis as a code owner September 14, 2021 15:07

steveharter self-assigned this Sep 14, 2021

ghost added the area-System.Text.Json label Sep 14, 2021

steveharter force-pushed the SrcGenGetBestType branch from 6e95673 to 2f8b5cd Compare September 14, 2021 15:11

dotnet deleted a comment from azure-pipelines bot Sep 14, 2021

steveharter removed the area-System.Text.Json label Sep 14, 2021

layomia reviewed Sep 17, 2021

View reviewed changes

...ons/tests/Microsoft.Extensions.Logging.Generators.Tests/LoggerMessageGeneratorParserTests.cs Outdated Show resolved Hide resolved

src/libraries/System.Text.Json/gen/Reflection/RoslynExtensions.cs Outdated Show resolved Hide resolved

steveharter force-pushed the SrcGenGetBestType branch 2 times, most recently from db8fb31 to 129b716 Compare September 17, 2021 20:50

steveharter mentioned this pull request Sep 20, 2021

[wasm][aot] Link step failing for Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests - wasm-opt getting killed #51961

Closed

steveharter force-pushed the SrcGenGetBestType branch from c06550c to 4dda364 Compare September 20, 2021 20:07

lewing reviewed Sep 21, 2021

View reviewed changes

steveharter force-pushed the SrcGenGetBestType branch from f6cf633 to e08eb3d Compare September 21, 2021 19:21

lewing reviewed Sep 23, 2021

View reviewed changes

src/libraries/tests.proj Outdated Show resolved Hide resolved

steveharter force-pushed the SrcGenGetBestType branch from 57b2ae9 to 8cf4af3 Compare September 23, 2021 16:51

runfoapp bot mentioned this pull request Sep 23, 2021

System.Text.RegularExpressions.Tests.RegexMatchTests.Match_ExcessPrefix Failing in CI #59541

Closed

steveharter force-pushed the SrcGenGetBestType branch from 8cf4af3 to 9b20c61 Compare September 24, 2021 16:52

eerhardt mentioned this pull request Sep 25, 2021

Remove invalid test exclusions. #59608

Merged

steveharter added 10 commits September 27, 2021 09:35

Use GetBestTypeByMetadataName instead of GetTypeByMetadataName

28db0c6

Fix merge issue

25dcab8

Add unit test to Logging generator

ff67850

Misc test cleanup

a3e6eb1

Attempt to address wasm test failure

9c89de0

temp: remove ref to new code

46b3905

temp: remove ref to new code (attempt 2)

051ed1b

Add back intended code

4d6c7a3

Add test.proj ProjectExclusions

7a4eee9

Fix test exclusion path

8e2ca47

steveharter force-pushed the SrcGenGetBestType branch from 32c3b4b to 8e2ca47 Compare September 27, 2021 14:37

Disambiguate Runtime-provided from Roslyn-provided helpers

e1ac8d8

ericstj approved these changes Sep 28, 2021

View reviewed changes

steveharter merged commit 04b6c41 into dotnet:main Sep 29, 2021

steveharter deleted the SrcGenGetBestType branch September 29, 2021 18:42

steveharter added a commit to steveharter/runtime that referenced this pull request Sep 29, 2021

Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataNa…

307b2e4

…me (dotnet#59092)

steveharter mentioned this pull request Sep 29, 2021

[release/6.0] Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataName #59774

Merged

danmoseley pushed a commit that referenced this pull request Sep 30, 2021

Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataNa…

9f66ef7

…me (#59092) (#59774)

ghost locked as resolved and limited conversation to collaborators Nov 3, 2021

Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataName #59092

Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataName #59092

Uh oh!

Conversation

steveharter commented Sep 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Sep 14, 2021

Uh oh!

Uh oh!

Uh oh!

steveharter commented Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lewing commented Sep 20, 2021

Uh oh!

lewing commented Sep 20, 2021

Uh oh!

steveharter commented Sep 21, 2021

Uh oh!

lewing commented Sep 21, 2021

Uh oh!

lewing Sep 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveharter Sep 21, 2021

Choose a reason for hiding this comment

Uh oh!

lewing Sep 22, 2021

Choose a reason for hiding this comment

Uh oh!

steveharter Sep 23, 2021

Choose a reason for hiding this comment

Uh oh!

lewing Sep 23, 2021

Choose a reason for hiding this comment

Uh oh!

steveharter Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

steveharter commented Sep 21, 2021

Uh oh!

Uh oh!

ericstj commented Sep 28, 2021

Uh oh!

ericstj commented Sep 28, 2021

Uh oh!

stephentoub commented Sep 28, 2021

Uh oh!

ericstj commented Sep 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericstj commented Sep 29, 2021

Uh oh!

azure-pipelines bot commented Sep 29, 2021

Uh oh!

steveharter commented Sep 29, 2021

Uh oh!

steveharter commented Sep 29, 2021

Uh oh!

github-actions bot commented Sep 29, 2021

Uh oh!

github-actions bot commented Sep 29, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

steveharter commented Sep 14, 2021 •

edited

Loading

steveharter commented Sep 20, 2021 •

edited

Loading

lewing Sep 21, 2021 •

edited

Loading

ericstj commented Sep 28, 2021 •

edited

Loading