Hook up initial infrastructure for unmanaged-to-managed vtable stubs.

docs/design/libraries/ComInterfaceGenerator/UnmanagedToManagedEH.md

@@ -0,0 +1,53 @@
+# Unmanaged to Managed Exception Handling
+
+As part of building out the "vtable method stub" generator as the first steps on our path to the COM source generator, we need to determine how to handle exceptions at the managed code boundary in the unmanaged-calling-managed direction. When implementing a Com Callable Wrapper (or equivalent concept), the source generator will need to generate an `[UnmanagedCallersOnly]` wrapper of a method. We do not support propogating exceptions across an `[UnmanagedCallersOnly]` method on all platforms, so we need to determine how to handle exceptions.
+
+We determined that there were three options:
+
+1. Do nothing
+2. Match the existing behavior in the runtime for `[PreserveSig]` on a method on a `[ComImport]` interface.
+3. Allow the user to provide their own "exception -> return value" translation.
+
+## Option 1: Do Nothing
+
+This option seems the simplest option. The source generator will not emit a `try-catch` block and will just let the exception propogate when in a wrapper for a `VirtualMethodTableIndex` stub. There's a big problem with this path though. One goal of our COM source generator design is to let developers drop down a particular method from the "HRESULT-swapped, all nice COM defaults" mechanism to a `VirtualMethodTableIndex`-attributed method for particular methods when they do not match the expected default behaviors. This feature becomes untenable if we do not wrap the stub in a `try-catch`, as suddenly there is no mechanism to emulate the exception handling of exceptions thrown by marshallers. For cases where all marshallers are known to not throw exceptions except in cases where the process is unrecoverable (i.e. `OutOfMemoryException`), this version may provide a slight performance improvement as there will be no exception handling in the generated stub.
+
+## Option 2: Match the existing behavior in the runtime for `[PreserveSig]` on a method on a `[ComImport]` interface
+
+The runtime has some existing built-in behavior for generating a return value from an `Exception` in an unmanaged-to-managed `COM` stub. The behavior depends on the return type, and matches the following table:
+
+| Type            | Return Value in Exceptional Scenario |
+|-----------------|--------------------------------------|
+| `void`          | Swallow the exception                |
+| `int`           | `exception.HResult`                  |
+| `uint`          | `(uint)exception.HResult`            |
+| `float`         | `float.NaN`                          |
+| `double`        | `double.NaN`                         |
+| All other types | `default`                            |
+
+We could match this behavior for all `VirtualMethodTableIndex`-attributed method stubs, but that would be forcibly encoding the rules of HResults and COM for our return types, and since COM is a Windows-centric technology, we don't want to lock users into this model in case they have different error code models that they want to integrate with. However, it does provide an exception handling boundary
+
+## Option 3: Allow the user to provide their own "exception -> return value" translation
+
+Another option would be to give the user control of how to handle the exception marshalling, with an option for some nice defaults to replicate the other options with an easy opt-in. We would provide the following enumeration and new members on `VirtualMethodTableIndexAttribute`:
+
+```diff
+namespace System.Runtime.InteropServices.Marshalling;
+
+public class VirtualMethodTableIndexAttribute
+{
++     public ExceptionMarshalling ExceptionMarshalling { get; }
++     public Type ExceptionMarshallingType { get; }
+}
+
++ public enum ExceptionMarshalling
++ {
++     None, // No exception handling (equivalent to Option 1)
++     Com, // Match the COM-focused model described in Option 2
++     Custom
++ }
+```
+
+When the user sets `ExceptionMarshalling = ExceptionMarshalling.Custom`, they must specify a marshaller type in `ExceptionMarshallingType` that unmarshals a `System.Exception` to the same unmanaged return type as the marshaller of the return type in the method's signature.
+
+To implement the `ExceptionMarshalling.Com` option, we will provide some marshallers to implement the different rules above, and select the correct rule given the unmanaged return type from the return type's marshaller. By basing the decision on the unmanaged type instead of the managed type, this mechanism will be able to kick in correctly for projects that use their own custom `HResult` struct that wraps an `int` or `uint`, as the `HResult` will be marshalled to an `int` or `uint` if it is well-defined.

src/libraries/System.Runtime.InteropServices/gen/ComInterfaceGenerator/ComInterfaceGeneratorHelpers.cs

@@ -46,6 +46,9 @@ internal static class ComInterfaceGeneratorHelpers
                 elementFactory,
                 new AttributedMarshallingModelOptions(runtimeMarshallingDisabled, MarshalMode.ManagedToUnmanagedIn, MarshalMode.ManagedToUnmanagedRef, MarshalMode.ManagedToUnmanagedOut));
 
+            // Include the native-to-managed this generator factory here as well as we use this function for all COM-related functionality
+            generatorFactory = new NativeToManagedThisMarshallerFactory(generatorFactory);
+
             generatorFactory = new ByValueContentsMarshalKindValidator(generatorFactory);
 
             return MarshallingGeneratorFactoryKey.Create((env.TargetFramework, env.TargetFrameworkVersion), generatorFactory);

src/libraries/System.Runtime.InteropServices/gen/ComInterfaceGenerator/Marshallers/NativeThisToManagedThisMarshallerFactory.cs

@@ -0,0 +1,66 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+using System;
+using System.Collections.Generic;
+using Microsoft.CodeAnalysis.CSharp;
+using Microsoft.CodeAnalysis.CSharp.Syntax;
+using static Microsoft.CodeAnalysis.CSharp.SyntaxFactory;
+
+namespace Microsoft.Interop
+{
+    internal sealed record NativeThisInfo(ManagedTypeInfo TypeKeyType) : MarshallingInfo;
+
+    internal sealed class NativeToManagedThisMarshallerFactory : IMarshallingGeneratorFactory
+    {
+        private readonly IMarshallingGeneratorFactory _inner;
+        public NativeToManagedThisMarshallerFactory(IMarshallingGeneratorFactory inner)
+        {
+            _inner = inner;
+        }
+
+        public IMarshallingGenerator Create(TypePositionInfo info, StubCodeContext context)
+            => info.MarshallingAttributeInfo is NativeThisInfo(ManagedTypeInfo typeKeyType) ? new Marshaller(typeKeyType) : _inner.Create(info, context);
+
+        private sealed class Marshaller : IMarshallingGenerator
+        {
+            private readonly ManagedTypeInfo _typeKeyType;
+
+            public Marshaller(ManagedTypeInfo typeKeyType) => _typeKeyType = typeKeyType;
+
+            public TypeSyntax AsNativeType(TypePositionInfo info) => MarshallerHelpers.SystemIntPtrType;
+            public IEnumerable<StatementSyntax> Generate(TypePositionInfo info, StubCodeContext context)
+            {
+                if (context.CurrentStage != StubCodeContext.Stage.Unmarshal)
+                {
+                    yield break;
+                }
+
+                (string managedIdentifier, string nativeIdentifier) = context.GetIdentifiers(info);
+
+                yield return ExpressionStatement(
+                    AssignmentExpression(SyntaxKind.SimpleAssignmentExpression,
+                    IdentifierName(managedIdentifier),
+                        InvocationExpression(
+                            MemberAccessExpression(SyntaxKind.SimpleMemberAccessExpression,
+                                GenericName(Identifier(TypeNames.IUnmanagedVirtualMethodTableProvider),
+                                    TypeArgumentList(
+                                        SingletonSeparatedList(
+                                            _typeKeyType.Syntax))),
+                                GenericName(Identifier("GetObjectForUnmanagedWrapper"),
+                                    TypeArgumentList(
+                                        SingletonSeparatedList(
+                                            info.ManagedType.Syntax)))),
+                            ArgumentList(
+                                SingletonSeparatedList(
+                                    Argument(IdentifierName(nativeIdentifier)))))));
+            }
+
+            public SignatureBehavior GetNativeSignatureBehavior(TypePositionInfo info) => SignatureBehavior.NativeType;
+            public ValueBoundaryBehavior GetValueBoundaryBehavior(TypePositionInfo info, StubCodeContext context) => ValueBoundaryBehavior.NativeIdentifier;
+            public bool IsSupported(TargetFramework target, Version version) => true;
+            public bool SupportsByValueMarshalKind(ByValueContentsMarshalKind marshalKind, StubCodeContext context) => false;
+            public bool UsesNativeIdentifier(TypePositionInfo info, StubCodeContext context) => true;
+        }
+    }
+}

src/libraries/System.Runtime.InteropServices/gen/ComInterfaceGenerator/UnmanagedToManagedStubGenerator.cs

@@ -0,0 +1,126 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+using System;
+using System.Collections.Generic;
+using System.Collections.Immutable;
+using System.Diagnostics;
+using System.Linq;
+using Microsoft.CodeAnalysis;
+using Microsoft.CodeAnalysis.CSharp;
+using Microsoft.CodeAnalysis.CSharp.Syntax;
+using static Microsoft.CodeAnalysis.CSharp.SyntaxFactory;
+
+namespace Microsoft.Interop
+{
+    internal sealed class UnmanagedToManagedStubGenerator
+    {
+        private const string ReturnIdentifier = "__retVal";
+        private const string InvokeSucceededIdentifier = "__invokeSucceeded";
+
+        private readonly BoundGenerators _marshallers;
+
+        private readonly NativeToManagedStubCodeContext _context;
+
+        public UnmanagedToManagedStubGenerator(
+            TargetFramework targetFramework,
+            Version targetFrameworkVersion,
+            ImmutableArray<TypePositionInfo> argTypes,
+            Action<TypePositionInfo, MarshallingNotSupportedException> marshallingNotSupportedCallback,
+            IMarshallingGeneratorFactory generatorFactory)
+        {
+            _context = new NativeToManagedStubCodeContext(targetFramework, targetFrameworkVersion, ReturnIdentifier, ReturnIdentifier);
+            _marshallers = new BoundGenerators(argTypes, CreateGenerator);
+
+            if (_marshallers.ManagedReturnMarshaller.Generator.UsesNativeIdentifier(_marshallers.ManagedReturnMarshaller.TypeInfo, _context))
+            {
+                // If we need a different native return identifier, then recreate the context with the correct identifier before we generate any code.
+                _context = new NativeToManagedStubCodeContext(targetFramework, targetFrameworkVersion, ReturnIdentifier, $"{ReturnIdentifier}{StubCodeContext.GeneratedNativeIdentifierSuffix}");
+            }
+
+            IMarshallingGenerator CreateGenerator(TypePositionInfo p)
+            {
+                try
+                {
+                    return generatorFactory.Create(p, _context);
+                }
+                catch (MarshallingNotSupportedException e)
+                {
+                    marshallingNotSupportedCallback(p, e);
+                    return new Forwarder();
+                }
+            }
+        }
+
+        /// <summary>
+        /// Generate the method body of the unmanaged-to-managed ComWrappers-based method stub.
+        /// </summary>
+        /// <param name="methodToInvoke">Name of the method on the managed type to invoke</param>
+        /// <returns>Method body of the stub</returns>
+        /// <remarks>
+        /// The generated code assumes it will be in an unsafe context.
+        /// </remarks>
+        public BlockSyntax GenerateStubBody(ExpressionSyntax methodToInvoke, CatchClauseSyntax? catchClause)
+        {
+            List<StatementSyntax> setupStatements = new();
+            GeneratedStatements statements = GeneratedStatements.Create(
+                _marshallers,
+                _context,
+                methodToInvoke);
+            bool shouldInitializeVariables = !statements.GuaranteedUnmarshal.IsEmpty || !statements.Cleanup.IsEmpty;
+            VariableDeclarations declarations = VariableDeclarations.GenerateDeclarationsForManagedToNative(_marshallers, _context, shouldInitializeVariables);
+
+            if (!statements.GuaranteedUnmarshal.IsEmpty)
+            {
+                setupStatements.Add(MarshallerHelpers.Declare(PredefinedType(Token(SyntaxKind.BoolKeyword)), InvokeSucceededIdentifier, initializeToDefault: true));
+            }
+
+            setupStatements.AddRange(declarations.Initializations);
+            setupStatements.AddRange(declarations.Variables);
+            setupStatements.AddRange(statements.Setup);
+
+            var tryStatements = new List<StatementSyntax>();
+            tryStatements.AddRange(statements.Unmarshal);
+
+            tryStatements.Add(statements.InvokeStatement);
+
+            if (!statements.GuaranteedUnmarshal.IsEmpty)
+            {
+                tryStatements.Add(ExpressionStatement(AssignmentExpression(SyntaxKind.SimpleAssignmentExpression,
+                    IdentifierName(InvokeSucceededIdentifier),
+                    LiteralExpression(SyntaxKind.TrueLiteralExpression))));
+            }
+
+            tryStatements.AddRange(statements.NotifyForSuccessfulInvoke);
+            tryStatements.AddRange(statements.PinnedMarshal);
+            tryStatements.AddRange(statements.Marshal);
+
+            List<StatementSyntax> allStatements = setupStatements;
+            List<StatementSyntax> finallyStatements = new List<StatementSyntax>();
+            if (!statements.GuaranteedUnmarshal.IsEmpty)
+            {
+                finallyStatements.Add(IfStatement(IdentifierName(InvokeSucceededIdentifier), Block(statements.GuaranteedUnmarshal)));
+            }
+
+            SyntaxList<CatchClauseSyntax> catchClauses = catchClause is not null ? SingletonList(catchClause) : default;
+
+            finallyStatements.AddRange(statements.Cleanup);
+            if (finallyStatements.Count > 0)
+            {
+                allStatements.Add(
+                    TryStatement(Block(tryStatements), catchClauses, FinallyClause(Block(finallyStatements))));
+            }
+            else
+            {
+                allStatements.Add(
+                    TryStatement(Block(tryStatements), catchClauses, @finally: null));
+            }
+
+            // Return
+            if (!_marshallers.IsUnmanagedVoidReturn)
+                allStatements.Add(ReturnStatement(IdentifierName(_context.GetIdentifiers(_marshallers.ManagedReturnMarshaller.TypeInfo).managed)));
+
+            return Block(allStatements);
+        }
+    }
+}