cryptography/security test failures on RHEL9/Fedora 36.

[tmds]

Some cryptography/security tests fail on RHEL9 and the next version of Fedora (36).
Both these distros have OpenSSL 3.0.

System.Security.Cryptography.Rsa.Tests.ImportExport.PublicOnlyPrivateExport fails with:

<test name="System.Security.Cryptography.Rsa.Tests.ImportExport.PublicOnlyPrivateExport" type="System.Security.Cryptography.Rsa.Tests.ImportExport" method="PublicOnlyPrivateExport" time="0.0074248" result="Fail">
<failure exception-type="Xunit.Sdk.ThrowsException">
<message>
Assert.Throws() Failure\nExpected: typeof(System.Security.Cryptography.CryptographicException)\nActual: typeof(System.OutOfMemoryException): Insufficient memory to continue the execution of the program.\n---- System.OutOfMemoryException : Insufficient memory to continue the execution of the program.
</message>
<stack-trace>
at Interop.Crypto.GetPkcs8PrivateKeySize(IntPtr pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 116 at Interop.Crypto.RentEncodePkcs8PrivateKey(SafeEvpPKeyHandle pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 134 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportPrivateKey[T](ExportPrivateKeyFunc`1 exporter) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 293 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportParameters(Boolean includePrivateParameters) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 430 at System.Security.Cryptography.Rsa.Tests.ImportExport.<>c__DisplayClass11_0.<PublicOnlyPrivateExport>b__0() in /home/tester/runtime/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/ImportExport.cs:line 237 ----- Inner Stack Trace ----- at Interop.Crypto.GetPkcs8PrivateKeySize(IntPtr pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 116 at Interop.Crypto.RentEncodePkcs8PrivateKey(SafeEvpPKeyHandle pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 134 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportPrivateKey[T](ExportPrivateKeyFunc`1 exporter) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 293 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportParameters(Boolean includePrivateParameters) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 430 at System.Security.Cryptography.Rsa.Tests.ImportExport.<>c__DisplayClass11_0.<PublicOnlyPrivateExport>b__0() in /home/tester/runtime/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/ImportExport.cs:line 237
</stack-trace>
</failure>
</test>

Other tests like System.Security.Cryptography.Rsa.Tests.RSAKeyFileTests.NoPrivKeyFromPublicOnly, System.Security.Cryptography.Rsa.Tests.RSAImportExportCspBlobTests.ExportImportPublicOnly, ... fail in a similar way: GetPkcs8PrivateKeySize throws OutOfMemoryException instead of CryptographicException.

System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix fails with:

<test name="System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix" type="System.Security.Cryptography.Tests.AsnEncodedDataTests" method="TestSubjectAlternativeName_Unix" time="0.0056087" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\n ↓ (pos 10)\nExpected: othername:<unsupported>, email:sanemail1@example.or···\nActual: othername: UPN::subjectupn1@example.org, email:sane···\n ↑ (pos 10)
</message>
<stack-trace>
at System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix() in /home/tester/runtime/src/libraries/System.Security.Cryptography/tests/AsnEncodedDataTests.cs:line 136
</stack-trace>
</failure>
</test>

System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail (same for SslStreamAlpnTest_Sync) fails with:

<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail" time="0.4657042" result="Fail">
<failure exception-type="System.AggregateException">
<message>
System.AggregateException : One or more errors occurred. (Authentication failed, see inner exception.) (Authentication failed, see inner exception.)\n---- System.Security.Authentication.AuthenticationException : Authentication failed, see inner exception.\n-------- Interop+OpenSsl+SslException : SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.\n------------ Interop+Crypto+OpenSslCryptographicException : error:0A000460:SSL routines::reason(1120)\n---- System.Security.Authentication.AuthenticationException : Authentication failed, see inner exception.\n-------- Interop+OpenSsl+SslException : SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.\n------------ Interop+Crypto+OpenSslCryptographicException : error:0A0000EB:SSL routines::no application protocol
</message>
<stack-trace>
at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88 at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55 at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail() in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 168 --- End of stack trace from previous location --- ----- Inner Stack Trace #1 (System.Security.Authentication.AuthenticationException) ----- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 418 at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120 ----- Inner Stack Trace ----- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 384 at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 161 ----- Inner Stack Trace ----- ----- Inner Stack Trace #2 (System.Security.Authentication.AuthenticationException) ----- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 418 at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120 ----- Inner Stack Trace ----- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 384 at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 161 ----- Inner Stack Trace -----
</stack-trace>
</failure>
</test>

System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success (same for SslStreamAlpnTest_Sync) fails for a number of combinations:

<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1, h2], serverProtocols: [h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.4008237" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>Assert.Equal() Failure\nExpected: \nActual: h2</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>
<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1], serverProtocols: [http/1.1, h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.3577751" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\nExpected: \nActual: http/1.1
</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>
<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1, h2], serverProtocols: [http/1.1, h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.4638295" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\nExpected: \nActual: http/1.1
</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>

cc @bartonjs @omajid

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Some cryptography/security tests fail on RHEL9 and the next version of Fedora (36).
Both these distros have OpenSSL 3.0.

System.Security.Cryptography.Rsa.Tests.ImportExport.PublicOnlyPrivateExport fails with:

<test name="System.Security.Cryptography.Rsa.Tests.ImportExport.PublicOnlyPrivateExport" type="System.Security.Cryptography.Rsa.Tests.ImportExport" method="PublicOnlyPrivateExport" time="0.0074248" result="Fail">
<failure exception-type="Xunit.Sdk.ThrowsException">
<message>
Assert.Throws() Failure\nExpected: typeof(System.Security.Cryptography.CryptographicException)\nActual: typeof(System.OutOfMemoryException): Insufficient memory to continue the execution of the program.\n---- System.OutOfMemoryException : Insufficient memory to continue the execution of the program.
</message>
<stack-trace>
at Interop.Crypto.GetPkcs8PrivateKeySize(IntPtr pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 116 at Interop.Crypto.RentEncodePkcs8PrivateKey(SafeEvpPKeyHandle pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 134 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportPrivateKey[T](ExportPrivateKeyFunc`1 exporter) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 293 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportParameters(Boolean includePrivateParameters) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 430 at System.Security.Cryptography.Rsa.Tests.ImportExport.<>c__DisplayClass11_0.<PublicOnlyPrivateExport>b__0() in /home/tester/runtime/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/ImportExport.cs:line 237 ----- Inner Stack Trace ----- at Interop.Crypto.GetPkcs8PrivateKeySize(IntPtr pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 116 at Interop.Crypto.RentEncodePkcs8PrivateKey(SafeEvpPKeyHandle pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 134 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportPrivateKey[T](ExportPrivateKeyFunc`1 exporter) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 293 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportParameters(Boolean includePrivateParameters) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 430 at System.Security.Cryptography.Rsa.Tests.ImportExport.<>c__DisplayClass11_0.<PublicOnlyPrivateExport>b__0() in /home/tester/runtime/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/ImportExport.cs:line 237
</stack-trace>
</failure>
</test>

Other tests like System.Security.Cryptography.Rsa.Tests.RSAKeyFileTests.NoPrivKeyFromPublicOnly, System.Security.Cryptography.Rsa.Tests.RSAImportExportCspBlobTests.ExportImportPublicOnly, ... fail in a similar way: GetPkcs8PrivateKeySize throws OutOfMemoryException instead of CryptographicException.

System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix fails with:

<test name="System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix" type="System.Security.Cryptography.Tests.AsnEncodedDataTests" method="TestSubjectAlternativeName_Unix" time="0.0056087" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\n ↓ (pos 10)\nExpected: othername:<unsupported>, email:sanemail1@example.or···\nActual: othername: UPN::subjectupn1@example.org, email:sane···\n ↑ (pos 10)
</message>
<stack-trace>
at System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix() in /home/tester/runtime/src/libraries/System.Security.Cryptography/tests/AsnEncodedDataTests.cs:line 136
</stack-trace>
</failure>
</test>

System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail (same for SslStreamAlpnTest_Sync) fails with:

<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail" time="0.4657042" result="Fail">
<failure exception-type="System.AggregateException">
<message>
System.AggregateException : One or more errors occurred. (Authentication failed, see inner exception.) (Authentication failed, see inner exception.)\n---- System.Security.Authentication.AuthenticationException : Authentication failed, see inner exception.\n-------- Interop+OpenSsl+SslException : SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.\n------------ Interop+Crypto+OpenSslCryptographicException : error:0A000460:SSL routines::reason(1120)\n---- System.Security.Authentication.AuthenticationException : Authentication failed, see inner exception.\n-------- Interop+OpenSsl+SslException : SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.\n------------ Interop+Crypto+OpenSslCryptographicException : error:0A0000EB:SSL routines::no application protocol
</message>
<stack-trace>
at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88 at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55 at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail() in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 168 --- End of stack trace from previous location --- ----- Inner Stack Trace #1 (System.Security.Authentication.AuthenticationException) ----- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 418 at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120 ----- Inner Stack Trace ----- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 384 at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 161 ----- Inner Stack Trace ----- ----- Inner Stack Trace #2 (System.Security.Authentication.AuthenticationException) ----- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 418 at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120 ----- Inner Stack Trace ----- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 384 at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 161 ----- Inner Stack Trace -----
</stack-trace>
</failure>
</test>

System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success (same for SslStreamAlpnTest_Sync) fails for a number of combinations:

<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1, h2], serverProtocols: [h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.4008237" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>Assert.Equal() Failure\nExpected: \nActual: h2</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>
<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1], serverProtocols: [http/1.1, h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.3577751" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\nExpected: \nActual: http/1.1
</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>
<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1, h2], serverProtocols: [http/1.1, h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.4638295" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\nExpected: \nActual: http/1.1
</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>

cc @bartonjs @omajid

Author:	tmds
Assignees:	-
Labels:	area-System.Net.Security, untriaged
Milestone:	-

[vcsjones]

Some cryptography/security tests fail on RHEL9 and the next version of Fedora (36).

I will take a look at the GetPkcs8PrivateKeySize issue. I'm assuming this will reproduce in Rawhide if it occurs in Fedora 36?

[omajid]

Yup, Rawhide should be good enough.

(I don't think the Fedora 36 branch exists yet. I think @tmds meant Rawhide when he wrote "the next version of Fedora (36)" above.)

[vcsjones]

@bartonjs I was able to reproduce the OOM using just some plain C, and yes it looks like OpenSSL 3 report an out-of-memory at the end of the error queue:

2044BDB8FFFF0000:error:068000DE:asn1 encoding routines:asn1_template_ex_i2d:illegal zero content:crypto/asn1/tasn_enc.c:374:
2044BDB8FFFF0000:error:1C8C0100:Provider routines:key_to_p8info:malloc failure:providers/implementations/encode_decode/encode_key2any.c:94:

Since CreateOpenSslCryptographicException spins though the error queue and we use the last one to determine if it is an out of memory, we end up throwing an out of memory.

What do you think we should do here? Some ideas:

1. Clear the error queue in CryptoNative_GetPkcs8PrivateKeySize and set our own if EVP_PKEY2PKCS8 returns null.
2. Catch the OutOfMemoryException in GetPkcs8PrivateKeySize (and any other managed methods that call the native get key size)
3. Stop throwing OOMs completely, or wrap the OOM with a CryptographicException in the throw helper.
4. File a issue with OpenSSL

[bartonjs]

A malloc failure does seem unexpected to me there. They'll probably say that we should be reading the first error instead of the last one, which is the change that I/we need to do before we can move DSA over to the EVP_PKEY API.

If you agree it's weird, then (4)'s not bad, but we probably need to do (1) also.

[wfurt]

any chance we are trying to allocate negative number?

[vcsjones]

any chance we are trying to allocate negative number?

It's not us allocating, it's OpenSSL setting the error here:

https://github.com/openssl/openssl/blob/e5fb4b1469f317aa92768cdf804dfa29b72cb8f3/providers/implementations/encode_decode/encode_key2any.c#L94

    if ((p8info = PKCS8_PRIV_KEY_INFO_new()) == NULL
        || (derlen = k2d(key, &der)) <= 0
        || !PKCS8_pkey_set0(p8info, OBJ_nid2obj(key_nid), 0,
                            params_type, params, der, derlen)) {
        ERR_raise(ERR_LIB_PROV, ERR_R_MALLOC_FAILURE); // <-- Error is set here

I'm guessing the ERR_R_MALLOC_FAILURE makes sense if p8info is null, but it's the PKCS8_pkey_set0 that fails and it lumps that in with the allocation failure.

[bartonjs]

Oh, it's PKCS8_pkey_set0 that failed? I guessed k2d returned -1 since "return -1" is the line right after the error:068000DE:asn1 encoding routines:asn1_template_ex_i2d:illegal zero content:crypto/asn1/tasn_enc.c:374: error.

[vcsjones]

Oh, it's PKCS8_pkey_set0 that failed?

It seems I misremembered the part of the if failed for something I checked only 2 hours ago 😄. You're right. That also lines up with the error queue.