Kestrel emits GOAWAY frame in response to WINDOW_UPDATE on closed stream

[deanward81]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

We're in the process of migrating a high traffic 1P HTTP/2 user from .NET Framework HTTP.SYS (using HttpListener) to .NET 8 Kestrel. We've observed a low rate of GOAWAY frames being emitted from Kestrel, all with the following error message:

A frame of type WINDOW_UPDATE was received after stream {stream_number} was reset or aborted

{stream_number} in this case is always an odd number indicating a client-initiated stream.

Based upon the HTTP/2 spec and the behaviour in HTTP.SYS I suspect this is a bug in Kestrel's implementation. Section 6.2 of the spec says:

WINDOW_UPDATE can be sent by a peer that has sent a frame bearing the END_STREAM flag. This means that a receiver could receive a WINDOW_UPDATE frame on a "half-closed (remote)" or "closed" stream. A receiver MUST NOT treat this as an error (see Section 5.1).

And section 5.1 says:

WINDOW_UPDATE or RST_STREAM frames can be received in this state for a short period after a DATA or HEADERS frame containing an END_STREAM flag is sent. Until the remote peer receives and processes RST_STREAM or the frame bearing the END_STREAM flag, it might send frames of these types. Endpoints MUST ignore WINDOW_UPDATE or RST_STREAM frames received in this state, though endpoints MAY choose to treat frames that arrive a significant time after sending END_STREAM as a connection error (Section 5.4.1) of type PROTOCOL_ERROR.

Inspecting HTTP.SYS source code confirms that it simply drops the WINDOW_UPDATE frame if the stream it was received was closed. Ideally Kestrel should mirror this behaviour.

There was a similar fix for RST_STREAM: #32442 (fixed by #32449) and I see that dotnet/runtime#28820 was fixed back in 2019, but it doesn't handle ignoring WINDOW_UPDATE frames so they don't result in GOAWAY frames for the connection.

These appear to be the offending lines of code in Kestrel:

aspnetcore/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs

Lines 1159 to 1163 in 43b81a9

	if (stream.RstStreamReceived)
	{
	// Hard abort, do not allow any more frames on this stream.
	throw CreateReceivedFrameStreamAbortedException(stream);
	}

In this particular case the 1P client (which is Golang-based) has a very high volume of client cancellations so we routinely see in-flight streams being prematurely closed - Golang seems to be pretty eager with sending WINDOW_UPDATE frames so we seem to hit a race here pretty quickly whereby the server thinks the stream is closed but the client has not yet acknowledged it. I've been unable to reproduce this using a .NET client.

Expected Behavior

Kestrel should ignore the WINDOW_UPDATE frame on a closed stream, as per the spec.

Steps To Reproduce

Note: I have been unable to reproduce this using the .NET HttpClientand initiating a client disconnect using a CancellationToken.

Based upon what we've seen with a Golang client in production.

1. Initiate a new connection to a HTTP/2 endpoint
2. Initiate a request, but cancel it immediately. This should hopefully mean the server's record of the stream's state marks it as closed.
3. If the race condition occurs on the client, it can send a WINDOW_UPDATE frame after the stream was closed.
4. Server should send a GOAWAY frame in response. Underlying connection can no longer be used.

Exceptions (if any)

No response

.NET Version

8.0.19

Anything else?

No response

[deanward81]

@DeagleGross issue for WINDOW_UPDATE + GOAWAY as discussed!

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• Quarantine RST_STREAM_IncompleteRequest_AdditionalWindowUpdateFrame_ConnectionAborted #53744