Skip to content

HTTP/3 stress tests (requires libmsquic.so) #54762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
CarnaViire wants to merge 1 commit into from
Closed

HTTP/3 stress tests (requires libmsquic.so) #54762

CarnaViire wants to merge 1 commit into from

Conversation

@CarnaViire
Copy link
Member

@CarnaViire CarnaViire commented Jun 25, 2021

This is a temporary workaround to run stress tests for HTTP/3 with a pre-built libmsquic.so (until we have proper packaging for Linux).

How to run the tests

Before running, you will need to acquire libmsquic.so and libmsquic.lttng.so and put them to stress dir.
To run the stress for HTTP/3, you need to specify 3.0 on both client and server.

cd ${RUNTIME_DIR}/src/libraries/System.Net.Http/tests/StressTests/HttpStress/
cp ${MSQUIC_LIBS_DIR}/libmsquic.so .
cp ${MSQUIC_LIBS_DIR}/libmsquic.lttng.so .
./run-docker-compose.ps1 -clientstressargs "-maxExecutionTime 10 -http 3.0" -serverstressargs "-http 3.0"

How to get libmsquic.so

I will paste a link to pre-built libs, but if you wish to build it yourself, you can use Dockerfile from msquic repo, run it and then copy the libs from the container to local machine.
It might be helpful to substitute container entrypoint from "/run_endpoint.sh" to "/bin/bash" so the container would not terminate prematurely at the end of the script.
If building docker image on Windows, shell scripts need to be updated from CRLF to LF EOL or they won't work (with a very confusing error "command not found"). From the top of my head it's "scripts/install-powershell-docker.sh", "submodules/openssl/config" and "scripts/run_endpoint.sh".

Leveraging local changes to System.Net.Quic.dll or System.Net.Http.dll

I would strongly advise against building whole clr+libs by passing -b flag to run-docker-compose.ps1 as it takes tons of time. With my updates to Dockerfile, container will already have nightly dotnet build from main. It should be enough to just substitute specific dll -- build it locally for Linux, copy to stress dir and uncomment COPY command. (see Dockerfile)

Results

I haven't run for too long (maxExecutionTime=10 minutes) but it was enough for requests to stop going through and just time out - I'd suspect improper stream closure might be the reason.

HttpStress Run Final Report

[06/25/2021 17:57:10] Total: 110,577 Runtime: 00:11:00
	 0: GET                       Success: 6,861	Canceled: 504	Fail: 8
	 1: GET Partial               Success: 6,867	Canceled: 498	Fail: 7
	 2: GET Headers               Success: 6,788	Canceled: 578	Fail: 7
	 3: GET Parameters            Success: 6,871	Canceled: 500	Fail: 2
	 4: GET Aborted               Success: 0	Canceled: 505	Fail: 6,864
	 5: POST                      Success: 6,836	Canceled: 532	Fail: 2
	 6: POST Multipart Data       Success: 6,841	Canceled: 529	Fail: 1
	 7: POST Duplex               Success: 6,838	Canceled: 532	Fail: 1
	 8: POST Duplex Slow          Success: 6,836	Canceled: 534	Fail: 0
	 9: POST Duplex Dispose       Success: 6,890	Canceled: 479	Fail: 2
	10: POST ExpectContinue       Success: 0	Canceled: 530	Fail: 6,842
	11: HEAD                      Success: 6,887	Canceled: 482	Fail: 4
	12: PUT                       Success: 6,851	Canceled: 519	Fail: 3
	13: PUT Slow                  Success: 6,627	Canceled: 738	Fail: 8
	14: GET Slow                  Success: 6,816	Canceled: 549	Fail: 8
	    TOTAL                     Success: 88,809	Canceled: 8,009	Fail: 13,759

Latency(ms) : n=110577, p50=10.89, p75=17.29, p99=386.3, p999=1002.77, max=60018.42

Of all the scenarios, GET Aborted and POST ExpectContinue are the only ones that are not working at all. Other scenarios failures are timeouts that I mentioned before

Observed failures

  1. System.Exception: Expected status code OK, got InternalServerError
    Scenario: POST ExpectContinue
  2. System.Net.Http.HttpRequestException: An error occurred while sending the request.
    ---> System.Net.Quic.QuicStreamAbortedException: Stream aborted by peer (258).
    Scenario: GET Aborted
  3. System.Net.Http.HttpRequestException: Error while copying content to a stream.
    ---> System.IO.IOException: An error occurred while sending the request.
    ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
    ---> System.Net.Quic.QuicStreamAbortedException: Stream aborted by peer (258).
    Scenario: GET Aborted
  4. System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
    ---> System.TimeoutException: Read was canceled
    Scenarios: GET, GET Partial, GET Headers, GET Parameters
    POST, POST Multipart Data, POST Duplex Dispose, POST ExpectContinue
    HEAD
    PUT, PUT Slow
    GET Slow
  5. System.Exception: Completed unexpectedly
    Scenario: GET Aborted
  6. System.Net.Http.HttpRequestException: Error while copying content to a stream.
    ---> System.IO.IOException: An error occurred while sending the request.
    ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
    ---> System.Net.Http.HttpRequestException: The response ended prematurely, with at least X additional bytes expected.
    Scenario: GET Aborted
  7. System.Net.Http.HttpRequestException: Error while copying content to a stream.
    ---> System.IO.IOException: An error occurred while sending the request.
    ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
    ---> System.Net.Quic.QuicOperationAbortedException: Operation aborted.
    Scenario: POST Duplex

Additional problems

  • Kestrel does not seem to respect MsQuicApi.IsQuicSupported=false (continues with creating QuicListener which fails with NRE)
  • Our System.Net.Quic tracing does not work ("Exception in Command Processing for EventSource "Microsoft-System-Net-Quic": Event VerboseMessage has ID 5 which is already in use.")
  • Event source for System.Net.Quic has unconsistent naming "Microsoft-System-Net-Quic" (changing it to "Private.InternalDiagnostics.System.Net.Quic" doesn't fix the problem above)

cc @ManickaP @wfurt @geoffkizer

@CarnaViire CarnaViire added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Jun 25, 2021
@ghost ghost added the area-System.Net.Http label Jun 25, 2021
@ghost
Copy link

ghost commented Jun 25, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

This is a temporary workaround to run stress tests for HTTP/3 with a pre-built libmsquic.so (until we have proper packaging for Linux).

How to run the tests

Before running, you will need to acquire libmsquic.so and libmsquic.lttng.so and put them to stress dir.
To run the stress for HTTP/3, you need to specify 3.0 on both client and server.

cd ${RUNTIME_DIR}/src/libraries/System.Net.Http/tests/StressTests/HttpStress/
cp ${MSQUIC_LIBS_DIR}/libmsquic.so .
cp ${MSQUIC_LIBS_DIR}/libmsquic.lttng.so .
./run-docker-compose.ps1 -clientstressargs "-maxExecutionTime 10 -http 3.0" -serverstressargs "-http 3.0"

How to get libmsquic.so

I will paste a link to pre-built libs, but if you wish to build it yourself, you can use Dockerfile from msquic repo, run it and then copy the libs from the container to local machine.
It might be helpful to substitute container entrypoint from "/run_endpoint.sh" to "/bin/bash" so the container would not terminate prematurely at the end of the script.
If building docker image on Windows, shell scripts need to be updated from CRLF to LF EOL or they won't work (with a very confusing error "command not found"). From the top of my head it's "scripts/install-powershell-docker.sh", "submodules/openssl/config" and "scripts/run_endpoint.sh".

Leveraging local changes to System.Net.Quic.dll or System.Net.Http.dll

I would strongly advise against building whole clr+libs by passing -b flag to run-docker-compose.ps1 as it takes tons of time. With my updates to Dockerfile, container will already have nightly dotnet build from main. It should be enough to just substitute specific dll -- build it locally for Linux, copy to stress dir and uncomment COPY command. (see Dockerfile)

Results

I haven't run for too long (maxExecutionTime=10 minutes) but it was enough for requests to stop going through and just time out - I'd suspect improper stream closure might be the reason.

HttpStress Run Final Report

[06/25/2021 17:57:10] Total: 110,577 Runtime: 00:11:00
	 0: GET                       Success: 6,861	Canceled: 504	Fail: 8
	 1: GET Partial               Success: 6,867	Canceled: 498	Fail: 7
	 2: GET Headers               Success: 6,788	Canceled: 578	Fail: 7
	 3: GET Parameters            Success: 6,871	Canceled: 500	Fail: 2
	 4: GET Aborted               Success: 0	Canceled: 505	Fail: 6,864
	 5: POST                      Success: 6,836	Canceled: 532	Fail: 2
	 6: POST Multipart Data       Success: 6,841	Canceled: 529	Fail: 1
	 7: POST Duplex               Success: 6,838	Canceled: 532	Fail: 1
	 8: POST Duplex Slow          Success: 6,836	Canceled: 534	Fail: 0
	 9: POST Duplex Dispose       Success: 6,890	Canceled: 479	Fail: 2
	10: POST ExpectContinue       Success: 0	Canceled: 530	Fail: 6,842
	11: HEAD                      Success: 6,887	Canceled: 482	Fail: 4
	12: PUT                       Success: 6,851	Canceled: 519	Fail: 3
	13: PUT Slow                  Success: 6,627	Canceled: 738	Fail: 8
	14: GET Slow                  Success: 6,816	Canceled: 549	Fail: 8
	    TOTAL                     Success: 88,809	Canceled: 8,009	Fail: 13,759

Latency(ms) : n=110577, p50=10.89, p75=17.29, p99=386.3, p999=1002.77, max=60018.42

Of all the scenarios, GET Aborted and POST ExpectContinue are the only ones that are not working at all. Other scenarios failures are timeouts that I mentioned before

Observed failures

  1. System.Exception: Expected status code OK, got InternalServerError
    Scenario: POST ExpectContinue
  2. System.Net.Http.HttpRequestException: An error occurred while sending the request.
    ---> System.Net.Quic.QuicStreamAbortedException: Stream aborted by peer (258).
    Scenario: GET Aborted
  3. System.Net.Http.HttpRequestException: Error while copying content to a stream.
    ---> System.IO.IOException: An error occurred while sending the request.
    ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
    ---> System.Net.Quic.QuicStreamAbortedException: Stream aborted by peer (258).
    Scenario: GET Aborted
  4. System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
    ---> System.TimeoutException: Read was canceled
    Scenarios: GET, GET Partial, GET Headers, GET Parameters
    POST, POST Multipart Data, POST Duplex Dispose, POST ExpectContinue
    HEAD
    PUT, PUT Slow
    GET Slow
  5. System.Exception: Completed unexpectedly
    Scenario: GET Aborted
  6. System.Net.Http.HttpRequestException: Error while copying content to a stream.
    ---> System.IO.IOException: An error occurred while sending the request.
    ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
    ---> System.Net.Http.HttpRequestException: The response ended prematurely, with at least X additional bytes expected.
    Scenario: GET Aborted
  7. System.Net.Http.HttpRequestException: Error while copying content to a stream.
    ---> System.IO.IOException: An error occurred while sending the request.
    ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
    ---> System.Net.Quic.QuicOperationAbortedException: Operation aborted.
    Scenario: POST Duplex

Additional problems

  • Kestrel does not seem to respect MsQuicApi.IsQuicSupported=false (continues with creating QuicListener which fails with NRE)
  • Our System.Net.Quic tracing does not work ("Exception in Command Processing for EventSource "Microsoft-System-Net-Quic": Event VerboseMessage has ID 5 which is already in use.")
  • Event source for System.Net.Quic has unconsistent naming "Microsoft-System-Net-Quic" (changing it to "Private.InternalDiagnostics.System.Net.Quic" doesn't fix the problem above)

cc @ManickaP @wfurt @geoffkizer

Author: CarnaViire
Assignees: -
Labels:

* NO MERGE *, area-System.Net.Http
Milestone: -

@CarnaViire CarnaViire marked this pull request as draft June 25, 2021 20:09
antonfirsov
antonfirsov reviewed Jun 28, 2021
View reviewed changes
src/libraries/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs
public bool IsCancellationRequested { get; private set; }

public Version HttpVersion => _config.HttpVersion;
public Version HttpVersion => _client.DefaultRequestVersion;
Copy link
Contributor

@antonfirsov antonfirsov Jun 28, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this work with the -http config switch? Or am I missing something?

Edit: I see you leave it for the server + ALPN

@ManickaP ManickaP mentioned this pull request Jul 2, 2021
Merged
@CarnaViire
Copy link
Member Author

CarnaViire commented Jul 7, 2021

Closing in favor of #55098

@CarnaViire CarnaViire closed this Jul 7, 2021
@karelz karelz added this to the 6.0.0 milestone Jul 15, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Aug 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

1 more reviewer

@antonfirsov antonfirsov antonfirsov left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area-System.Net.Http NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons)

Projects

None yet

Milestone

6.0.0

Development

Successfully merging this pull request may close these issues.

3 participants

@CarnaViire @antonfirsov @karelz