diff --git a/README.md b/README.md index 7c66388d4a..5c8d78618d 100644 --- a/README.md +++ b/README.md @@ -289,12 +289,12 @@ Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Mean | Error | StdDev | -|------------- |---------:|---------:|---------:| -| Build_TUnit | 981.9 ms | 19.54 ms | 37.64 ms | -| Build_NUnit | 794.8 ms | 12.52 ms | 10.45 ms | -| Build_xUnit | 785.2 ms | 8.75 ms | 7.30 ms | -| Build_MSTest | 838.8 ms | 11.85 ms | 9.90 ms | +| Method | Mean | Error | StdDev | Median | +|------------- |-----------:|---------:|----------:|-----------:| +| Build_TUnit | 1,160.1 ms | 63.92 ms | 183.40 ms | 1,089.8 ms | +| Build_NUnit | 789.3 ms | 10.36 ms | 8.65 ms | 789.9 ms | +| Build_xUnit | 829.0 ms | 16.51 ms | 28.91 ms | 824.6 ms | +| Build_MSTest | 842.7 ms | 16.83 ms | 42.83 ms | 831.1 ms | @@ -313,10 +313,10 @@ Job=.NET 9.0 Runtime=.NET 9.0 ``` | Method | Mean | Error | StdDev | |------------- |--------:|---------:|---------:| -| Build_TUnit | 1.899 s | 0.0209 s | 0.0174 s | -| Build_NUnit | 1.486 s | 0.0216 s | 0.0192 s | -| Build_xUnit | 1.492 s | 0.0121 s | 0.0113 s | -| Build_MSTest | 1.511 s | 0.0107 s | 0.0100 s | +| Build_TUnit | 1.911 s | 0.0349 s | 0.0343 s | +| Build_NUnit | 1.481 s | 0.0173 s | 0.0153 s | +| Build_xUnit | 1.514 s | 0.0215 s | 0.0201 s | +| Build_MSTest | 1.514 s | 0.0151 s | 0.0141 s | @@ -335,10 +335,10 @@ Job=.NET 9.0 Runtime=.NET 9.0 ``` | Method | Mean | Error | StdDev | |------------- |--------:|---------:|---------:| -| Build_TUnit | 1.887 s | 0.0374 s | 0.0350 s | -| Build_NUnit | 1.476 s | 0.0238 s | 0.0223 s | -| Build_xUnit | 1.472 s | 0.0237 s | 0.0210 s | -| Build_MSTest | 1.521 s | 0.0152 s | 0.0143 s | +| Build_TUnit | 1.914 s | 0.0380 s | 0.0568 s | +| Build_NUnit | 1.485 s | 0.0254 s | 0.0226 s | +| Build_xUnit | 1.486 s | 0.0121 s | 0.0101 s | +| Build_MSTest | 1.541 s | 0.0131 s | 0.0116 s | ### Scenario: A single test that completes instantly (including spawning a new process and initialising the test framework) @@ -356,13 +356,13 @@ Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Mean | Error | StdDev | -|---------- |----------:|---------:|---------:| -| TUnit_AOT | 75.67 ms | 1.481 ms | 1.455 ms | -| TUnit | 479.63 ms | 5.521 ms | 4.894 ms | -| NUnit | 699.32 ms | 8.003 ms | 7.095 ms | -| xUnit | 723.38 ms | 6.808 ms | 6.035 ms | -| MSTest | 628.77 ms | 7.284 ms | 6.457 ms | +| Method | Mean | Error | StdDev | +|---------- |----------:|----------:|----------:| +| TUnit_AOT | 69.20 ms | 0.754 ms | 0.589 ms | +| TUnit | 491.47 ms | 9.787 ms | 18.620 ms | +| NUnit | 703.36 ms | 9.870 ms | 7.706 ms | +| xUnit | 725.22 ms | 7.797 ms | 6.912 ms | +| MSTest | 623.46 ms | 12.439 ms | 11.026 ms | @@ -381,11 +381,11 @@ Job=.NET 9.0 Runtime=.NET 9.0 ``` | Method | Mean | Error | StdDev | |---------- |------------:|----------:|----------:| -| TUnit_AOT | 24.69 ms | 0.491 ms | 1.233 ms | -| TUnit | 829.43 ms | 16.315 ms | 26.346 ms | -| NUnit | 1,286.61 ms | 10.981 ms | 10.272 ms | -| xUnit | 1,341.42 ms | 16.429 ms | 15.367 ms | -| MSTest | 1,144.21 ms | 8.782 ms | 8.215 ms | +| TUnit_AOT | 24.11 ms | 0.479 ms | 1.319 ms | +| TUnit | 826.41 ms | 16.113 ms | 18.555 ms | +| NUnit | 1,290.46 ms | 9.301 ms | 8.700 ms | +| xUnit | 1,347.28 ms | 9.441 ms | 7.883 ms | +| MSTest | 1,148.47 ms | 12.201 ms | 11.413 ms | @@ -402,13 +402,13 @@ AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Mean | Error | StdDev | -|---------- |------------:|----------:|----------:| -| TUnit_AOT | 57.57 ms | 2.075 ms | 6.119 ms | -| TUnit | 869.51 ms | 16.896 ms | 25.289 ms | -| NUnit | 1,311.68 ms | 8.231 ms | 7.296 ms | -| xUnit | 1,366.04 ms | 20.997 ms | 19.641 ms | -| MSTest | 1,169.46 ms | 8.864 ms | 7.402 ms | +| Method | Mean | Error | StdDev | Median | +|---------- |------------:|----------:|----------:|------------:| +| TUnit_AOT | 55.51 ms | 1.773 ms | 5.173 ms | 53.87 ms | +| TUnit | 868.51 ms | 16.877 ms | 26.768 ms | 870.72 ms | +| NUnit | 1,337.78 ms | 19.672 ms | 18.401 ms | 1,334.68 ms | +| xUnit | 1,376.66 ms | 14.994 ms | 13.292 ms | 1,374.15 ms | +| MSTest | 1,181.27 ms | 12.312 ms | 11.516 ms | 1,177.15 ms | ### Scenario: A test that takes 50ms to execute, repeated 100 times (including spawning a new process and initialising the test framework) @@ -428,11 +428,11 @@ Job=.NET 9.0 Runtime=.NET 9.0 ``` | Method | Mean | Error | StdDev | |---------- |------------:|----------:|----------:| -| TUnit_AOT | 246.7 ms | 14.07 ms | 41.49 ms | -| TUnit | 738.7 ms | 31.08 ms | 91.64 ms | -| NUnit | 14,125.2 ms | 276.86 ms | 484.90 ms | -| xUnit | 14,574.1 ms | 290.60 ms | 573.61 ms | -| MSTest | 14,539.4 ms | 285.42 ms | 543.04 ms | +| TUnit_AOT | 228.0 ms | 11.88 ms | 35.04 ms | +| TUnit | 642.0 ms | 19.77 ms | 58.28 ms | +| NUnit | 14,360.7 ms | 283.06 ms | 584.57 ms | +| xUnit | 14,502.6 ms | 287.08 ms | 532.12 ms | +| MSTest | 14,499.4 ms | 285.50 ms | 570.18 ms | @@ -451,11 +451,11 @@ Job=.NET 9.0 Runtime=.NET 9.0 ``` | Method | Mean | Error | StdDev | |---------- |------------:|----------:|----------:| -| TUnit_AOT | 73.59 ms | 0.377 ms | 0.294 ms | -| TUnit | 895.14 ms | 17.563 ms | 17.250 ms | -| NUnit | 6,287.98 ms | 12.844 ms | 12.014 ms | -| xUnit | 6,427.61 ms | 14.939 ms | 13.974 ms | -| MSTest | 6,260.22 ms | 9.162 ms | 8.122 ms | +| TUnit_AOT | 74.01 ms | 0.711 ms | 0.594 ms | +| TUnit | 891.52 ms | 17.564 ms | 18.793 ms | +| NUnit | 6,287.51 ms | 12.436 ms | 11.025 ms | +| xUnit | 6,454.07 ms | 19.839 ms | 17.586 ms | +| MSTest | 6,272.87 ms | 25.795 ms | 24.129 ms | @@ -474,11 +474,11 @@ Job=.NET 9.0 Runtime=.NET 9.0 ``` | Method | Mean | Error | StdDev | |---------- |-----------:|---------:|---------:| -| TUnit_AOT | 110.4 ms | 2.06 ms | 1.93 ms | -| TUnit | 930.8 ms | 18.31 ms | 19.59 ms | -| NUnit | 7,499.7 ms | 22.01 ms | 20.59 ms | -| xUnit | 7,557.0 ms | 17.44 ms | 16.31 ms | -| MSTest | 7,437.2 ms | 23.13 ms | 21.64 ms | +| TUnit_AOT | 109.0 ms | 0.49 ms | 0.41 ms | +| TUnit | 956.1 ms | 18.97 ms | 24.67 ms | +| NUnit | 7,509.9 ms | 10.56 ms | 9.36 ms | +| xUnit | 7,568.1 ms | 14.17 ms | 12.57 ms | +| MSTest | 7,468.6 ms | 24.46 ms | 22.88 ms |