-
Notifications
You must be signed in to change notification settings - Fork 3
Closed
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | c9742b32bae35fd546f6ed844d48ccf195626ba7 |
| Compare | 06da0fcf4573115aac76f4afd65cdd75592e86e2 |
| Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 150.41 μs | 180.32 μs | 1.20 | 0.13 | False | |||||
| Count - Duration of single invocation | 146.91 μs | 176.54 μs | 1.20 | 0.11 | False | |||||
| Count - Duration of single invocation | 4.59 ms | 5.78 ms | 1.26 | 0.47 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 180.32438831417625 > 157.98780751352163.
IsChangePoint: Marked as a change because one of 2/8/2022 5:20:12 PM, 4/12/2022 9:30:02 AM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -15.518871169413462 (T) = (0 -184986.3021064966) / Math.Sqrt((16749797.530059392 / (39)) + (18376730.696161773 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.20477332882209567 = (153544.48648639765 - 184986.3021064966) / 153544.48648639765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 176.5434323836276 > 147.39094209542412.
IsChangePoint: Marked as a change because one of 2/8/2022 5:20:12 PM, 4/12/2022 9:30:02 AM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -16.392677592919902 (T) = (0 -173817.50909027885) / Math.Sqrt((12107584.993065977 / (39)) + (16227698.966626214 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.21631264325361793 = (142905.288417721 - 173817.50909027885) / 142905.288417721 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.777950534883722 > 4.893494749759616.
IsChangePoint: Marked as a change because one of 2/8/2022 5:20:12 PM, 5/28/2022 6:51:39 AM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -2.736529522841597 (T) = (0 -7797969.687648497) / Math.Sqrt((1856373071419.701 / (27)) + (11256633414887.352 / (17))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (17) - 2, .025) and -0.42861250182112015 = (5458421.844767601 - 7797969.687648497) / 5458421.844767601 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | c9742b32bae35fd546f6ed844d48ccf195626ba7 |
| Compare | 06da0fcf4573115aac76f4afd65cdd75592e86e2 |
| Diff | Diff |
Regressions in System.Memory.ReadOnlySpan
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IndexOfString - Duration of single invocation | 54.33 ns | 60.34 ns | 1.11 | 0.04 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'Details
Payloads
Histogram
System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "~", comparisonType: Ordinal)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.337011373575095 > 57.41876366313417.
IsChangePoint: Marked as a change because one of 3/11/2022 5:51:22 PM, 4/22/2022 2:26:33 AM, 5/20/2022 7:31:10 PM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -29.64862707648864 (T) = (0 -60.49308400400296) / Math.Sqrt((1.3728518958571132 / (39)) + (0.040762593689606774 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.11364766139175811 = (54.3197692602371 - 60.49308400400296) / 54.3197692602371 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | c9742b32bae35fd546f6ed844d48ccf195626ba7 |
| Compare | 06da0fcf4573115aac76f4afd65cdd75592e86e2 |
| Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 1.18 ms | 1.48 ms | 1.26 | 0.01 | False | |||||
| Count - Duration of single invocation | 1.47 ms | 1.60 ms | 1.09 | 0.47 | False | |||||
| Count - Duration of single invocation | 1.31 ms | 1.88 ms | 1.44 | 0.44 | False | |||||
| Count - Duration of single invocation | 26.28 ms | 29.17 ms | 1.11 | 0.59 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w.+-]+@[\w.-]+.[\w.-]+", Options: Compiled)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4835637087121212 > 1.2406808564903846.
IsChangePoint: Marked as a change because one of 4/12/2022 9:30:02 AM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -89.3609531826917 (T) = (0 -1491843.7059533317) / Math.Sqrt((47502505.60150411 / (39)) + (52103885.62084537 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.25683681637140054 = (1186982.8179130023 - 1491843.7059533317) / 1186982.8179130023 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w\.+-]+@[\w\.-]+\.[\w\.-]+", Options: None)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5981918522292993 > 1.5355785065643275.
IsChangePoint: Marked as a change because one of 4/10/2022 3:30:35 PM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -18.866775690769977 (T) = (0 -1598012.0248283988) / Math.Sqrt((6421856273.703958 / (39)) + (3388392.9559549103 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.1789845724787084 = (1355413.855389747 - 1598012.0248283988) / 1355413.855389747 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w\.+-]+@[\w\.-]+\.[\w\.-]+", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.8776654275689224 > 1.3736638767841323.
IsChangePoint: Marked as a change because one of 4/11/2022 9:20:21 PM, 5/31/2022 10:47:41 AM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -6.957814569876432 (T) = (0 -1716543.757457231) / Math.Sqrt((6641493514.930099 / (38)) + (15382118547.981792 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2692207578733936 = (1352439.0826488975 - 1716543.757457231) / 1352439.0826488975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w]+://[^/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.167125716666664 > 12.2680208896875.
IsChangePoint: Marked as a change because one of 4/21/2022 2:39:24 PM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -7.064529035174161 (T) = (0 -46153694.65438597) / Math.Sqrt((55279300831084.75 / (25)) + (272950646692545.34 / (19))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (19) - 2, .025) and -1.6539501255903228 = (17390565.93767749 - 46153694.65438597) / 17390565.93767749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


