Skip to content

[Perf] Changes at 5/31/2022 10:14:36 PM #5734

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline c9742b32bae35fd546f6ed844d48ccf195626ba7
Compare 06da0fcf4573115aac76f4afd65cdd75592e86e2
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 150.41 μs 180.32 μs 1.20 0.13 False
Count - Duration of single invocation 146.91 μs 176.54 μs 1.20 0.11 False
Count - Duration of single invocation 4.59 ms 5.78 ms 1.26 0.47 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 180.32438831417625 > 157.98780751352163.
IsChangePoint: Marked as a change because one of 2/8/2022 5:20:12 PM, 4/12/2022 9:30:02 AM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -15.518871169413462 (T) = (0 -184986.3021064966) / Math.Sqrt((16749797.530059392 / (39)) + (18376730.696161773 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.20477332882209567 = (153544.48648639765 - 184986.3021064966) / 153544.48648639765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 176.5434323836276 > 147.39094209542412.
IsChangePoint: Marked as a change because one of 2/8/2022 5:20:12 PM, 4/12/2022 9:30:02 AM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -16.392677592919902 (T) = (0 -173817.50909027885) / Math.Sqrt((12107584.993065977 / (39)) + (16227698.966626214 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.21631264325361793 = (142905.288417721 - 173817.50909027885) / 142905.288417721 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.777950534883722 > 4.893494749759616.
IsChangePoint: Marked as a change because one of 2/8/2022 5:20:12 PM, 5/28/2022 6:51:39 AM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -2.736529522841597 (T) = (0 -7797969.687648497) / Math.Sqrt((1856373071419.701 / (27)) + (11256633414887.352 / (17))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (17) - 2, .025) and -0.42861250182112015 = (5458421.844767601 - 7797969.687648497) / 5458421.844767601 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 18.04
Baseline c9742b32bae35fd546f6ed844d48ccf195626ba7
Compare 06da0fcf4573115aac76f4afd65cdd75592e86e2
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 54.33 ns 60.34 ns 1.11 0.04 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'
Details

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "~", comparisonType: Ordinal)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.337011373575095 > 57.41876366313417.
IsChangePoint: Marked as a change because one of 3/11/2022 5:51:22 PM, 4/22/2022 2:26:33 AM, 5/20/2022 7:31:10 PM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -29.64862707648864 (T) = (0 -60.49308400400296) / Math.Sqrt((1.3728518958571132 / (39)) + (0.040762593689606774 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.11364766139175811 = (54.3197692602371 - 60.49308400400296) / 54.3197692602371 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline c9742b32bae35fd546f6ed844d48ccf195626ba7
Compare 06da0fcf4573115aac76f4afd65cdd75592e86e2
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 1.18 ms 1.48 ms 1.26 0.01 False
Count - Duration of single invocation 1.47 ms 1.60 ms 1.09 0.47 False
Count - Duration of single invocation 1.31 ms 1.88 ms 1.44 0.44 False
Count - Duration of single invocation 26.28 ms 29.17 ms 1.11 0.59 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w.+-]+@[\w.-]+.[\w.-]+", Options: Compiled)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4835637087121212 > 1.2406808564903846.
IsChangePoint: Marked as a change because one of 4/12/2022 9:30:02 AM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -89.3609531826917 (T) = (0 -1491843.7059533317) / Math.Sqrt((47502505.60150411 / (39)) + (52103885.62084537 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.25683681637140054 = (1186982.8179130023 - 1491843.7059533317) / 1186982.8179130023 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w\.+-]+@[\w\.-]+\.[\w\.-]+", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5981918522292993 > 1.5355785065643275.
IsChangePoint: Marked as a change because one of 4/10/2022 3:30:35 PM, 5/31/2022 5:02:26 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -18.866775690769977 (T) = (0 -1598012.0248283988) / Math.Sqrt((6421856273.703958 / (39)) + (3388392.9559549103 / (5))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (5) - 2, .025) and -0.1789845724787084 = (1355413.855389747 - 1598012.0248283988) / 1355413.855389747 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w\.+-]+@[\w\.-]+\.[\w\.-]+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.8776654275689224 > 1.3736638767841323.
IsChangePoint: Marked as a change because one of 4/11/2022 9:20:21 PM, 5/31/2022 10:47:41 AM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -6.957814569876432 (T) = (0 -1716543.757457231) / Math.Sqrt((6641493514.930099 / (38)) + (15382118547.981792 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2692207578733936 = (1352439.0826488975 - 1716543.757457231) / 1352439.0826488975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w]+://[^/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.167125716666664 > 12.2680208896875.
IsChangePoint: Marked as a change because one of 4/21/2022 2:39:24 PM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -7.064529035174161 (T) = (0 -46153694.65438597) / Math.Sqrt((55279300831084.75 / (25)) + (272950646692545.34 / (19))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (19) - 2, .025) and -1.6539501255903228 = (17390565.93767749 - 46153694.65438597) / 17390565.93767749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions