Benchmark test for GetFullPath with redundant segments #1394

carlossanlop · 2020-07-11T04:20:53Z

Adding a benchmark test to compare results of calling Path.GetFullPath using paths that contain redundant segments.

Related to: Add APIs that remove redundant segments from paths #37939

Background

The GetFullPath currently calls an internal method that does not handle all Windows edge cases. Now that we want to make this method public, those edge cases need to be handled, and require adding extra functionality that will affect the original performance and allocations.

The Unix code also changed a bit.

Results

UPDATE: I am posting new results to match the suggested changes in this PR (49a3ee5) and in the runtime PR (dotnet/runtime@3b72390).

I added two benchmarks with two long paths. One has no redundant segments, the other does.

Code

private string _testPathNoRedundantSegments;
private string _testPathWithRedundantSegments;

[GlobalSetup]
public void SetupPaths()
{
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
    // This fully qualified path will be normalized by the Windows P/Invoke
    _testPathNoRedundantSegments = @"C:\repos\runtime\src\coreclr\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Path.cs";
    // This unqualified path will be analyzed by our RedundantSegments.Windows code
    _testPathWithRedundantSegments = @"runtime\src\coreclr\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Extra\..\Path.cs";
}
else
{
    // Both paths will be analyzed by our RedundantSegments.Unix code
    _testPathNoRedundantSegments = "/home/user/runtime/src/coreclr/runtime/src/libraries/System.Private.CoreLib/src/System/IO/Path.cs";
    _testPathWithRedundantSegments = "runtime/src/coreclr/runtime/src/libraries/System.Private.CoreLib/src/System/IO/Extra/../Path.cs";
}
}

[Benchmark]
public void GetFullPathNoRedundantSegments() => Path.GetFullPath(_testPathNoRedundantSegments);

[Benchmark]
public void GetFullPathWithRedundantSegments() => Path.GetFullPath(_testPathWithRedundantSegments);

Ubuntu benchmarks

Results before

root@calopepc:/home/carlos/performance# python3 ./scripts/benchmarks_ci.py -f net6.0 --bdn-arguments="--artifacts /home/carlos/perf_before --coreRun /home/carlos/runtime/artifacts/bin/testhost/net6.0-Linux-Release-x64/shared/Microsoft.NETCore.App/6.0.0/corerun" --filter System.IO.Tests.Perf_Path.GetFullPath*
...
[2021/01/27 10:11:03][INFO] |                           Method |       Mean |    Error |   StdDev |     Median |        Min |        Max |  Gen 0 | Gen 1 | Gen 2 | Allocated |
[2021/01/27 10:11:03][INFO] |--------------------------------- |-----------:|---------:|---------:|-----------:|-----------:|-----------:|-------:|------:|------:|----------:|
[2021/01/27 10:11:03][INFO] |       GetFullPathForLegacyLength |   747.0 ns | 10.33 ns |  9.66 ns |   743.9 ns |   736.3 ns |   771.5 ns |      - |     - |     - |         - |
[2021/01/27 10:11:03][INFO] |    GetFullPathForTypicalLongPath | 1,817.9 ns | 27.18 ns | 25.43 ns | 1,810.0 ns | 1,784.8 ns | 1,854.2 ns |      - |     - |     - |         - |
[2021/01/27 10:11:03][INFO] |     GetFullPathForReallyLongPath | 3,554.0 ns | 56.20 ns | 52.57 ns | 3,526.2 ns | 3,492.5 ns | 3,669.3 ns |      - |     - |     - |         - |
[2021/01/27 10:11:03][INFO] |   GetFullPathNoRedundantSegments |   446.0 ns | 11.21 ns | 12.91 ns |   443.1 ns |   428.7 ns |   469.5 ns |      - |     - |     - |         - |
[2021/01/27 10:11:03][INFO] | GetFullPathWithRedundantSegments | 1,830.0 ns | 23.48 ns | 21.97 ns | 1,837.9 ns | 1,784.3 ns | 1,851.9 ns | 0.1914 |     - |     - |   1,232 B |

Results after

root@calopepc:/home/carlos/performance# python3 ./scripts/benchmarks_ci.py -f net6.0 --bdn-arguments="--artifacts /home/carlos/perf_after --coreRun /home/carlos/runtime/artifacts/bin/testhost/net6.0-Linux-Release-x64/shared/Microsoft.NETCore.App/6.0.0/corerun" --filter System.IO.Tests.Perf_Path.GetFullPath*
...
[2021/01/27 10:15:29][INFO] |                           Method |       Mean |    Error |   StdDev |     Median |        Min |        Max |  Gen 0 | Gen 1 | Gen 2 | Allocated |
[2021/01/27 10:15:29][INFO] |--------------------------------- |-----------:|---------:|---------:|-----------:|-----------:|-----------:|-------:|------:|------:|----------:|
[2021/01/27 10:15:29][INFO] |       GetFullPathForLegacyLength |   742.7 ns |  7.30 ns |  6.47 ns |   743.7 ns |   730.9 ns |   751.4 ns |      - |     - |     - |         - |
[2021/01/27 10:15:29][INFO] |    GetFullPathForTypicalLongPath | 1,813.8 ns | 33.48 ns | 27.96 ns | 1,810.8 ns | 1,762.9 ns | 1,876.0 ns |      - |     - |     - |         - |
[2021/01/27 10:15:29][INFO] |     GetFullPathForReallyLongPath | 3,624.2 ns | 41.87 ns | 39.17 ns | 3,629.2 ns | 3,543.8 ns | 3,672.0 ns |      - |     - |     - |         - |
[2021/01/27 10:15:29][INFO] |   GetFullPathNoRedundantSegments |   457.5 ns |  6.51 ns |  6.09 ns |   459.0 ns |   440.8 ns |   465.2 ns |      - |     - |     - |         - |
[2021/01/27 10:15:29][INFO] | GetFullPathWithRedundantSegments | 1,824.1 ns | 14.29 ns | 13.36 ns | 1,822.9 ns | 1,802.6 ns | 1,843.1 ns | 0.1903 |     - |     - |   1,232 B |

Results comparison

carlos@calopepc:~/performance/src/tools/ResultsComparer$ ~/runtime/.dotnet/dotnet run -c release --base ~/perf_before --diff ~/perf_after --threshold 0.0000000000001%
summary:
worse: 2, geomean: 1.032
total diff: 2

| Slower                                                   | diff/base | Base Median (ns) | Diff Median (ns) | Modality|
| -------------------------------------------------------- | ---------:| ----------------:| ----------------:| --------:|
| System.IO.Tests.Perf_Path.GetFullPathNoRedundantSegments |      1.04 |           443.14 |           458.98 |         |
| System.IO.Tests.Perf_Path.GetFullPathForReallyLongPath   |      1.03 |          3526.19 |          3629.22 |         |

No Faster results for the provided threshold = 0.0000000000001% and noise filter = 0.3ns.

Windows benchmarks

Results before

carlos@CALOPEPC> D:\performance> | RRS>
py .\scripts\benchmarks_ci.py -f net6.0 --bdn-arguments="--artifacts D:\perf_before\ --coreRun D:\runtime\artifacts\bin\testhost\net6.0-windows-Release-x64\shared\Microsoft.NETCore.App\6.0.0\CoreRun.exe" --filter System.IO.Tests.Perf_Path.GetFullPath*

[2021/01/27 10:57:52][INFO] |--------------------------------- |-----------:|---------:|---------:|-----------:|-----------:|-----------:|-------:|------:|------:|----------:|
[2021/01/27 10:57:52][INFO] |       GetFullPathForLegacyLength |   409.9 ns |  4.67 ns |  4.14 ns |   408.9 ns |   404.7 ns |   418.6 ns | 0.0674 |     - |     - |     432 B |
[2021/01/27 10:57:52][INFO] |    GetFullPathForTypicalLongPath | 1,065.5 ns |  7.34 ns |  6.86 ns | 1,064.8 ns | 1,053.2 ns | 1,078.8 ns | 0.1604 |     - |     - |   1,032 B |
[2021/01/27 10:57:52][INFO] |     GetFullPathForReallyLongPath | 1,841.4 ns | 21.94 ns | 20.53 ns | 1,830.8 ns | 1,816.6 ns | 1,878.9 ns | 0.3171 |     - |     - |   2,032 B |
[2021/01/27 10:57:52][INFO] |   GetFullPathNoRedundantSegments |   227.8 ns |  1.30 ns |  1.08 ns |   227.3 ns |   226.6 ns |   229.6 ns |      - |     - |     - |         - |
[2021/01/27 10:57:52][INFO] | GetFullPathWithRedundantSegments |   430.3 ns |  2.39 ns |  2.12 ns |   429.9 ns |   426.7 ns |   435.3 ns | 0.0711 |     - |     - |     448 B |

Results after

carlos@CALOPEPC> D:\performance> | RRS>
py .\scripts\benchmarks_ci.py -f net6.0 --bdn-arguments="--artifacts D:\perf_after\ --coreRun D:\runtime\artifacts\bin\testhost\net6.0-windows-Release-x64\shared\Microsoft.NETCore.App\6.0.0\CoreRun.exe" --filter System.IO.Tests.Perf_Path.GetFullPath*

[2021/01/27 11:04:36][INFO] |                           Method |       Mean |    Error |   StdDev |     Median |        Min |        Max |  Gen 0 | Gen 1 | Gen 2 | Allocated |
[2021/01/27 11:04:36][INFO] |--------------------------------- |-----------:|---------:|---------:|-----------:|-----------:|-----------:|-------:|------:|------:|----------:|
[2021/01/27 11:04:36][INFO] |       GetFullPathForLegacyLength |   402.0 ns |  3.03 ns |  2.83 ns |   402.4 ns |   397.4 ns |   406.3 ns | 0.0675 |     - |     - |     432 B |
[2021/01/27 11:04:36][INFO] |    GetFullPathForTypicalLongPath | 1,154.6 ns |  7.00 ns |  5.84 ns | 1,156.3 ns | 1,137.0 ns | 1,160.6 ns | 0.1622 |     - |     - |   1,032 B |
[2021/01/27 11:04:36][INFO] |     GetFullPathForReallyLongPath | 1,902.3 ns | 21.47 ns | 19.04 ns | 1,902.5 ns | 1,871.9 ns | 1,932.5 ns | 0.3177 |     - |     - |   2,032 B |
[2021/01/27 11:04:36][INFO] |   GetFullPathNoRedundantSegments |   225.0 ns |  1.82 ns |  1.61 ns |   224.9 ns |   222.1 ns |   228.1 ns |      - |     - |     - |         - |
[2021/01/27 11:04:36][INFO] | GetFullPathWithRedundantSegments |   502.4 ns |  3.55 ns |  3.15 ns |   502.7 ns |   495.5 ns |   508.1 ns | 0.0708 |     - |     - |     448 B |

Results comparison

carlos@CALOPEPC> D:\performance\src\tools\ResultsComparer> | RRS>
dotnet run -c release --base D:\perf_before\ --diff D:\perf_after\ --threshold 0.000000000001%

summary:
better: 2, geomean: 1.013
worse: 3, geomean: 1.097
total diff: 5

| Slower                                                     | diff/base | Base Median (ns) | Diff Median (ns) | Modality|
| ---------------------------------------------------------- | ---------:| ----------------:| ----------------:| --------:|
| System.IO.Tests.Perf_Path.GetFullPathWithRedundantSegments |      1.17 |           429.90 |           502.69 |         |
| System.IO.Tests.Perf_Path.GetFullPathForTypicalLongPath    |      1.09 |          1064.83 |          1156.26 |         |
| System.IO.Tests.Perf_Path.GetFullPathForReallyLongPath     |      1.04 |          1830.82 |          1902.45 |         |

| Faster                                                   | base/diff | Base Median (ns) | Diff Median (ns) | Modality|
| -------------------------------------------------------- | ---------:| ----------------:| ----------------:| --------:|
| System.IO.Tests.Perf_Path.GetFullPathForLegacyLength     |      1.02 |           408.91 |           402.42 |         |
| System.IO.Tests.Perf_Path.GetFullPathNoRedundantSegments |      1.01 |           227.28 |           224.94 |         |

src/benchmarks/micro/libraries/System.Runtime.Extensions/Perf.Path.cs

billwert · 2020-07-13T18:01:27Z

I think this test should have more typical cases compared to these edge cases. (Or at least include them.) I suspect ..\.\..\.\.\folder1\.\toremove\.\.\..\folder2\.\toremove\toremove\..\.\.\..\.\folder3\.\.\....\.\.\..\\. is not a very common path.

billwert · 2020-07-13T18:05:16Z

(And I know we already have a few other tests, but its not clear to me that these give a ton of benefit over those.)

billwert · 2020-07-14T22:11:23Z

One other thought about these tests. It looks like the expected improvement is ~2%, right? In general our system isn't going to flag that as a regression if we lost this optimization somehow. That's typically below the noise threshold we hold. @DrewScoggins can opine some here.

carlossanlop · 2021-01-26T23:58:35Z

@adamsitnik @jozkee @jkotas I ran these benchmarks, and they are giving me a different result than what we saw with the Stopwatch test in dotnet/runtime#37939 (comment) . Would you mind taking a look? (Results in the description)

carlossanlop · 2021-01-27T00:41:20Z

Updated the description to include Windows benchmark tests too.

jozkee · 2021-01-27T17:01:48Z

src/benchmarks/micro/libraries/System.Runtime.Extensions/Perf.Path.cs

        private readonly string _testPath500 = PerfUtils.CreateString(500);
        private readonly string _testPath1000 = PerfUtils.CreateString(1000);
+        private readonly string _testPathNoRedundantSegments = "/home/user/runtime/src/coreclr/runtime/src/libraries/System.Private.CoreLib/src/System/IO/Path.cs";
+        private readonly string _testPathWithRedundantSegments = "/home/user/runtime/src/coreclr/runtime/src/libraries/System.Private.CoreLib/src/System/IO/..//./Path.cs";


Did you consider adding rooted paths like C:\ProgramData and paths with flipped separators?

I had a conversation with Adam a few minutes ago, we touched the topic of Windows, precisely.
I added a new commit, can you please take a look? It has comments describing the purpose of each new path.

What about the flipped separators? It looks to me like a common case that is worth checking for regression, at least on windows.

I really doubt flipped separators will cause a problem. We were calling Path.IsDirectorySeparator before, and we are still calling that (see RedundantSegmentsHelper.Unix.cs in my PR).

In Windows, both \ and / are considered valid separators.
In Linux, only / is a valid separator.

adamsitnik

LGTM, thank you @carlossanlop !

carlossanlop mentioned this pull request Jul 11, 2020

Add APIs that remove redundant segments from paths dotnet/runtime#37939

Closed

adamsitnik reviewed Jul 13, 2020

View reviewed changes

src/benchmarks/micro/libraries/System.Runtime.Extensions/Perf.Path.cs Outdated Show resolved Hide resolved

carlossanlop and others added 2 commits January 26, 2021 15:01

Benchmark test for GetFullPath that has no redundant segments

78c2cd9

Add test for path with redundant segments

49a3ee5

jozkee reviewed Jan 27, 2021

View reviewed changes

Different paths between Windows and Unix

d04b315

adamsitnik approved these changes Jan 27, 2021

View reviewed changes

adamsitnik merged commit 1894120 into dotnet:master Jan 27, 2021

carlossanlop deleted the RRS branch January 27, 2021 19:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Benchmark test for GetFullPath with redundant segments #1394

Benchmark test for GetFullPath with redundant segments #1394

Uh oh!

carlossanlop commented Jul 11, 2020 •

edited

Loading

Uh oh!

Uh oh!

billwert commented Jul 13, 2020

Uh oh!

billwert commented Jul 13, 2020

Uh oh!

billwert commented Jul 14, 2020

Uh oh!

carlossanlop commented Jan 26, 2021 •

edited

Loading

Uh oh!

carlossanlop commented Jan 27, 2021

Uh oh!

jozkee Jan 27, 2021

Uh oh!

carlossanlop Jan 27, 2021

Uh oh!

jozkee Jan 27, 2021

Uh oh!

carlossanlop Jan 27, 2021 •

edited

Loading

Uh oh!

adamsitnik left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Benchmark test for GetFullPath with redundant segments #1394

Benchmark test for GetFullPath with redundant segments #1394

Uh oh!

Conversation

carlossanlop commented Jul 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Results

Results before

Results after

Results comparison

Results before

Results after

Results comparison

Uh oh!

Uh oh!

billwert commented Jul 13, 2020

Uh oh!

billwert commented Jul 13, 2020

Uh oh!

billwert commented Jul 14, 2020

Uh oh!

carlossanlop commented Jan 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlossanlop commented Jan 27, 2021

Uh oh!

jozkee Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

carlossanlop Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

jozkee Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

carlossanlop Jan 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

carlossanlop commented Jul 11, 2020 •

edited

Loading

carlossanlop commented Jan 26, 2021 •

edited

Loading

carlossanlop Jan 27, 2021 •

edited

Loading