Add MonoAOTLLVM to local script #3338

LoopedBard3 · 2023-09-12T20:24:40Z

Adds MonoAOTLLVM scenario support to the local testing script. Because the MonoAOTLLVM toolchain is only setup to run one BDN test set per invocation, the tool runs multiple which then can be compared using the ResultsComparer tool.

scripts/benchmarks_local.py

matouskozak · 2023-09-15T16:23:30Z

Tested locally on osx-arm64 and I'm getting. It looks similar to dotnet/runtime#91819. Is it possible that it is a Mono issue?

[2023/09/15 15:53:45][INFO] 
[2023/09/15 15:53:45][INFO]   Mono Warning: --llvm not enabled in this runtime.
[2023/09/15 15:53:45][INFO]   Mono Ahead of Time compiler - compiling assembly /Users/matouskozk/work/repos/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-SWNCKQ/bin/net8.0/osx-arm64/publish/System.Private.CoreLib.dll
[2023/09/15 15:53:45][INFO]   AOTID 7F041A4C-D241-DDA9-C9C5-07C9162C2327
[2023/09/15 15:53:45][INFO]   * Assertion at /Users/matouskozk/work/repos/baseline_runtime/runtime/src/mono/mono/mini/mini-arm64.c:3915, condition `mono_class_value_size (ins->klass, NULL) == 16' not met
``

matouskozak · 2023-09-16T16:17:28Z

Tested locally on osx-arm64 and I'm getting. It looks similar to dotnet/runtime#91819. Is it possible that it is a Mono issue?

[2023/09/15 15:53:45][INFO] 
[2023/09/15 15:53:45][INFO]   Mono Warning: --llvm not enabled in this runtime.
[2023/09/15 15:53:45][INFO]   Mono Ahead of Time compiler - compiling assembly /Users/matouskozk/work/repos/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-SWNCKQ/bin/net8.0/osx-arm64/publish/System.Private.CoreLib.dll
[2023/09/15 15:53:45][INFO]   AOTID 7F041A4C-D241-DDA9-C9C5-07C9162C2327
[2023/09/15 15:53:45][INFO]   * Assertion at /Users/matouskozk/work/repos/baseline_runtime/runtime/src/mono/mono/mini/mini-arm64.c:3915, condition `mono_class_value_size (ins->klass, NULL) == 16' not met
``

I tested the script on Ubuntu 22.04 x64 (WSL) and it looks to run correctly.

Edit: Upon more detailed inspection I believe that while the script runs with Ubuntu on x64 it doesn't use LLVM backend but mini.

matouskozak · 2023-09-18T13:34:25Z

Tested locally on osx-arm64 and I'm getting. It looks similar to dotnet/runtime#91819. Is it possible that it is a Mono issue?

[2023/09/15 15:53:45][INFO] 
[2023/09/15 15:53:45][INFO]   Mono Warning: --llvm not enabled in this runtime.
[2023/09/15 15:53:45][INFO]   Mono Ahead of Time compiler - compiling assembly /Users/matouskozk/work/repos/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-SWNCKQ/bin/net8.0/osx-arm64/publish/System.Private.CoreLib.dll
[2023/09/15 15:53:45][INFO]   AOTID 7F041A4C-D241-DDA9-C9C5-07C9162C2327
[2023/09/15 15:53:45][INFO]   * Assertion at /Users/matouskozk/work/repos/baseline_runtime/runtime/src/mono/mono/mini/mini-arm64.c:3915, condition `mono_class_value_size (ins->klass, NULL) == 16' not met
``

I re-tested the script on osx-arm64 with clang-16 and I'm getting the same error. It looks like it built without enabled LLVM, report from mono-aot-cross --llvm --version:

Mono Warning: --llvm not enabled in this runtime.
Mono JIT compiler version 9.0.0.0 (42.42.42.42424)
Copyright (C) Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           
        SIGSEGV:       normal
        Architecture:  arm64
        Misc:          softdebug 
        Interpreter:   yes
        LLVM:          supported, not enabled.
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

scripts/benchmarks_local.py

matouskozak · 2023-09-22T14:40:49Z

After the last change which added /p:MonoAOTEnableLLVM=true and /p:MonoBundleLLVMOptimizer=true flags for arm64. I can run MonoAOTLLVM on osx-arm64 machine and mono-aot-cross --llvm --version reports:

Mono JIT compiler version 9.0.0.0 (42.42.42.42424)
Copyright (C) Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           
        SIGSEGV:       normal
        Architecture:  arm64
        Misc:          softdebug 
        Interpreter:   yes
        LLVM:          yes(1600)
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

However, I'm concerned about the Ubuntu scenario where mono-aot-cross --llvm --version reports:

Mono Warning: --llvm not enabled in this runtime.
Mono JIT compiler version 9.0.0.0 (42.42.42.42424)
Copyright (C) Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:
        SIGSEGV:       normal
        Architecture:  amd64
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          supported, not enabled.
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

and based on some testing I believe the AOT is running with mini not llvm backend. Any thoughts on what might be missing? Could it be related to recent issues with LLVM on Ubuntu? I'm unsure whether this is a x64 vs arm64 or Linux vs Mac issue with MonoAOTLLVM (I suspect the latter).
Edit: I manage to get llvm enabled on Ubuntu by adding /p:MonoEnableLLVM=true and /p:MonoAOTEnableLLVM=true build flags. (/p:MonoBundleLLVMOptimizer=true seems to be optional in this scenario)

How do we test that the machines running in perf lab are doing AOT-llvm and not using mini? Could we verify by using mono-aot-cross --llvm --version on the lab machine (or other way) that we actually do AOT-llvm? @kotlarmilos

matouskozak · 2023-09-22T14:45:36Z

Not related to this PR but I found that when running MonoJIT with --commits flag the build fails on building bash build.sh -subset libs.pretest -configuration Release -os osx -arch arm64 -bl -testscope innerloop /p:RuntimeFlavor=mono /p:RuntimeArtifactsPath=./runtime/artifacts/bin/mono/osx.arm64.Release

The error is

/Users/matouskozk/work/repos/performance/scripts/runtime/eng/liveBuilds.targets(63,5): error : The Mono artifacts path does not exist '/Users/matouskozk/work/repos/performance/scripts/runtime/eng/runtime/artifacts/bin/mono/osx.arm64.Release/'. The 'mono' subset must be built before building this project. Configuration: 'Release'. To use a different configuration, specify the 'RuntimeConfiguration' property.

I think that the artifacts get place in .runtime/artifacts... and not into ./runtime/eng/runtime/artifacts.... However, this doesn't seem to cause problems when using --local-test-repo instead of --commits.

fanyang-mono · 2023-09-22T18:47:42Z

It would be nice to include a brief instruction on how to use this script. I see that common usage was mentioned in the comment at the beginning of the script. It would be very helpful to also list all the possible switches and what they do.

fanyang-mono · 2023-09-22T19:56:35Z

When using the script, I noticed in the log, it said

[INFO] Killing any running dotnet, vstest, or msbuild processes... (ignore system cannot find path specified)

This seems to be a overkill for local runs.

kotlarmilos · 2023-09-22T21:04:06Z

How do we test that the machines running in perf lab are doing AOT-llvm and not using mini? Could we verify by using mono-aot-cross --llvm --version on the lab machine (or other way) that we actually do AOT-llvm? @kotlarmilos

Yes, I think it is a good idea. Let's fix the naming and add mono-aot-cross --llvm --version command in a follow-up PR.

Please post your findings when you get a chance.

LoopedBard3 · 2023-09-22T21:46:44Z

How do we test that the machines running in perf lab are doing AOT-llvm and not using mini? Could we verify by using mono-aot-cross --llvm --version on the lab machine (or other way) that we actually do AOT-llvm? @kotlarmilos

Yes, I think it is a good idea. Let's fix the naming and add mono-aot-cross --llvm --version command in a follow-up PR.

Please post your findings when you get a chance.

If it is based on the mono-aot-cross executable, should we be able to pull down the built artifact and test on a like system, or is there system specific things that we would need to check for on the specific machines? If it can be tested locally, I can pull down the build mono-aot-cross to wsl and get the output.

kotlarmilos · 2023-09-23T10:49:28Z

If it is based on the mono-aot-cross executable, should we be able to pull down the built artifact and test on a like system, or is there system specific things that we would need to check for on the specific machines? If it can be tested locally, I can pull down the build mono-aot-cross to wsl and get the output.

Yes, I think we can retrieve it from artifacts and use it on linux-x64.

… to use should be the ones present in the Versions.props, otherwise built artifacts end up with unexpected paths causing failing builds. As such, also removed the framework from the artifact path for this reason. Removed the cross arguments as they should not be necessary for local builds.

…m-engine-path and removed the TODO setup required for the mono-libclang-path since it is not strictly required.

…in = OSX).

fanyang-mono · 2023-09-26T15:19:19Z

When using the script, I noticed in the log, it said
[INFO] Killing any running dotnet, vstest, or msbuild processes... (ignore system cannot find path specified)
This seems to be a overkill for local runs.

I understand that this needs to be done to ensure the accuracy of the results. However, this prevented me from doing other work, such as building other repos. One possible solution to this issue could be running the microbenchmarks inside another vm or a codespace or a docker container or some other controlled environment for use cases of validating a certain commit hash for causing performance changes.

…LLVM runs.

fanyang-mono · 2023-09-26T20:28:56Z

It would be nice for this script to support running microbenchmarks from a local dotnet/performance repo. It is beneficial for use cases like adding new microbenchmarks and editing existing microbenchmarks.

LoopedBard3 · 2023-09-26T23:33:55Z

It would be nice for this script to support running microbenchmarks from a local dotnet/performance repo. It is beneficial for use cases like adding new microbenchmarks and editing existing microbenchmarks.

Do you mean in a different local dotnet/performance repo? Since the current setup uses the scripts in the dotnet/performance repo that it is being run from, local changes should be testable/reflected properly when made to the same repo as the script. Please let me know if this has not been your experience or if you mean some other setup.

fanyang-mono · 2023-09-27T15:43:09Z

It would be nice for this script to support running microbenchmarks from a local dotnet/performance repo. It is beneficial for use cases like adding new microbenchmarks and editing existing microbenchmarks.

Do you mean in a different local dotnet/performance repo? Since the current setup uses the scripts in the dotnet/performance repo that it is being run from, local changes should be testable/reflected properly when made to the same repo as the script. Please let me know if this has not been your experience or if you mean some other setup.

You are right. I confused myself. Please ignore this comment.

matouskozak

LGTM!

As mentioned by @fanyang-mono, for future work I would consider removing the kill all dotnet processes.

Additionally, when running with admin privileges, it would be nice that only the microbenchmarks are run with higher privileges and building of runtime is done with standard user privileges. This matters when working with local repo as you usually must then clean everything so that you can rebuild in normal user mode.

kotlarmilos

Thank you!

fanyang-mono · 2023-10-05T18:46:57Z

I've opened an issue (#3385) to captured most of the suggestion, which are not included in this PR. Feel free to add more, if I missed anything.

LoopedBard3 added the enhancement New feature or request label Sep 12, 2023

LoopedBard3 requested review from DrewScoggins, caaavik-msft, cincuranet and sblom September 12, 2023 20:24

LoopedBard3 self-assigned this Sep 12, 2023

LoopedBard3 marked this pull request as ready for review September 12, 2023 20:37

LoopedBard3 requested review from SamMonoRT and radical September 13, 2023 21:26

SamMonoRT requested a review from kotlarmilos September 14, 2023 18:57

kotlarmilos reviewed Sep 15, 2023

View reviewed changes

scripts/benchmarks_local.py Show resolved Hide resolved

ivanpovazan reviewed Sep 18, 2023

View reviewed changes

scripts/benchmarks_local.py Outdated Show resolved Hide resolved

LoopedBard3 marked this pull request as draft September 18, 2023 17:32

LoopedBard3 mentioned this pull request Sep 22, 2023

Fix default repo_storage_path with os.getcwd #3363

Merged

LoopedBard3 added 6 commits September 25, 2023 11:01

First pass, untested.

7088502

Add clr to teh MonoAOTLLVM build runtime dependency call.

f406df0

Add mono_libclang_path as an optional argument/parameter.

d189f04

Build clr and libs separately.

f71cdb2

Fix mono-libclang-path dependency build argument.

f4dedaa

LoopedBard3 added 6 commits September 25, 2023 11:01

Removed unnecessary generate_layout.

1dd3854

Channel map update not necessary for this PR.

26b8de9

Move the mono-libclang-path argument above the required when WASM was…

be3141b

…m-engine-path and removed the TODO setup required for the mono-libclang-path since it is not strictly required.

Fix MonoAOTLLVM on windows.

e8f15a2

Add some prereq information and clarify the possible OS options (Darw…

7668088

…in = OSX).

Add arm64 LLVM enable when running on an arm64 system.

16c57ac

LoopedBard3 force-pushed the AddMonoAOTLLVMToLocalScript branch from b165fea to 16c57ac Compare September 25, 2023 18:01

matouskozak mentioned this pull request Sep 26, 2023

[mono] Fix linux-x64 AOT-llvm perf job dotnet/runtime#92644

Merged

Move the llvm enable and aot enable to the build_args for all MonoAOT…

dbdcd7a

…LLVM runs.

matouskozak approved these changes Oct 2, 2023

View reviewed changes

LoopedBard3 marked this pull request as ready for review October 2, 2023 21:50

kotlarmilos self-requested a review October 3, 2023 10:19

kotlarmilos approved these changes Oct 3, 2023

View reviewed changes

LoopedBard3 enabled auto-merge (squash) October 3, 2023 17:59

LoopedBard3 disabled auto-merge October 3, 2023 17:59

LoopedBard3 merged commit 77df656 into dotnet:main Oct 3, 2023

LoopedBard3 deleted the AddMonoAOTLLVMToLocalScript branch October 3, 2023 18:01

fanyang-mono mentioned this pull request Oct 5, 2023

Improvement request for the local testing script for microbenchmarks #3385

Closed

6 tasks

Add MonoAOTLLVM to local script #3338

Add MonoAOTLLVM to local script #3338

Uh oh!

Conversation

LoopedBard3 commented Sep 12, 2023

Uh oh!

Uh oh!

matouskozak commented Sep 15, 2023

Uh oh!

matouskozak commented Sep 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matouskozak commented Sep 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

matouskozak commented Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matouskozak commented Sep 22, 2023

Uh oh!

fanyang-mono commented Sep 22, 2023

Uh oh!

fanyang-mono commented Sep 22, 2023

Uh oh!

kotlarmilos commented Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LoopedBard3 commented Sep 22, 2023

Uh oh!

kotlarmilos commented Sep 23, 2023

Uh oh!

fanyang-mono commented Sep 26, 2023

Uh oh!

fanyang-mono commented Sep 26, 2023

Uh oh!

LoopedBard3 commented Sep 26, 2023

Uh oh!

fanyang-mono commented Sep 27, 2023

Uh oh!

matouskozak left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kotlarmilos left a comment

Choose a reason for hiding this comment

Uh oh!

fanyang-mono commented Oct 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

matouskozak commented Sep 16, 2023 •

edited

Loading

matouskozak commented Sep 18, 2023 •

edited

Loading

matouskozak commented Sep 22, 2023 •

edited

Loading

kotlarmilos commented Sep 22, 2023 •

edited

Loading

matouskozak left a comment •

edited

Loading