[wasm] `dotnet-runtime-perf` pipeline completely broken

[radical]

The wasm perf runs are broken after #77852 . I haven't investigated why that would be breaking the runs, but all of the wasm ones broke. I don't know if some of the non-wasm ones are breaking too.

1. previous passing pipeline run which is building exactly the previous commit
2. and the failing run

The wasm runs are failing with:

[2022/11/03 22:06:50][INFO] $ dotnet exec MicroBenchmarks.dll --anyCategories Libraries Runtime "" --category-exclusion-filter NoInterpreter NoWASM NoMono --wasmEngine /home/helixbot/.jsvu/v8 --cli /home/helixbot/work/B85009EF/p/dotnet/dotnet --wasmDataDir /home/helixbot/work/B85009EF/p/wasm-data --logBuildOutput --generateBinLog "" --partition-count 30 --partition-index 10 --artifacts /home/helixbot/work/B85009EF/w/9F070920/e/artifacts/BenchmarkDotNet.Artifacts --packages /home/helixbot/work/B85009EF/w/9F070920/e/performance/artifacts/packages --runtimes wasmnet70 --buildTimeout 1200
[2022/11/03 22:06:50][INFO] Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
[2022/11/03 22:06:50][INFO] 
[2022/11/03 22:06:50][INFO] File name: 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
[2022/11/03 22:06:50][INFO] $ popd
[2022/11/03 22:06:50][ERROR] Process exited with status -6
[2022/11/03 22:06:50][INFO] Run failure registered
Traceback (most recent call last):
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/benchmarks_ci.py", line 326, in <module>
    __main(sys.argv[1:])
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/benchmarks_ci.py", line 295, in __main
    args
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/micro_benchmarks.py", line 357, in run
    status = dotnet.exec(asm_path, success_exit_codes, verbose, *run_args)
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/dotnet.py", line 54, in exec
    return RunCommand(cmdline, success_exit_codes, verbose=verbose).run(working_dir)
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/performance/common.py", line 251, in run
    returncode, quoted_cmdline)
subprocess.CalledProcessError: Command '$ dotnet exec MicroBenchmarks.dll --anyCategories Libraries Runtime "" --category-exclusion-filter NoInterpreter NoWASM NoMono --wasmEngine /home/helixbot/.jsvu/v8 --cli /home/helixbot/work/B85009EF/p/dotnet/dotnet --wasmDataDir /home/helixbot/work/B85009EF/p/wasm-data --logBuildOutput --generateBinLog "" --partition-count 30 --partition-index 10 --artifacts /home/helixbot/work/B85009EF/w/9F070920/e/artifacts/BenchmarkDotNet.Artifacts --packages /home/helixbot/work/B85009EF/w/9F070920/e/performance/artifacts/packages --runtimes wasmnet70 --buildTimeout 1200' died with <Signals.SIGABRT: 6>.

cc @lewing @AaronRobinsonMSFT

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The wasm perf runs are broken after #77852 . I haven't investigated why that would be breaking the runs, but all of the wasm ones broke. I don't know if some of the non-wasm ones are breaking too.

1. previous passing pipeline run which is building exactly the previous commit
2. and the failing run

The wasm runs are failing with:

[2022/11/03 22:06:50][INFO] $ dotnet exec MicroBenchmarks.dll --anyCategories Libraries Runtime "" --category-exclusion-filter NoInterpreter NoWASM NoMono --wasmEngine /home/helixbot/.jsvu/v8 --cli /home/helixbot/work/B85009EF/p/dotnet/dotnet --wasmDataDir /home/helixbot/work/B85009EF/p/wasm-data --logBuildOutput --generateBinLog "" --partition-count 30 --partition-index 10 --artifacts /home/helixbot/work/B85009EF/w/9F070920/e/artifacts/BenchmarkDotNet.Artifacts --packages /home/helixbot/work/B85009EF/w/9F070920/e/performance/artifacts/packages --runtimes wasmnet70 --buildTimeout 1200
[2022/11/03 22:06:50][INFO] Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
[2022/11/03 22:06:50][INFO] 
[2022/11/03 22:06:50][INFO] File name: 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
[2022/11/03 22:06:50][INFO] $ popd
[2022/11/03 22:06:50][ERROR] Process exited with status -6
[2022/11/03 22:06:50][INFO] Run failure registered
Traceback (most recent call last):
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/benchmarks_ci.py", line 326, in <module>
    __main(sys.argv[1:])
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/benchmarks_ci.py", line 295, in __main
    args
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/micro_benchmarks.py", line 357, in run
    status = dotnet.exec(asm_path, success_exit_codes, verbose, *run_args)
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/dotnet.py", line 54, in exec
    return RunCommand(cmdline, success_exit_codes, verbose=verbose).run(working_dir)
  File "/home/helixbot/work/B85009EF/w/9F070920/e/performance/scripts/performance/common.py", line 251, in run
    returncode, quoted_cmdline)
subprocess.CalledProcessError: Command '$ dotnet exec MicroBenchmarks.dll --anyCategories Libraries Runtime "" --category-exclusion-filter NoInterpreter NoWASM NoMono --wasmEngine /home/helixbot/.jsvu/v8 --cli /home/helixbot/work/B85009EF/p/dotnet/dotnet --wasmDataDir /home/helixbot/work/B85009EF/p/wasm-data --logBuildOutput --generateBinLog "" --partition-count 30 --partition-index 10 --artifacts /home/helixbot/work/B85009EF/w/9F070920/e/artifacts/BenchmarkDotNet.Artifacts --packages /home/helixbot/work/B85009EF/w/9F070920/e/performance/artifacts/packages --runtimes wasmnet70 --buildTimeout 1200' died with <Signals.SIGABRT: 6>.

cc @lewing @AaronRobinsonMSFT

Author:	radical
Assignees:	-
Labels:	arch-wasm, perf-pipeline
Milestone:	-

[radical]

• I tested with Aaron's commit (8036885) intact
• Between the two CI runs:
  • dotnet/performance repo had some new commits, and removing dotnet/performance@860803b seems to fix the issue
  • Even with the commit, building for net7.0 works fine
  • The 8.0 sdk installed in dotnet/runtime, and the Wasm-tools workload isn't quite on net8.0 yet. And now we also have Revert "Update assembly version from hardcoded to MajorVersion" #77899 . I think this performance commit should be reverted till the sdk+runtime becomes usable for net8.0 .

cc @LoopedBard3 @ericstj

@AaronRobinsonMSFT this looks completely unrelated to your PR 👍

[AaronRobinsonMSFT]

@AaronRobinsonMSFT this looks completely unrelated to your PR 👍

❤️ Thanks @radical. Your diligence here is most appreciated.

[radical]

Closing this as the pipeline has been green after the reverts, and changes.