source-build: build llvm-project. #16757

tmds · 2023-06-20T11:33:28Z

Contributes to dotnet/source-build#1215.

@MichaelSimons @premun ptal

cc @omajid @ashnaga

premun · 2023-06-20T13:02:27Z

I am looking into why the repo didn't get initialized correctly in the build as this is expected to work

premun · 2023-06-20T14:06:19Z

Fix is here - dotnet/arcade-services#2689

MichaelSimons · 2023-06-20T14:09:43Z

eng/Version.Details.xml

      <Sha>4234ffac87e305da80597cb46dc0d4a87fc4f7c2</Sha>
      <SourceBuild RepoName="xliff-tasks" ManagedOnly="true" />
    </Dependency>
+    <Dependency Name="runtime.linux-x64.Microsoft.NETCore.Runtime.ObjWriter" Version="8.0.0-preview.6.23316.4" CoherentParentDependency="Microsoft.NETCore.App.Runtime.win-x64">


Ideally this would be defined within the runtime's version.details.xml. Additionally the use of SourceBuildTarball should be updated to just SourceBuild. I am trying to remove the SourceBuildTarball attribute.

I copied it from the old PR. I will update it.

I have no idea what this does. Can you give a short description how this is used, or point me to some doc?

Ideally this would be defined within the runtime's version.details.xml.

You want me to make a PR against https://github.com/dotnet/runtime?
and add

<Dependency Name="runtime.linux-x64.Microsoft.NETCore.Runtime.ObjWriter" Version="8.0.0-preview.6.23316.4" CoherentParentDependency="Microsoft.NETCore.App.Runtime.win-x64"> <Uri>https://github.com/dotnet/llvm-project</Uri> <Sha>30e9e6bc2e9f04e0a75daf4b8088ee91f66069da</Sha> </Dependency>

to its eng/Version.Details.xml?

It looks like these are already defined:

https://github.com/dotnet/runtime/blob/178425101dd6f697682031a9a953450d82719eb6/eng/Version.Details.xml#L15-L46

Can you give a short description how this is used

It does two things.

It indicates that the dependency should be included in the VMR. It specified the digest to include. The version details graph is walked and the first entry wins if there are multiple for a repo.

It tells the source build infra to pull in the intermediate package for the repo when building the repo level source build. The intermediate contents pre-populate the local nuget cache for prebuilt detection.

It looks like these are already defined:

One will need the SourceBuild element added to it.

One will need the element added to it.

I don't understand what you mean.

got it, the SourceBuild element.

Apologies, I used angle brackets in my original response and it wasn't rendered. I updated my comment.

premun · 2023-06-20T15:35:18Z

@tmds @MichaelSimons I don't have a fix for the problem and I don't know what's going wrong there but.. I won't have time until Friday to fix this.

Can we, as a workaround (and reading the PR here too), check this in without the <SourceBuild> tag? This would only add the source-mappings.json record and not initialize the repository. I would then do that manually through a PR into dotnet/dotnet to unblock this.

We can then wait until the source build tag propagates from runtime where we want to put it anyway.

MichaelSimons · 2023-06-20T16:07:43Z

I have no objection to @premun's proposed workaround.

tmds · 2023-06-21T07:42:25Z

check this in without the <SourceBuild> tag?

Was this handled by 246798d, or is there something else I should do?

premun · 2023-06-21T10:07:52Z

Ah, it's now failing because the repo project cannot build the folder that is not there.

I think we'd need to do this in 3 stages:

Add the source-mappings.json record only
Initialize the LLVM repo in the VMR through a PR to dotnet/dotnet
Add the repo project and everything else

Question - we first need to do dotnet/llvm-project#439 anyway, right?

tmds · 2023-06-21T11:04:01Z

Add the source-mappings.json record only

Created #16768.

Initialize the LLVM repo in the VMR through a PR to dotnet/dotnet

I assume we don't need to wait for dotnet/llvm-project#439 to do this? The sources can be added and won't get compiled until we add this PR.

@premun can you add the LLVM repo in the VMR?
Is the source-mappings.json update needed for that?

premun · 2023-06-21T14:31:56Z

So.. I investigated this a bit and git is unable to apply a patch representing the LLVM repo. It only returns a generic error:

error: git apply: failed to read: No such file or directory

I could initialize the repo more manually but looking at it, htere's a high number of quite large binaries so I don't think we should just do it without a proper preparation and cloaking rule setup.

MichaelSimons · 2023-06-22T11:56:28Z

I could initialize the repo more manually but looking at it, htere's a high number of quite large binaries so I don't think we should just do it without a proper preparation and cloaking rule setup.

Agreed - the repo is 3.0 G on disk.

jkotas · 2023-06-22T13:11:40Z

All binaries that I can see are under tests. It should be fine to cloak all test directories.

jkotas · 2023-06-22T13:16:34Z

Also, it should be ok to cloak the following top-level directories:

libclc
lld
lldb
mlir
openmp
polly
third-party

directhex · 2023-06-22T13:37:24Z

Also, it should be ok to cloak the following top-level directories:
libclc
lld
lldb
mlir
openmp
polly
third-party

Do we want the non-cloaked folders to cover only CoreCLR's needs (i.e. objwriter's deps) or also Mono? We need to keep lld for Mono. I think bolt, flang and pstl can go though?

jkotas · 2023-06-22T13:54:50Z

We need to keep lld for Mono.

I did not know that. We should keep lld for Mono then.

omajid · 2023-06-22T16:42:13Z

We should an issue to track/fix all the UX issues related to adding a new repo to the VMR. @tmds, is that something you can do?

tmds · 2023-06-22T16:51:30Z

We should an issue to track/fix all the UX issues related to adding a new repo to the VMR. @tmds, is that something you can do?

We're looking into fixing the issues as we encounter them. If there are things we're not fixing now, I think @premun and @MichaelSimons will create issues to track them.

premun · 2023-06-23T07:44:45Z

I will start looking today into the failing git apply as well as the right way to cloak the llvm-project repository

tmds · 2023-06-23T08:08:48Z

@premun thanks. I will let you and @MichaelSimons add the llvm-project in the vmr, and once it's in there I'll continue working towards dotnet/source-build#1215.

premun · 2023-06-23T08:13:49Z

@directhex what is the right way to test whether the trimmed llvm repo works for Mono?

src/VirtualMonoRepo/source-mappings.json

Co-authored-by: Přemek Vysoký <[email protected]>

premun · 2023-06-23T09:05:26Z

Looking at the build, to make this work, we'd have to temporarily add the <SourceBuild> tag back so that the LLVM repo is part of the dependency tree here. That is if we want to use this PR for prototyping this.

tmds · 2023-06-23T10:12:49Z

@premun since I have this PR open, I assume you will have permissions to push to my branch. Feel free to do so.

This reverts commit 246798d.

premun · 2023-06-23T13:47:10Z

Seems like pstl is needed:

##[error]/vmr/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/SourceBuildArcadeBuild.targets(121,5): error MSB3026: Could not copy "/vmr/src/llvm-project/libcxx/test/std/pstl" to "/vmr/src/llvm-project/artifacts/source-build/self/src/libcxx/test/std/pstl". Beginning retry 6 in 1000ms. Could not find file '/vmr/src/llvm-project/libcxx/test/std/pstl'.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=318493&view=logs&j=941f2e24-48a0-594a-6b1a-0e16aaeb8606&t=25681878-3d3d-50c4-a78d-b74bb03f4ec7

premun · 2023-06-23T14:33:58Z

I've done few more investigations and, interestingly, if the /lldb directory is included, then git diff + git apply fails.
If I exclude /lldb, it works. If I sync only /lldb, it also works. It only breaks when I sync the whole repo and git apply fails.

I suspect this is a bug in git itself. I posted a bugreport to the git mailing list but had other fires today so didn't spend more on this.

But right now, it's failing on some build error that I will let @directhex or possibly someone else hash out.

EDIT: These are the largest files in lldb, a possibility:

8.9M    ./unittests/Process/minidump/Inputs/fizzbuzz_wow64.dmp
8.9M    ./test/API/functionalities/postmortem/wow64_minidump/fizzbuzz_wow64.dmp
476K    ./source/Plugins/Instruction/ARM/EmulateInstructionARM.cpp
368K    ./source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp
324K    ./test/API/functionalities/postmortem/elf-core/linux-ppc64le.core
292K    ./source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp
256K    ./source/Core/IOHandlerCursesGUI.cpp
248K    ./test/API/functionalities/postmortem/netbsd-core/2lwp_t2_SIGSEGV.aarch64.core
244K    ./test/API/functionalities/postmortem/netbsd-core/2lwp_process_SIGSEGV.aarch64.core
228K    ./tools/debugserver/source/RNBRemote.cpp
228K    ./source/Target/Process.cpp
224K    ./test/API/functionalities/postmortem/netbsd-core/1lwp_SIGSEGV.aarch64.core
216K    ./source/Plugins/Process/gdb-remote/ProcessGDBRemote.cpp
188K    ./source/Commands/CommandObjectTarget.cpp
180K    ./source/Plugins/LanguageRuntime/RenderScript/RenderScriptRuntime/RenderScriptRuntime.cpp
168K    ./source/Target/Target.cpp
156K    ./tools/debugserver/source/MacOSX/MachProcess.mm
152K    ./source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
152K    ./source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp
148K    ./test/API/functionalities/postmortem/elf-core/linux-aarch64-pac.core
144K    ./test/API/functionalities/postmortem/elf-core/linux-aarch64-sve-full.core
144K    ./test/API/functionalities/postmortem/elf-core/linux-aarch64-sve-fpsimd.core
144K    ./source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp
140K    ./source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp
128K    ./source/Plugins/ObjectFile/ELF/ObjectFileELF.cpp
128K    ./source/Interpreter/CommandInterpreter.cpp
124K    ./tools/lldb-vscode/lldb-vscode.cpp
120K    ./test/API/functionalities/postmortem/netbsd-core/2lwp_t2_SIGSEGV.amd64.core
120K    ./test/API/functionalities/postmortem/netbsd-core/2lwp_process_SIGSEGV.amd64.core
120K    ./source/Plugins/LanguageRuntime/ObjC/AppleObjCRuntime/AppleObjCRuntimeV2.cpp
120K    ./include/lldb/Target/Process.h
116K    ./tools/debugserver/source/MacOSX/x86_64/DNBArchImplX86_64.cpp
112K    ./source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp
112K    ./source/Core/ValueObject.cpp
108K    ./unittests/UnwindAssembly/x86/Testx86AssemblyInspectionEngine.cpp
108K    ./unittests/SymbolFile/PDB/Inputs/test-pdb.pdb
108K    ./source/Expression/DWARFExpression.cpp
104K    ./source/Commands/CommandObjectType.cpp
104K    ./packages/Python/lldbsuite/test/lldbtest.py
100K    ./unittests/SymbolFile/PDB/Inputs/test-pdb-types.pdb
100K    ./test/Shell/Minidump/Windows/Sigsegv/Inputs/sigsegv.pdb
100K    ./test/Shell/Minidump/Windows/Sigsegv/Inputs/sigsegv.dmp
100K    ./test/API/functionalities/postmortem/netbsd-core/1lwp_SIGSEGV.amd64.core

tmds · 2023-06-23T14:41:17Z

But right now, it's failing on some build error that I will let @directhex or possibly someone else hash out.

It looks like something the changes made in dotnet/llvm-project#439 to llvvm.proj would fix.

premun · 2023-06-26T07:44:14Z

Ok, so got in touch with git maintainers and the problem with git not processing the patch is that in October, they introduced a 1GB cap on patch size due to some problems with types they use during parsing.

The changed happened here: git/git@f1c0e3946e
I was suggested to use git-repo-filter which is a python tool that is supposed to work faster but we could only use that during initialization, not VMR synchronization.
git maintainers will fix the error to contain the real reason but there is no override as it would mean fixing quite a few things on their side first.

I am not in favour of bringing python into the mix so I think we should just detect >1GB patches and in that case, we can just piece the patches by top-level folder (recursively) and solve this.
That being said, the cloaked llvm repo now fits just under 1GB (it's a bit over 900MB now) so it's not strictly blocking and we're not losing any files or so. I will log an issue for this improvement and hopefully we don't initialize any huge repo soon.

tmds · 2023-07-04T09:10:50Z

@premun what is still needed to finish this PR?

premun · 2023-07-04T09:20:23Z

@premun what is still needed to finish this PR?

I don't have full context on how llvm-project is used in the build. My understanding is that runtime uses it?
The repo that actually depends on llvm-project should have the appropriate record in its Version.Details.xml declared as a source build dependency via the SourceBuild tag. I don't know if that should be runtime then?
The dependency should then point at the SHA that we care about - is that the new versioned-main commit I saw?
We should remove the tag that we added here artificially and flow here 'runtime' which has the SourceBuild tag inside.
We should make sure in this PR that the cloaking rules for LLVM do not omit something that breaks the build by building whatever configuration that uses LLVM in a PR build here.
Possibly, we could even run a full VMR build with all SB legs (might need to do this internally) to make sure we don't break other builds.

tmds · 2023-07-04T09:38:16Z

Can we start by getting the llvm-project into the vmr so it gets built but without wiring it up further?

For that, I believe we need to finish this PR, and dotnet/llvm-project#439?
Is that right? Or do we need something more?

premun · 2023-07-04T09:56:33Z

Yes, that PR has to go through so that we can verify that it source-builds at the repo level. There's no point adding the repo in the VMR yet though. Or is there a reason for that?

Personally, I would do that as late as possible. We were already about to do that once and didn't end up doing that. I think adding the repo will increase the size of the repo quite a lot so until we can plug it in and start using it, I'd rather wait.

tmds · 2023-07-04T10:24:24Z

Yes, that PR has to go through so that we can verify that it source-builds at the repo level. There's no point adding the repo in the VMR yet though. Or is there a reason for that?

I'm still stuck at get the llvm-project in the vmr (dotnet/source-build#1215 (comment)).

tmds · 2024-02-02T13:38:47Z

We no longer need to integrate llvm-project to have NativeAOT in .NET 9.

mthalman · 2024-02-02T14:13:44Z

We no longer need to integrate llvm-project to have NativeAOT in .NET 9.

Why is that?

steveisok · 2024-02-02T14:26:13Z

We no longer need to integrate llvm-project to have NativeAOT in .NET 9.

Why is that?

They switched to a c# version of the objwriter.

source-build: build llvm-project.

1443ae4

tmds requested a review from a team as a code owner June 20, 2023 11:33

tmds mentioned this pull request Jun 20, 2023

Support building sdks that can build self-contained / ReadyToRun / NativeAOT applications without using pre-builts from nuget.org dotnet/source-build#1215

Closed

Add llvm-project to source-mappings.json.

1ebf66a

tmds requested a review from a team as a code owner June 20, 2023 11:51

steveisok requested a review from directhex June 20, 2023 12:36

mthalman requested a review from jkotas June 20, 2023 13:25

premun mentioned this pull request Jun 20, 2023

Fix initialization of a new repository in VMR dotnet/arcade-services#2688

Closed

3 tasks

MichaelSimons reviewed Jun 20, 2023

View reviewed changes

Remove ObjWriter dependency from eng/Version.Details.xml.

246798d

MichaelSimons added the Area-SourceBuild label Jun 22, 2023

premun reviewed Jun 23, 2023

View reviewed changes

src/VirtualMonoRepo/source-mappings.json Show resolved Hide resolved

Update src/VirtualMonoRepo/source-mappings.json

31cfd5f

Co-authored-by: Přemek Vysoký <[email protected]>

Revert "Remove ObjWriter dependency from eng/Version.Details.xml."

e46c3d3

This reverts commit 246798d.

Add pstl

fd69619

premun mentioned this pull request Jun 26, 2023

Latest git version does not support >1GB patches dotnet/arcade-services#2710

Closed

tmds closed this Feb 2, 2024

source-build: build llvm-project. #16757

source-build: build llvm-project. #16757

Uh oh!

Conversation

tmds commented Jun 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

premun commented Jun 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

premun commented Jun 20, 2023

Uh oh!

MichaelSimons Jun 20, 2023

Choose a reason for hiding this comment

Uh oh!

tmds Jun 20, 2023

Choose a reason for hiding this comment

Uh oh!

tmds Jun 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mthalman Jun 20, 2023

Choose a reason for hiding this comment

Uh oh!

MichaelSimons Jun 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmds Jun 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaelSimons Jun 20, 2023

Choose a reason for hiding this comment

Uh oh!

premun commented Jun 20, 2023

Uh oh!

MichaelSimons commented Jun 20, 2023

Uh oh!

tmds commented Jun 21, 2023

Uh oh!

premun commented Jun 21, 2023

Uh oh!

tmds commented Jun 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

premun commented Jun 21, 2023

Uh oh!

MichaelSimons commented Jun 22, 2023

Uh oh!

jkotas commented Jun 22, 2023

Uh oh!

jkotas commented Jun 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

directhex commented Jun 22, 2023

Uh oh!

jkotas commented Jun 22, 2023

Uh oh!

omajid commented Jun 22, 2023

Uh oh!

tmds commented Jun 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

premun commented Jun 23, 2023

Uh oh!

tmds commented Jun 23, 2023

Uh oh!

premun commented Jun 23, 2023

Uh oh!

Uh oh!

premun commented Jun 23, 2023

Uh oh!

tmds commented Jun 23, 2023

Uh oh!

premun commented Jun 23, 2023

Uh oh!

premun commented Jun 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tmds commented Jun 20, 2023 •

edited

Loading

premun commented Jun 20, 2023 •

edited

Loading

tmds Jun 20, 2023 •

edited

Loading

MichaelSimons Jun 20, 2023 •

edited

Loading

tmds Jun 20, 2023 •

edited

Loading

tmds commented Jun 21, 2023 •

edited

Loading

jkotas commented Jun 22, 2023 •

edited

Loading

tmds commented Jun 22, 2023 •

edited

Loading

premun commented Jun 23, 2023 •

edited

Loading

premun commented Jun 26, 2023 •

edited

Loading