Compile times of large generated functions

[AntonReinhard]

One of the things Julia (deservedly) prides itself on is how dynamic it is and how well it understands its own code, i.e., meta-programming. This works great, but has a soft-limitation, because compile times get totally out of hand.

I'm generating some large functions and have collected a few data points (on 1.12.1, the top-level function's body is marked @noinline):

I understand that compilation is a hard problem and scales non-linearly considering many necessary optimizations. However, other languages (Fortran, C/C++, ...) can handle much larger codes and functions than this. The largest function in the plot above is ~37k lines long and takes about 5 minutes to compile. The next larger problem size (which would definitely still be relevant to me) would be about 160k lines long and I have not been able to compile this yet at all, even after hours of waiting.

Why is this such a problem for Julia? Is there any effort to tackle this issue? I know there has been lots of work on precompilation and making larger/complex packages like the plotting libraries compile faster, but this seems like a different topic.

I'd be glad to provide test results or performance profiling if that helps.

[vchuravy]

Can you share the code that generated these plots?

[AntonReinhard]

Sure, here it is:

https://github.com/AntonReinhard/QEDFeynmanDiagrams.jl/tree/experiments/experiments/01_comp_time

It should be enough to install the respective packages, run the run.jl and then run the plot.jl. The actual code generation happens in ComputableDAGs.jl . I tried several things already before to improve the compile times in that project, the best results so far were adding the @noinline, nothing else (adding closures to break up the function, adding noop extern c calls to prevent optimization across those points) did much.

[mikmoore]

Is this issue specific to @generated or is it essentially identical if you paste the same code into a non-@generated body? If it's not @generated specifically, this looks like a duplicate of #48991 (but I like the graph).

[AntonReinhard]

It's not a generated function at all, it's currently relying on RGF.jl for a little more freedom in what can be done inside (most importantly allowing eval calls).

I could try generating the code and putting it in a separate file and look what the times look like then, but I'd be very confused if that makes a difference.

[mikmoore]

I misread your "generated" as "@generated", then. My bad.

[xal-0]

I think this is hitting some pathological behaviour in LateLowerGCFrame::ComputeLiveSets. For make_nphoton_compton(n), as n increases, I see it spending longer and longer in LargeSparseBitVector::operator|=, up to almost half the total compile time.