Provide more useful info when sysimage generation fails

[IanButterworth]

There's very little to go off here (which is frustrating after a long build process)

✖ [13m:52s] PackageCompiler: compiling incremental system image
ERROR: failed process: Process(`/home/ian/.julia/juliaup/julia-1.8.2+0.x64/bin/julia --color=yes --startup-file=no --cpu-target=native -O3 --sysimage=/home/ian/.julia/juliaup/julia-1.8.2+0.x64/lib/julia/sys.so --project=/home/ian/.julia/packages/Foo/A61yO --output-o=/tmp/jl_j0by5sNSt8.o /tmp/jl_ZurqZanGvy`, ProcessSignaled(9)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run
    @ ./process.jl:477 [inlined]
  [4] #15
    @ ~/.julia/packages/PackageCompiler/DKijm/ext/TerminalSpinners.jl:157 [inlined]
  [5] spin(f::PackageCompiler.var"#15#16"{Cmd}, s::PackageCompiler.TerminalSpinners.Spinner{Base.TTY})
    @ PackageCompiler.TerminalSpinners ~/.julia/packages/PackageCompiler/DKijm/ext/TerminalSpinners.jl:164
  [6] macro expansion
    @ ~/.julia/packages/PackageCompiler/DKijm/ext/TerminalSpinners.jl:157 [inlined]
  [7] create_sysimg_object_file(object_file::String, packages::Vector{String}, packages_sysimg::Set{Base.PkgId}; project::String, base_sysimage::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, extra_precompiles::String, incremental::Bool)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/DKijm/src/PackageCompiler.jl:364
  [8] create_sysimage(packages::Symbol; sysimage_path::String, project::String, precompile_execution_file::String, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, base_sysimage::Nothing, julia_init_c_file::Nothing, version::Nothing, soname::Nothing, compat_level::String, extra_precompiles::String)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/DKijm/src/PackageCompiler.jl:515

Also, any idea what the actual issue could be here?

[Octogonapus]

I would also like to note that our precompile script which exercises most of the functionality in our package runs without errors.

Also, this error is 100% reproducible and started happening after we updated our dependencies, so I suspect a dependency is having a bad interaction with PackageCompiler. However, there are many dependencies, so it will take a very long time if I have to check each one in isolation. Hence we are looking for guidance on how we can get more info out of PackageCompiler.

[DilumAluthge]

Perhaps the process was killed by the OOM killer? How much memory did you have available on the machine on which you ran PackageCompiler? Can you try again on a machine with more memory?

[IanButterworth]

A SIGKILL doesn't show anything with or without the spinner. Is there any way to detect an OOM kill?

julia> cmd = `julia -e "sleep(1000)"`
`julia -e 'sleep(1000)'`

julia> spinner = PackageCompiler.TerminalSpinners.Spinner(msg = "PackageCompiler: compiling incremental system image");

julia> PackageCompiler.TerminalSpinners.@spin spinner run(cmd)
✖ [01m:05s] PackageCompiler: compiling incremental system image
ERROR: failed process: Process(`julia -e 'sleep(1000)'`, ProcessSignaled(9)) [0]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:565 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:480
 [3] run(::Cmd)
   @ Base ./process.jl:477
 [4] (::var"#3#4")()
   @ Main ~/.julia/packages/PackageCompiler/DKijm/ext/TerminalSpinners.jl:157
 [5] spin(f::var"#3#4", s::PackageCompiler.TerminalSpinners.Spinner{Base.TTY})
   @ PackageCompiler.TerminalSpinners ~/.julia/packages/PackageCompiler/DKijm/ext/TerminalSpinners.jl:164
 [6] top-level scope
   @ ~/.julia/packages/PackageCompiler/DKijm/ext/TerminalSpinners.jl:157

julia> run(cmd)
ERROR: failed process: Process(`julia -e 'sleep(1000)'`, ProcessSignaled(9)) [0]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:565 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:480
 [3] run(::Cmd)
   @ Base ./process.jl:477
 [4] top-level scope
   @ REPL[12]:1

[DilumAluthge]

Is there any way to detect an OOM kill?

You could look in your system logs to try to find a "process killed" event that has a timestamp close to when you noticed your PackageCompiler process being killed.

[IanButterworth]

True but I meant in PackageCompiler/julia. I guess it's a very platform-specific thing to check, unfortunately

[DilumAluthge]

Yeah, I think the location of the logs is going to be different for different systems. And even on a single system, you may need to check multiple log locations.

I'm not sure if PackageCompiler can do a lot here. AFAICT, the Linux OOM killer always sends a SIGKILL, so we can't catch the signal or anything.

[Octogonapus]

We can confirm that this happens because the process is terminated by the OOM killer. I'm surprised that we don't see Killed in the terminal, though.

Also #740 correctly warns about memory usage in this case.