GitHub Actions step is hanging until timeout

[Lastique]

Describe the bug

At some point, a GitHub Actions step stops producing output and hangs until the workflow is terminated by timeout.

You can see the output stops being produced if you actively watch the job output in the browser. If you try opening an already hanging job in the browser, the step output is not displayed at all: the ">" sign next to the step is not visible and the icon next to is is a spinning yellow circle. The output becomes available once the workflow is terminated.

Here is an example of one such run. Here is the log with timestamps:

boost_log_gha_hang.txt.gz

I'm sure the processes running in the GHA step are not hanging (this is just a C++ compilation process, which does succeed normally).

This problem seems to be not unique to my setup, there are other reports as well. In my case it happens most often with macos-10.15 runner, but I've also seen it happen with ubuntu-20.04 runner.

To Reproduce

At the moment, this reproduces quite reliably by running the workflow on my Boost.Log repository (git revision boostorg/log@2f058c6). I intend to try working around the problem (e.g. by disabling the offending runner), so this may not stay reproducible in later revisions.

Expected behavior

The runner should not hang.

Runner Version and Platform

I'm using the free runners hosted by GitHub.

[Lastique]

After deeper investigation it turned out that a process indeed could hang while executing a job step. After this was fixed, the job no longer hangs.

However, I still find it incorrect that I could not access step logs while the job was hanging. This makes it impossible to know what's going on until the job times out or is cancelled. I'm leaving this bug open for this issue to be fixed.

[phadej]

I have seen this lately consitently, e.g.

• https://github.com/haskell/aeson/runs/4149850593?check_suite_focus=true is one job where the step doesn't take long (compare to other compiler versions).

This is very frustrating, also knowing that a runner spins there for 6 hours not doing anything meaningful.

[phadej]

Another job https://github.com/haskell/aeson/runs/4178377747?check_suite_focus=true these seem to fail always at the same spot, which is good as the failure is somewhat deterministic. But I have no idea what it may be. I added a memory limit compiler could use (quite conservative), but that doesn't seem to help. Sometimes job succeed sometimes they don't. I still suspect that it is some limit jobs hit, and then they are lost.

[phadej]

I tried out my out-of-memory hypothesis: When the memory gets used the whole job becomes a lot slower and unresponsive:

• https://github.com/haskell-CI/haskell-ci/runs/4179461975?check_suite_focus=true
• https://github.com/haskell-CI/haskell-ci/runs/4179761658?check_suite_focus=true

When I removed the runtime system memory limit the job failed

• https://github.com/haskell-CI/haskell-ci/runs/4180347133?check_suite_focus=true

In these cases the job however didn't timeout, but it was unresponsive and logs weren't streamed.

Can high memory usage cause slowdown of the job? Is there anyway to get analytics of how much resources job runners have used?

[phadej]

This problem started to happening quite a lot: e.g.

• https://github.com/haskell-hvr/cryptohash-md5/runs/4191002990?check_suite_focus=true
• https://github.com/haskellari/postgresql-libpq/runs/4191104523?check_suite_focus=true
• https://github.com/haskell-hvr/cryptohash-sha256/runs/4190923993?check_suite_focus=true
• https://github.com/ekmett/lens/runs/4191293179?check_suite_focus=true
• https://github.com/haskellari/microstache/runs/4190698729?check_suite_focus=true
• https://github.com/haskellari/lattices/runs/4191251590?check_suite_focus=true

Previously these jobs didn't have any problems.

[chris-griffin]

One observation is that while the timed-out step will not show any log detail in the Action GUI (it will continue to render as in-progress/yellow), detailed logs of that step are available via the gear icon => "View raw logs".

[nikola-jokic]

Hi everyone,

I will close this issue now since @chris-griffin explained how can you view raw logs. Thank you @chris-griffin!

@phadej, if you are still seeing this issue, and if you are running your workflows on your self-hosted runner, could you double-check if you have enough computing power? It might be due to that. If you cannot find what is causing this issue, please submit a new issue, and all the relevant information so we can re-produce it and see what is causing you this trouble.