Hook messages within 'while' arrive out of order on OTP 28.3

[blasphemetheus]

test hooks executes hook within while in api_test.exs:230 fails intermittently — hook callbacks from successive while-loop iterations are delivered out of order. Expected {20.0, 3} (iteration 2) but received {60.0, 2} (iteration 3). Observed in CI on Elixir 1.18.4 / OTP 28.3.

[polvalente]

@blasphemetheus could this be related to the recent outfeed lock fixes? Maybe this is not an issue anymore.

[blasphemetheus]

Shouldn't be related to the lock fixes — those addressed cross-client outfeed queue corruption (#1691), so different layer.

This is a message ordering race in apply_hook (outfeed.ex:255). Each while iteration spawns a new process that sends ref back to the outfeed loop before running the hook function. The loop then spawns the next iteration's process, and since BEAM doesn't guarantee ordering between messages from different sender processes, hook messages can arrive out of order.

I can try and draft a regression test for this issue specifically tho

[blasphemetheus]

yeah, regression test (blasphemetheus@6513259#diff-dd43e16f8870ad734b915e33a0c2b13703baffa1f33fb18f08ce9c96ece3327bR252) forces the race deterministically, the hook sleeps proportional to the iteration value, so earlier iterations (higher values) take longer.

only reason this is relevant is because a test asserts on message ordering, so maybe someone logging values from a while loop could see them out of order occasionally, but this isn't a real need for message ordering usually.

So I think a fix to that is nice to have but we could also just note the possibility of a race rather than eliminating it, up to y'all

[polvalente]

I think message ordering is a bit tricky to ensure.
For instance, I just had to deal with XLA hoisting a custom_call to the very top of the function due to data dependency resolution.

Given that we're moving in a direction where outfeed will likely vanish in favor of custom_call in the near future, we can leave this issue for after that rework

[josevalim]

General ordering is hard, I agree, but while should be ordered.