Overseer: subsystems communicate directly

[rphmeier]

The previous behavior was that messages would go first to the overseer and then be relayed onwards to the destination subsystem.

Now, subsystems communicate via channels directly to other subsystems. Additionally, they have the option of either sending a message through a bounded or unbounded channel. The unbounded channels should be used sparingly in situations where:

1. There is a communication cycle between subsystems
2. One of the parts of the cycle has a clear bound on the number of messages produced.

A few examples of places we might use send_unbounded_message are when we send locally generated backing statements to statement distribution or when we send assignments and approvals from approval voting to the approval distribution subsystem. This avoids deadlock when a network subsystem is .awaiting a send to the core subsystem, while the core subsystem sends to the networking subsystem at the same time. If buffers are full, that will deadlock. Unbounded channels avoids this situation.

SubsystemContext and the overseer itself have an invariant that when a subsystem sends a message after receiving a signal from the overseer, that the recipient of the message will not receive it until having observed the same signal. This allows subsystems to assume that other subsystems are aware of the same view of blocks as they are.

My change-set here upholds the same invariant by having each subsystem tag each message with the number of signals that it had observed before sending the message. If the recipient has not received at least that many signals, it pockets the message until it has received enough signals. This rests on the fact that the overseer sends signals to all subsystems equally. If one subsystem receives a signal, others can expect to receive it shortly.

I have also given SubsystemContext a SubsystemSender assoc. type which will be used to split sending and receiving. Follow-up PRs will make use of this feature.

[eskimor]

Our use of channels is still likely to lead to deadlocks, because we .await on both sides unconditionally. Future refactoring should alter subsystems to time-out messages to other subsystems so that progress can be made even under heavy load.

I think we could do better than this, but I need to check a few things first and think a bit more about it.

[coriolinus]

Core logic in OverseerSubsystemContext::recv LGTM. I agree that this maintains the message invariant.

I'm pleasantly surprised that making this change didn't require changing any of the individual subsystems.

[coriolinus]

It's too bad we don't have HKTs yet; if we did, you could simplify this into

let make_packet = for<T> |message: T| { ... }

and simply capture the values of timer and signals_received, which would simplify usage.

[eskimor]

Well we can do without, if we are sure this is the way to go, we can just expose the subsystem senders directly.

[rphmeier]

I had the same thought when I was writing the code, but unfortunately no HKT yet

[coriolinus]

I suspect that at some point we're going to want to debug which particular subsystem was the intended recipient of the failed message. Maybe an api like

impl AllMessages {
  pub fn recipient_name(&self) -> &'static str { ... }
}

would make that simple to add to this debug message.

[eskimor]

Looks sensible to me.

[drahnr]

Implementation wise, it looks good.

As a side effect here we lose the overseer as our probe point for messages though.

[ordian]

naming nit: what's the difference between parachain_subsystem and parachain_overseer prefix? They both are parameterized by subsystem_name.

[rphmeier]

subsystem_signals_sent sounds like the subsystems are sending the signals but they are actually receiving them. overseer_unbounded_sent has the same issue in the opposite direction.

[rphmeier]

These are OverseerSignals so overseer_signals_sent makes sense I think. But I could rename subsystem_bounded_sent etc.. not sure to what

[rphmeier]

bot merge

[ghost]

Waiting for commit status.