Nimbus Consensus Framework

[JoshOrndorff]

This PR introduces Nimbus -- a framework for writing slot-based parachain consensus algorithms. The nimbus framework strives to make non-trivial consensus algorithms a reality for cumulus parachains. This is an alternative approach to #371 that strives to be more reusable. In addition to the framework itself, this PR introduces three example consensus algorithms expressed in the Nimbus framework, and installs one of them in the rococo collator to provide a ready-to-use example of a chain built with Nimbus.

One of the included example consensus engines is the popular Aura round robin engine. I've written a comparison of Aura implementations to help explain the difference between the two approaches.

I've written a lot of docs about the Nimbus framework's architecture in the Nimbus Readme (which is part of this PR). I encourage reviewers to read that. Rather than re-iterate that in this PR description, I'll focus on the high-level value add of Nimbus by comparing it to the approach that wraps sc-consensus-aura.

There are two key differences between this approach and #371:

1. Reusability - The Nimbus framework is designed so that as much of it as possible can be reused among multiple slot-based consensus algorithms. The entire client-side worker is reused among all three included examples. This is in contrast to the existing Aura and Babe implementations as they exist in the Substrate repo. A development team who wanted to write a custom consensus algorithm in Nimbus will have much less code to write and understand. (Of course this is only true for consensus engines that fit within the slot-based category.) This is evidenced by the <100 lines implementation of Aura.
2. In-Runtime author checking - Nimbus checks whether the author of a block is eligible to author in the runtime rather than on the client-side. This makes it easier for relay chain validators, who only run wasm, to make the same eligibility checks. It also makes reusing the client side worker more realistic.

[cla-bot-2021]

User @atenjin, please sign the CLA here.

[xlc]

[bkchr]

So what is the difference between nimbus and making the authority set of Aura configurable?

By making the authority set configurable, I mean the following:

Aura frame crate:
trait Config {
   type AuthoritySet: AuthoritySet;
}

trait AuthoritySet {
   fn allowed_authors_for_block() -> Vec<ValidatorId>;
}

This is very sketchy, but it would allow you to use whatever authority selection algorithm you want.

[JoshOrndorff]

This is about more than changing the authority set. If you change the authority set in Aura, the authorities will still be rotated one at a time in order. So while extending Aura in this way does add generality, it still misses many consensus algorithms that nimbus supports. Here are some examples that would not work in this proposed Aura modification.

• Just like Aura except two authors are selected every slot.
• Authors are selected pseudorandomly such that authors who have not been selected in a long time having higher probability of being selected this time.

Focusing on the consensus layer alone, the core difference between nimbus and Aura is that Aura puts the round-robin logic directly in the consensus layer. So for a block to be aura valid it has to be signed by the author whose turn it is in this slot.

OTOH, Nimbus puts fewer checks in the consensus layer. Nimbus considers any block valid as long as it is signed by the same author id that is injected into the runtime. From there it is up to the runtime to reject blocks that where the author id that was injected is not eligible.

[bkchr]

This is about more than changing the authority set. If you change the authority set in Aura, the authorities will still be rotated one at a time in order. So while extending Aura in this way does add generality, it still misses many consensus algorithms that nimbus supports. Here are some examples that would not work in this proposed Aura modification.

* Just like Aura except two authors are selected every slot.

* Authors are selected pseudorandomly such that authors who have not been selected in a long time having higher probability of being selected this time.

All this would be supported by my idea. The authorityset trait clearly needs more functions, however it would be responsible for all of this. Aka checking that the author is correct etc. allowing multiple authors per block, having a different set per block. whatever.

[JoshOrndorff]

So your idea is about modifying more than the runtime side of Aura, right?

Currently the rotate-in-order logic is coded into the client-side worker. And the claim_slot implementation assumes that there will only be one author per slot.

Definitely all of that could be generalized. And once it is generalized, I think the remaining differences are just superficial. Here's one that comes to mind.

// Your exhaustive set idea:
fn allowed_authors_for_block() -> Vec<ValidatorId>;

// My propositional set idea:
fn can_author(author: AuthorId, slot: u32) -> bool;

It seems like our ideas have basically converged :) I think if you modify Aura that way, the name "authority round" doesn't make that much sense anymore, so we should change the name. Might I suggest Nimbus ;)

[bkchr]

Yeah I also know that this would require touching the client side :P

I will bring up the name, however I would like to not have that much special code here in cumulus that is mostly the same for Aura.

[JoshOrndorff]

I would like to not have that much special code here in cumulus...

Yeah, I agree with that completely. As this work progressed, I realized most of it is just the plain consensus engine, and only a small part of it is cumulus specific. It would make more sense to have the main consensus engine part directly in Substrate.

...that is mostly the same for Aura.

Yeah, I agree with that too. One of my goals here was not duplicating so much code to write a new consensus engine.

I will bring up the name

Haha cool. No pressure, I was just teasing because it's one of the only remaining differences.