From cbaf2f3d7f0491b72992bbe77afc3dbb50842be3 Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 2 Jun 2020 13:40:53 +0200 Subject: [PATCH 01/17] expand subsystems descriptions independent of subsystems overview The subsystems overview in https://github.com/paritytech/polkadot/pull/1161 is a very useful reference for what subsystems need to talk to each other, when, and why; it helps us design the actual messages. However, administratively, it belongs in a different PR. This commit brings in all the changes made there so far as a base for an independent PR. --- roadmap/implementors-guide/guide.md | 52 ++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 4e0d01898613..fddf18553105 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -913,17 +913,15 @@ Furthermore, the protocols by which subsystems communicate with each other shoul #### Description -The Candidate Backing subsystem is engaged in by validators in to contribute to the backing of parachain candidates submitted by other validators. +The Candidate Backing subsystem ensures at least one preliminary validator commits to each parablock's correctness. Parablocks for which no validator will assert correctness are discarded. If the block later proves invalid, the initial backers are slashable; this gives polkadot a rational threat model during subsequent stages. Its role is to produce backable candidates for inclusion in new relay-chain blocks. It does so by issuing signed [Statements](#Statement-type) and tracking received statements signed by other validators. Once enough statements are received, they can be combined into backing for specific candidates. -It also detects double-vote misbehavior by validators as it imports votes, passing on the misbehavior to the correct reporter and handler. - -When run as a validator, this is the subsystem which actually validates incoming candidates. - #### Protocol -This subsystem receives messages of the type [CandidateBackingSubsystemMessage](#Candidate-Backing-Subsystem-Message). +The **Candidate Selection** subsystem is the primary source of non-overseer messages into this subsystem. That subsystem generates appropriate [`CandidateBackingSubsystemMessage`s](#Candidate-Backing-Subsystem-Message), and passes them to this subsystem. + +This subsystem validates the candidates and generates an appropriate `Statement`. All `Statement`s are then passed on to the **Statement Distribution** subsystem to be gossiped to peers. Instances of `Statement::Invalid` are also passed back to the Candidate Selection subsystem, so it can take action against the originator of the candidate. To reduce traffic, we do not send valid `Statement`s back to the Candidate Selection subsystem. #### Functionality @@ -994,9 +992,44 @@ Dispatch a `PovFetchSubsystemMessage(relay_parent, candidate_hash, sender)` and (TODO: send statements to Statement Distribution subsystem, handle shutdown signal from candidate backing subsystem) +### Candidate Selection Subsystem + +#### Description + +The Candidate Selection subsystem monitors net traffic for two events: + +- a new parablock candidate is available +- a peer has seconded a parablock candidate + +This module is only ever interested in parablocks assigned to the particular parachain which this validator is currently handling. + +New parablock candidates may arrive from a potentially unbounded set of collators. This subsystem chooses either 0 or 1 of them per relay parent to second. If it chooses to second a candidate, it sends an apropriate message to the **Candidate Backing** subsystem to generate an appropriate `Statement`. + +All parablocks which peers have seconded are also sent to the Candidate Backing subsystem for re-validation. +As seconded peers are tallied, double-votes are detected. If found, a report is sent to the **Misbehavior Arbitration** subsystem. + +In the event that a parablock candidate proves invalid, this subsystem will receive a message back from the Candidate Backing subsystem indicating so. If that parablock candidate originated from a collator, this subsystem will blacklist that collator. If that parablock candidate originated from a peer, this subsystem generates a report for the **Misbehavior Arbitration** subsystem. + + +TODO: more details, protocol, etc + +### Misbehavior Arbitration Subsystem + +#### Description + +The Misbehavior Arbitration system collects reports of validator misbehavior, and slashes the stake of both misbehaving validator nodes and false accusers. + +#### TODOs + +- threshold of reports for deciding that a validator has misbehaved +- threshold of reports for deciding that an accusation was false +- what to do if there are enough reports to pass the first threshold, but not enough for the second +- time period over which to collect reports +- detailed protocol + --- -[TODO: subsystems for gathering data necessary for block authorship, for networking, for misbehavior reporting, etc.] +[TODO: subsystems for gathering data necessary for block authorship, for networking, etc.] ---- @@ -1086,7 +1119,10 @@ enum CandidateBackingSubsystemMessage { RegisterBackingWatcher(Hash, TODO), /// Note that the Candidate Backing subsystem should second the given candidate in the context of the /// given relay-parent (ref. by hash). This candidate must be validated. - Second(Hash, CandidateReceipt) + Second(Hash, CandidateReceipt), + /// The only difference between the `Validate` and `Second` variants is which `Statement` variant + /// is returned when the candidate is valid. + Validate(Hash, CandidateReceipt), } ``` From 45674fa8f990c1722a3a31f1519070f0cc7c5dd9 Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 2 Jun 2020 15:12:51 +0200 Subject: [PATCH 02/17] Reorder subsystem descriptions, add some messages Update ordering of subsystem descriptions to rough order of use, mirroring the order in the overview document. Added some message types. Added OverseerSignal variants to several types, such that each subsystem only needs to listen for a single type. --- roadmap/implementors-guide/guide.md | 145 ++++++++++++++++++---------- 1 file changed, 92 insertions(+), 53 deletions(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index fddf18553105..98182064408e 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -909,6 +909,27 @@ Furthermore, the protocols by which subsystems communicate with each other shoul --- +### Candidate Selection Subsystem + +#### Description + +The Candidate Selection subsystem is notified by the Overseer on two events: + +- a new parablock candidate is available +- a peer has seconded a parablock candidate + +This module is only ever interested in parablocks assigned to the particular parachain which this validator is currently handling. + +New parablock candidates may arrive from a potentially unbounded set of collators. This subsystem chooses either 0 or 1 of them per relay parent to second. If it chooses to second a candidate, it sends an apropriate message to the **Candidate Backing** subsystem to generate an appropriate `Statement`. + +All parablocks which peers have seconded are also sent to the Candidate Backing subsystem for re-validation. +As seconded peers are tallied, double-votes are detected. If found, a report is sent to the **Misbehavior Arbitration** subsystem. + +In the event that a parablock candidate proves invalid, this subsystem will receive a message back from the Candidate Backing subsystem indicating so. If that parablock candidate originated from a collator, this subsystem will blacklist that collator. If that parablock candidate originated from a peer, this subsystem generates a report for the **Misbehavior Arbitration** subsystem. + + +TODO: more details, protocol, etc + ### Candidate Backing Subsystem #### Description @@ -934,7 +955,6 @@ The subsystem should maintain a set of handles to Candidate Backing Jobs that ar *On CandidateBackingSubsystemMessage* * If the message corresponds to a particular relay-parent, forward the message to the Candidate Backing Job for that relay-parent, if any is live. - (big TODO: "contextual execution" * At the moment we only allow inclusion of _new_ parachain candidates validated by _current_ validators. * Allow inclusion of _old_ parachain candidates validated by _current_ validators. @@ -992,27 +1012,6 @@ Dispatch a `PovFetchSubsystemMessage(relay_parent, candidate_hash, sender)` and (TODO: send statements to Statement Distribution subsystem, handle shutdown signal from candidate backing subsystem) -### Candidate Selection Subsystem - -#### Description - -The Candidate Selection subsystem monitors net traffic for two events: - -- a new parablock candidate is available -- a peer has seconded a parablock candidate - -This module is only ever interested in parablocks assigned to the particular parachain which this validator is currently handling. - -New parablock candidates may arrive from a potentially unbounded set of collators. This subsystem chooses either 0 or 1 of them per relay parent to second. If it chooses to second a candidate, it sends an apropriate message to the **Candidate Backing** subsystem to generate an appropriate `Statement`. - -All parablocks which peers have seconded are also sent to the Candidate Backing subsystem for re-validation. -As seconded peers are tallied, double-votes are detected. If found, a report is sent to the **Misbehavior Arbitration** subsystem. - -In the event that a parablock candidate proves invalid, this subsystem will receive a message back from the Candidate Backing subsystem indicating so. If that parablock candidate originated from a collator, this subsystem will blacklist that collator. If that parablock candidate originated from a peer, this subsystem generates a report for the **Misbehavior Arbitration** subsystem. - - -TODO: more details, protocol, etc - ### Misbehavior Arbitration Subsystem #### Description @@ -1067,11 +1066,82 @@ struct BlockFinalizationEvent { } ``` +#### Overseer Signal + +Signals from the overseer to a subsystem to request change in execution that has to be obeyed by the subsystem. + +```rust +enum OverseerSignal { + /// Signal to start work localized to the relay-parent hash. + StartWork(Hash), + /// Signal to stop (or phase down) work localized to the relay-parent hash. + StopWork(Hash), +} +``` + +#### Candidate Selection Subsystem Message + +These messages are sent from the overseer to the Candidate Selection subsystem when new parablocks are available for validation. + +```rust +enum CandidateSelectionSubsystemMessage { + /// Start or stop work on blocks related to a given relay parent + Signal(OverseerSignal), + /// A new parachain candidate has arrived from a collator and should be considered for seconding. + NewCandidate(PoV, ParachainBlock), +} +``` + +If this subsystem chooses to second a parachain block, it dispatches a `CandidateBackingSubsystemMessage`. + +#### Candidate Backing Subsystem Message + +```rust +enum CandidateBackingSubsystemMessage { + /// Start or stop work on blocks related to a given relay parent + Signal(OverseerSignal), + /// Registers a stream listener for updates to the set of backable candidates that could be backed + /// in a child of the given relay-parent, referenced by its hash. + RegisterBackingWatcher(Hash, TODO), + /// Note that the Candidate Backing subsystem should second the given candidate in the context of the + /// given relay-parent (ref. by hash). This candidate must be validated. + Second(Hash, CandidateReceipt), +} +``` + +#### Validation Request Type + +Various modules request that the Candidate Validation subsystem validate a block with this message + +```rust +enum PoVOrigin { + /// The proof of validity is available here. + Included(PoV), + /// We need to fetch proof of validity from some peer on the network. + Network(CandidateReceipt), +} + +enum CandidateValidationSubsystemMessage { + /// Start or stop work on blocks related to a given relay parent + Signal(OverseerSignal), + /// Validate a candidate and issue a Statement + Validate(CandidateHash, RelayHash, PoVOrigin), +} +``` + #### Statement Type + +The Candidate Validation subsystem issues these messages in reponse to `ValidationRequest`s. + ```rust /// A statement about the validity of a parachain candidate. enum Statement { /// A statement about a new candidate being seconded by a validator. This is an implicit validity vote. + /// + /// The main semantic difference between `Seconded` and `Valid` comes from the fact that every validator may + /// second only 1 candidate; this places an upper bound on the total number of candidates whose validity + /// needs to be checked. A validator who seconds more than 1 parachain candidate per relay head is subject + /// to slashing. Seconded(CandidateReceipt), /// A statement about the validity of a candidate, based on candidate's hash. Valid(Hash), @@ -1095,37 +1165,6 @@ struct SignedStatement { } ``` - -#### Overseer Signal - -Signals from the overseer to a subsystem to request change in execution that has to be obeyed by the subsystem. - -```rust -enum OverseerSignal { - /// Signal to start work localized to the relay-parent hash. - StartWork(Hash), - /// Signal to stop (or phase down) work localized to the relay-parent hash. - StopWork(Hash), -} -``` - - -#### Candidate Backing subsystem Message - -```rust -enum CandidateBackingSubsystemMessage { - /// Registers a stream listener for updates to the set of backable candidates that could be backed - /// in a child of the given relay-parent, referenced by its hash. - RegisterBackingWatcher(Hash, TODO), - /// Note that the Candidate Backing subsystem should second the given candidate in the context of the - /// given relay-parent (ref. by hash). This candidate must be validated. - Second(Hash, CandidateReceipt), - /// The only difference between the `Validate` and `Second` variants is which `Statement` variant - /// is returned when the candidate is valid. - Validate(Hash, CandidateReceipt), -} -``` - #### Host Configuration The internal-to-runtime configuration of the parachain host. This is expected to be altered only by governance procedures. From cc68890ef58ba9810f29394c14dbb4e8652c129e Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 2 Jun 2020 16:02:56 +0200 Subject: [PATCH 03/17] add some more message types, Statement Distribution overview --- roadmap/implementors-guide/guide.md | 42 +++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 98182064408e..82f8c2c7014c 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1012,6 +1012,10 @@ Dispatch a `PovFetchSubsystemMessage(relay_parent, candidate_hash, sender)` and (TODO: send statements to Statement Distribution subsystem, handle shutdown signal from candidate backing subsystem) +### Statement Distribution Subsystem + +The statement distribution subsystem sends statements to peer nodes, detects double-voting, and tabulates when a sufficient portion of the validator set has unanimously judged a candidate. When judgment is not unanimous, it escalates the issue to misbehavior arbitration. + ### Misbehavior Arbitration Subsystem #### Description @@ -1056,6 +1060,7 @@ struct BlockImportEvent { ``` #### Block Finalization Event + ```rust /// Indicates that a new block has been finalized. struct BlockFinalizationEvent { @@ -1165,6 +1170,43 @@ struct SignedStatement { } ``` +#### Statement Distribution Subsystem Message + +```rust +enum StatementDistributionSubsystemMessage { + /// Start or stop work on blocks related to a given relay parent + Signal(OverseerSignal), + /// A peer has seconded a candidate and we need to double-check them + Peer(SignedStatement), + /// We have validated a candidate and want to share our judgment with our peers + /// + /// The statement distribution subsystem is responsible for signing this statement. + Judge(Statement), +} +``` + +#### Misbehavior Arbitration Subsystem Message + +```rust +enum MisbehaviorArbitrationSubsystemMessage { + /// Start or stop work on blocks related to a given relay parent + Signal(OverseerSignal), + /// These validator nodes disagree on this candidate's validity, please figure it out + /// + /// Most likely, the list of statments all agree except for the final one. That's not + /// guaranteed, though; if somehow we become aware of lots of + /// statements disagreeing about the validity of a candidate before taking action, + /// this message should be dispatched with all of them, in arbitrary order. + /// + /// This variant is also used when our own validity checks disagree with others'. + CandidateValidityDisagreement(CandidateReceipt, Vec), + /// I've noticed a peer contradicting itself about a particular candidate + SelfContradiction(CandidateReceipt, SignedStatement, SignedStatement), + /// This peer has seconded more than one parachain candidate for this relay parent head + DoubleVote(CandidateReceipt, SignedStatement, SignedStatement), +} +``` + #### Host Configuration The internal-to-runtime configuration of the parachain host. This is expected to be altered only by governance procedures. From 4679c5f004d5d118a8e2d29a0a100cb898667ca6 Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 2 Jun 2020 16:21:19 +0200 Subject: [PATCH 04/17] add more detail on Statement Distribution, Misbehavior Arbitration --- roadmap/implementors-guide/guide.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 82f8c2c7014c..183c840e4d85 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1016,14 +1016,21 @@ Dispatch a `PovFetchSubsystemMessage(relay_parent, candidate_hash, sender)` and The statement distribution subsystem sends statements to peer nodes, detects double-voting, and tabulates when a sufficient portion of the validator set has unanimously judged a candidate. When judgment is not unanimous, it escalates the issue to misbehavior arbitration. +Statement Distribution is the only subsystem which has any direction notion of peer nodes or of our own cryptographic keys. It is responsible for deciding when a quorum exists, and for detecting a variety of peer node misbehaviors for reporting to Misbehavior Arbitration. It's the main point of contact (via the Overseer) with peer nodes. It also responds to secondment by peer nodes by forwarding those votes to validation to double-check the peers. + +Once a sufficient quorum has agreed that a candidate is valid, this subsystem notifies the Overseer, which in turn engages block production mechanisms to include the parablock. + ### Misbehavior Arbitration Subsystem #### Description The Misbehavior Arbitration system collects reports of validator misbehavior, and slashes the stake of both misbehaving validator nodes and false accusers. +Misbehavior Arbitration is the only subsystem which has the ability to slash nodes. It also handles coordinating all validators to validate a particular block where disagreement exists about its validity, and slashing the minority voters. + #### TODOs +- Why would any node cooperate with a full-group vote where the minority is slashed? It seems like they have incentive to just not vote. Perhaps they're slashed if they don't vote one way or the other by some elapsed time period? But that makes them vulnerable to DoS attacks. - threshold of reports for deciding that a validator has misbehaved - threshold of reports for deciding that an accusation was false - what to do if there are enough reports to pass the first threshold, but not enough for the second @@ -1181,7 +1188,7 @@ enum StatementDistributionSubsystemMessage { /// We have validated a candidate and want to share our judgment with our peers /// /// The statement distribution subsystem is responsible for signing this statement. - Judge(Statement), + Share(Statement), } ``` From 97de4c45070b5ddc79d697062704ceebc0ace6a3 Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Wed, 3 Jun 2020 15:03:14 +0200 Subject: [PATCH 05/17] intentionally punt MA details for a future PR --- roadmap/implementors-guide/guide.md | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 49a85cde443a..20372f23d2aa 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1033,16 +1033,9 @@ Once a sufficient quorum has agreed that a candidate is valid, this subsystem no The Misbehavior Arbitration system collects reports of validator misbehavior, and slashes the stake of both misbehaving validator nodes and false accusers. -Misbehavior Arbitration is the only subsystem which has the ability to slash nodes. It also handles coordinating all validators to validate a particular block where disagreement exists about its validity, and slashing the minority voters. +It is not yet fully specified; that problem is postponed to a future PR. -#### TODOs - -- Why would any node cooperate with a full-group vote where the minority is slashed? It seems like they have incentive to just not vote. Perhaps they're slashed if they don't vote one way or the other by some elapsed time period? But that makes them vulnerable to DoS attacks. -- threshold of reports for deciding that a validator has misbehaved -- threshold of reports for deciding that an accusation was false -- what to do if there are enough reports to pass the first threshold, but not enough for the second -- time period over which to collect reports -- detailed protocol +One policy question we've decided even so: in the event that MA has to call all validators to check some block about which some validators disagree, the minority voters all get slashed, and the majority voters all get rewarded. Validators which abstain have a minor slash penalty, but probably not in the same order of magnitude as those who vote wrong. --- From c938d16a7a1acc05c4b28af33ea0ff63d7662b3b Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Thu, 4 Jun 2020 10:37:21 +0200 Subject: [PATCH 06/17] reduce duplication from overseer signal handling --- roadmap/implementors-guide/guide.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 20372f23d2aa..2e6d4737a35f 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1091,14 +1091,19 @@ enum OverseerSignal { } ``` +All subsystems have their own message types; all of them need to be able to listen for overseer signals as well. There are currently two proposals for how to handle that with unified communication channels: + +1. Retaining the `OverseerSignal` definition above, add `enum FromOverseer {Signal(OverseerSignal), Message(T)}`. +1. Add a generic varint to `OverseerSignal`: `Message(T)`. + +Either way, there will be some top-level type encapsulating messages from the overseer to each subsystem. + #### Candidate Selection Subsystem Message These messages are sent from the overseer to the Candidate Selection subsystem when new parablocks are available for validation. ```rust enum CandidateSelectionSubsystemMessage { - /// Start or stop work on blocks related to a given relay parent - Signal(OverseerSignal), /// A new parachain candidate has arrived from a collator and should be considered for seconding. NewCandidate(PoV, ParachainBlock), } @@ -1110,8 +1115,6 @@ If this subsystem chooses to second a parachain block, it dispatches a `Candidat ```rust enum CandidateBackingSubsystemMessage { - /// Start or stop work on blocks related to a given relay parent - Signal(OverseerSignal), /// Registers a stream listener for updates to the set of backable candidates that could be backed /// in a child of the given relay-parent, referenced by its hash. RegisterBackingWatcher(Hash, TODO), @@ -1134,8 +1137,6 @@ enum PoVOrigin { } enum CandidateValidationSubsystemMessage { - /// Start or stop work on blocks related to a given relay parent - Signal(OverseerSignal), /// Validate a candidate and issue a Statement Validate(CandidateHash, RelayHash, PoVOrigin), } @@ -1181,8 +1182,6 @@ struct SignedStatement { ```rust enum StatementDistributionSubsystemMessage { - /// Start or stop work on blocks related to a given relay parent - Signal(OverseerSignal), /// A peer has seconded a candidate and we need to double-check them Peer(SignedStatement), /// We have validated a candidate and want to share our judgment with our peers @@ -1196,8 +1195,6 @@ enum StatementDistributionSubsystemMessage { ```rust enum MisbehaviorArbitrationSubsystemMessage { - /// Start or stop work on blocks related to a given relay parent - Signal(OverseerSignal), /// These validator nodes disagree on this candidate's validity, please figure it out /// /// Most likely, the list of statments all agree except for the final one. That's not From 1d6337b29dfb096b3202266c5e7f8bd52878e5ee Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Thu, 4 Jun 2020 11:10:38 +0200 Subject: [PATCH 07/17] reword for clarity --- roadmap/implementors-guide/guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 2e6d4737a35f..75340168f335 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1023,7 +1023,7 @@ Dispatch a `PovFetchSubsystemMessage(relay_parent, candidate_hash, sender)` and The statement distribution subsystem sends statements to peer nodes, detects double-voting, and tabulates when a sufficient portion of the validator set has unanimously judged a candidate. When judgment is not unanimous, it escalates the issue to misbehavior arbitration. -Statement Distribution is the only subsystem which has any direction notion of peer nodes or of our own cryptographic keys. It is responsible for deciding when a quorum exists, and for detecting a variety of peer node misbehaviors for reporting to Misbehavior Arbitration. It's the main point of contact (via the Overseer) with peer nodes. It also responds to secondment by peer nodes by forwarding those votes to validation to double-check the peers. +Statement Distribution is the only subsystem which has any notion of peer nodes or of our own cryptographic keys. It is responsible for deciding when a quorum exists, and for detecting a variety of peer node misbehaviors for reporting to Misbehavior Arbitration. It's the main point of contact (via the Overseer) with peer nodes. On receiving a signed statement from a peer seconding a candidate block, it sends the Candidate Receipt to the Candidate Backing subsystem to double-check the peer's judgment. Once a sufficient quorum has agreed that a candidate is valid, this subsystem notifies the Overseer, which in turn engages block production mechanisms to include the parablock. From 0803ee511b561490aed1cb6ab688318c20b796ea Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Thu, 4 Jun 2020 11:12:18 +0200 Subject: [PATCH 08/17] clarify: other modules and subsystems also get to talk to the network --- roadmap/implementors-guide/guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 75340168f335..1171cb1f298a 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1023,7 +1023,7 @@ Dispatch a `PovFetchSubsystemMessage(relay_parent, candidate_hash, sender)` and The statement distribution subsystem sends statements to peer nodes, detects double-voting, and tabulates when a sufficient portion of the validator set has unanimously judged a candidate. When judgment is not unanimous, it escalates the issue to misbehavior arbitration. -Statement Distribution is the only subsystem which has any notion of peer nodes or of our own cryptographic keys. It is responsible for deciding when a quorum exists, and for detecting a variety of peer node misbehaviors for reporting to Misbehavior Arbitration. It's the main point of contact (via the Overseer) with peer nodes. On receiving a signed statement from a peer seconding a candidate block, it sends the Candidate Receipt to the Candidate Backing subsystem to double-check the peer's judgment. +Statement Distribution is the only subsystem which has any notion of peer nodes or of our own cryptographic keys. It is responsible for deciding when a quorum exists, and for detecting a variety of peer node misbehaviors for reporting to Misbehavior Arbitration. Within the Validity module, it's the main point of contact (via the Overseer) with peer nodes. On receiving a signed statement from a peer seconding a candidate block, it sends the Candidate Receipt to the Candidate Backing subsystem to double-check the peer's judgment. Once a sufficient quorum has agreed that a candidate is valid, this subsystem notifies the Overseer, which in turn engages block production mechanisms to include the parablock. From 07df04aa49969f58aa3a86d290315f7660efc898 Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 9 Jun 2020 13:45:36 +0200 Subject: [PATCH 09/17] finish current work on candidate selection --- roadmap/implementors-guide/guide.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 0800695e2a6f..b042021e2da4 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1002,8 +1002,16 @@ As seconded peers are tallied, double-votes are detected. If found, a report is In the event that a parablock candidate proves invalid, this subsystem will receive a message back from the Candidate Backing subsystem indicating so. If that parablock candidate originated from a collator, this subsystem will blacklist that collator. If that parablock candidate originated from a peer, this subsystem generates a report for the **Misbehavior Arbitration** subsystem. +#### Protocol + +The Candidate Selection protocol is currently intentionally unspecified pending further discussion. + +Several approaches have been selected, but all have some issues: -TODO: more details, protocol, etc +- The most straightforward approach is for this subsystem to simply second the first valid parablock candidate which it sees per relay head. However, that protocol is vulnerable to a single collator which, as an attack or simply through chance, gets its block candidate to the node more often than its fair share of the time. +- It may be possible to do some BABE-like selection algorithm to choose an "Official" collator for the round, but that is tricky because the collator which produces the PoV does not necessarily actually produce the block. +- We could use relay-chain BABE randomness to generate some delay `D` on the order of 1 second, +- 1 second. The collator would then second the first valid parablock which arrives after `D`, or in case none has arrived by `2*D`, the last valid parablock which has arrived. This makes it very hard for a collator to game the system to always get its block nominated, but it reduces the maximum throughput of the system by introducing delay into an already tight schedule. +- A variation of that scheme would be to randomly choose a number `I`, and have a fixed acceptance window `D` for parablock candidates. At the end of the period `D`, count `C`: the number of parablock candidates received. Second the one with index `I % C`. Its drawback is the same: it must wait the full `D` period before seconding any of its received candidates, reducing throughput. ### Candidate Backing Subsystem From 45a548ada5f24440ed720bebdfbae30ecbeafa96 Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 9 Jun 2020 14:02:20 +0200 Subject: [PATCH 10/17] update candidate backing subsystem description according to current thought --- roadmap/implementors-guide/guide.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index b042021e2da4..726acbad7545 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -988,10 +988,7 @@ Furthermore, the protocols by which subsystems communicate with each other shoul #### Description -The Candidate Selection subsystem is notified by the Overseer on two events: - -- a new parablock candidate is available -- a peer has seconded a parablock candidate +The Candidate Selection subsystem is notified by the Overseer when a new parablock candidate is available. This module is only ever interested in parablocks assigned to the particular parachain which this validator is currently handling. @@ -1017,10 +1014,12 @@ Several approaches have been selected, but all have some issues: #### Description -The Candidate Backing subsystem ensures at least one preliminary validator commits to each parablock's correctness. Parablocks for which no validator will assert correctness are discarded. If the block later proves invalid, the initial backers are slashable; this gives polkadot a rational threat model during subsequent stages. +The Candidate Backing subsystem ensures every parablock considered for relay block inclusion has been seconded by at least one validator, and approved by a quorum. Parablocks for which no validator will assert correctness are discarded. If the block later proves invalid, the initial backers are slashable; this gives polkadot a rational threat model during subsequent stages. Its role is to produce backable candidates for inclusion in new relay-chain blocks. It does so by issuing signed [Statements](#Statement-type) and tracking received statements signed by other validators. Once enough statements are received, they can be combined into backing for specific candidates. +Note that though the candidate backing subsystem attempts to produce as many backable candidates as possible, it does _not_ attempt to choose a single authoritative one. The choice of which actually gets included is ultimately up to the block author, by whatever metrics it may use; those are opaque to this subsystem. + #### Protocol The **Candidate Selection** subsystem is the primary source of non-overseer messages into this subsystem. That subsystem generates appropriate [`CandidateBackingSubsystemMessage`s](#Candidate-Backing-Subsystem-Message), and passes them to this subsystem. @@ -1197,6 +1196,9 @@ enum CandidateBackingSubsystemMessage { /// Note that the Candidate Backing subsystem should second the given candidate in the context of the /// given relay-parent (ref. by hash). This candidate must be validated. Second(Hash, CandidateReceipt), + /// Note a peer validator's statement about a particular candidate. Disagreements about validity must be escalated + /// to a broader check by Misbehavior Arbitration. Agreements are simply tallied until a quorum is reached. + Statement(Statement), } ``` From 460d58149e9b66cd737cd30388f84a00effdd3ff Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 9 Jun 2020 14:09:37 +0200 Subject: [PATCH 11/17] update mechanism for candidate backing to report collator misbehavior to candidate selection --- roadmap/implementors-guide/guide.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 726acbad7545..b1cb389d9b15 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1020,11 +1020,13 @@ Its role is to produce backable candidates for inclusion in new relay-chain bloc Note that though the candidate backing subsystem attempts to produce as many backable candidates as possible, it does _not_ attempt to choose a single authoritative one. The choice of which actually gets included is ultimately up to the block author, by whatever metrics it may use; those are opaque to this subsystem. +Once a sufficient quorum has agreed that a candidate is valid, this subsystem notifies the Overseer, which in turn engages block production mechanisms to include the parablock. + #### Protocol The **Candidate Selection** subsystem is the primary source of non-overseer messages into this subsystem. That subsystem generates appropriate [`CandidateBackingSubsystemMessage`s](#Candidate-Backing-Subsystem-Message), and passes them to this subsystem. -This subsystem validates the candidates and generates an appropriate `Statement`. All `Statement`s are then passed on to the **Statement Distribution** subsystem to be gossiped to peers. Instances of `Statement::Invalid` are also passed back to the Candidate Selection subsystem, so it can take action against the originator of the candidate. To reduce traffic, we do not send valid `Statement`s back to the Candidate Selection subsystem. +This subsystem validates the candidates and generates an appropriate `Statement`. All `Statement`s are then passed on to the **Statement Distribution** subsystem to be gossiped to peers. When this subsystem decides that a candidate is invalid, and it was recommended to us to second by our own Candidate Selection subsystem, a message is sent to the Candidate Selection subsystem with the candidate's hash so that the collator which recommended it can be penalized. #### Functionality @@ -1098,9 +1100,7 @@ Dispatch a `PovFetchSubsystemMessage(relay_parent, candidate_hash, sender)` and The statement distribution subsystem sends statements to peer nodes, detects double-voting, and tabulates when a sufficient portion of the validator set has unanimously judged a candidate. When judgment is not unanimous, it escalates the issue to misbehavior arbitration. -Statement Distribution is the only subsystem which has any notion of peer nodes or of our own cryptographic keys. It is responsible for deciding when a quorum exists, and for detecting a variety of peer node misbehaviors for reporting to Misbehavior Arbitration. Within the Validity module, it's the main point of contact (via the Overseer) with peer nodes. On receiving a signed statement from a peer seconding a candidate block, it sends the Candidate Receipt to the Candidate Backing subsystem to double-check the peer's judgment. - -Once a sufficient quorum has agreed that a candidate is valid, this subsystem notifies the Overseer, which in turn engages block production mechanisms to include the parablock. +Statement Distribution is the only validity module subsystem which has any notion of peer nodes or of our own cryptographic keys. It is responsible for signing statements that we have generated and forwarding them, and for detecting a variety of peer node misbehaviors for reporting to Misbehavior Arbitration. Within the Validity module, it's the main point of contact (via the Overseer) with peer nodes. On receiving a signed statement from a peer, assuming the peer receipt state machine is in an appropriate state, it sends the Candidate Receipt to the Candidate Backing subsystem to double-check the peer's judgment. ### Misbehavior Arbitration Subsystem @@ -1181,6 +1181,8 @@ These messages are sent from the overseer to the Candidate Selection subsystem w enum CandidateSelectionSubsystemMessage { /// A new parachain candidate has arrived from a collator and should be considered for seconding. NewCandidate(PoV, ParachainBlock), + /// We recommended a particular candidate to be seconded, but it was invalid; penalize the collator. + Invalid(CandidateReceipt), } ``` From f0848341c7e2e7c595b7e49292d893f626a114ad Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 9 Jun 2020 14:29:44 +0200 Subject: [PATCH 12/17] sketch out the peer receipt state machine --- roadmap/implementors-guide/guide.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index b1cb389d9b15..8103f557a268 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1102,6 +1102,14 @@ The statement distribution subsystem sends statements to peer nodes, detects dou Statement Distribution is the only validity module subsystem which has any notion of peer nodes or of our own cryptographic keys. It is responsible for signing statements that we have generated and forwarding them, and for detecting a variety of peer node misbehaviors for reporting to Misbehavior Arbitration. Within the Validity module, it's the main point of contact (via the Overseer) with peer nodes. On receiving a signed statement from a peer, assuming the peer receipt state machine is in an appropriate state, it sends the Candidate Receipt to the Candidate Backing subsystem to double-check the peer's judgment. +#### Peer Recipet State Machine + +There is a very simple state machine which governs which messages we are willing to receive from peer validators. Not depicted in the state machine: on initial receipt of any `SignedStatement`, validate that the provided signature does in fact sign the included data. + +A: Initial State. Receive `SignedStatement(Statement::Second)`: extract `Statement`, forward to Candidate Backing, proceed to B. Receive any other `SignedStatement` variant: drop it. +B: Receive any `SignedStatement`: extract `Statement`, forward to Candidate Backing. Receive `OverseerMessage::StopWork`: proceed to C. +C: Receive any message for this block: drop it. + ### Misbehavior Arbitration Subsystem #### Description From a46bd64a41bf23ecb3eff9d2c1ad0fa4e725a87a Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 9 Jun 2020 15:13:05 +0200 Subject: [PATCH 13/17] Fix typo in roadmap/implementors-guide/guide.md Co-authored-by: Robert Habermeier --- roadmap/implementors-guide/guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index 8103f557a268..ae3c08745825 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1102,7 +1102,7 @@ The statement distribution subsystem sends statements to peer nodes, detects dou Statement Distribution is the only validity module subsystem which has any notion of peer nodes or of our own cryptographic keys. It is responsible for signing statements that we have generated and forwarding them, and for detecting a variety of peer node misbehaviors for reporting to Misbehavior Arbitration. Within the Validity module, it's the main point of contact (via the Overseer) with peer nodes. On receiving a signed statement from a peer, assuming the peer receipt state machine is in an appropriate state, it sends the Candidate Receipt to the Candidate Backing subsystem to double-check the peer's judgment. -#### Peer Recipet State Machine +#### Peer Receipt State Machine There is a very simple state machine which governs which messages we are willing to receive from peer validators. Not depicted in the state machine: on initial receipt of any `SignedStatement`, validate that the provided signature does in fact sign the included data. From ec8732295e583b9c4fe5f49cd80eb9465ccf4728 Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 9 Jun 2020 15:14:16 +0200 Subject: [PATCH 14/17] Don't specify 'peer validators', as messages from non-validator peers are ignored regardless --- roadmap/implementors-guide/guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index ae3c08745825..bac9fcfa350a 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1104,7 +1104,7 @@ Statement Distribution is the only validity module subsystem which has any notio #### Peer Receipt State Machine -There is a very simple state machine which governs which messages we are willing to receive from peer validators. Not depicted in the state machine: on initial receipt of any `SignedStatement`, validate that the provided signature does in fact sign the included data. +There is a very simple state machine which governs which messages we are willing to receive from peers. Not depicted in the state machine: on initial receipt of any `SignedStatement`, validate that the provided signature does in fact sign the included data. A: Initial State. Receive `SignedStatement(Statement::Second)`: extract `Statement`, forward to Candidate Backing, proceed to B. Receive any other `SignedStatement` variant: drop it. B: Receive any `SignedStatement`: extract `Statement`, forward to Candidate Backing. Receive `OverseerMessage::StopWork`: proceed to C. From e8137ad3a18a15f23fbbba022f88f732023e442a Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 9 Jun 2020 15:17:36 +0200 Subject: [PATCH 15/17] clarify instancing of peer receipt state machine --- roadmap/implementors-guide/guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index bac9fcfa350a..ca37f7b291f5 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1104,7 +1104,7 @@ Statement Distribution is the only validity module subsystem which has any notio #### Peer Receipt State Machine -There is a very simple state machine which governs which messages we are willing to receive from peers. Not depicted in the state machine: on initial receipt of any `SignedStatement`, validate that the provided signature does in fact sign the included data. +There is a very simple state machine which governs which messages we are willing to receive from peers. Not depicted in the state machine: on initial receipt of any `SignedStatement`, validate that the provided signature does in fact sign the included data. Note that each individual parablock candidate gets its own instance of this state machine; it is perfectly legal to receive a `Valid(X)` before a `Seconded(Y)` from peer `A`. A: Initial State. Receive `SignedStatement(Statement::Second)`: extract `Statement`, forward to Candidate Backing, proceed to B. Receive any other `SignedStatement` variant: drop it. B: Receive any `SignedStatement`: extract `Statement`, forward to Candidate Backing. Receive `OverseerMessage::StopWork`: proceed to C. From d8085fa0a711a0e4be7f991b78d5edc55a6b008a Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 9 Jun 2020 15:57:47 +0200 Subject: [PATCH 16/17] add section on peer knowledge tracking --- roadmap/implementors-guide/guide.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index ca37f7b291f5..ca17cfebde27 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -1110,6 +1110,14 @@ A: Initial State. Receive `SignedStatement(Statement::Second)`: extract `Stateme B: Receive any `SignedStatement`: extract `Statement`, forward to Candidate Backing. Receive `OverseerMessage::StopWork`: proceed to C. C: Receive any message for this block: drop it. +#### Peer Knowledge Tracking + +The peer receipt state machine implies that for parsimony of network resources, we should model the knowledge of our peers, and help them out. For example, let's consider a case with peers A, B, and C, and candidate M. A sends us a `Statement::Second(M)`. We've double-checked it, and it's valid. While we're checking it, we receive a copy of A's `Statement::Second` from `B`, along with a `Statement::Valid` signed by B. + +Our response to `B` is just a `Statement::Valid` which we've signed. However, we haven't heard anything about this from C. Therefore, we send it everything we have: first a copy of A's `Statement::Second`, then both our and B's `Statement::Valid`. + +This system implies a certain level of duplication of messages--we received A's `Statement::Second` from both our peers, and C may experience the same--but it minimizes the degree to which messages are simply dropped. + ### Misbehavior Arbitration Subsystem #### Description From 6fc13167549fb80a508a02ebd68a8769a5ad0aee Mon Sep 17 00:00:00 2001 From: Peter Goodspeed-Niklaus Date: Tue, 9 Jun 2020 16:44:04 +0200 Subject: [PATCH 17/17] fix typo in roadmap/implementors-guide/guide.md Co-authored-by: Max Inden --- roadmap/implementors-guide/guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roadmap/implementors-guide/guide.md b/roadmap/implementors-guide/guide.md index ca17cfebde27..523c879de53d 100644 --- a/roadmap/implementors-guide/guide.md +++ b/roadmap/implementors-guide/guide.md @@ -992,7 +992,7 @@ The Candidate Selection subsystem is notified by the Overseer when a new parablo This module is only ever interested in parablocks assigned to the particular parachain which this validator is currently handling. -New parablock candidates may arrive from a potentially unbounded set of collators. This subsystem chooses either 0 or 1 of them per relay parent to second. If it chooses to second a candidate, it sends an apropriate message to the **Candidate Backing** subsystem to generate an appropriate `Statement`. +New parablock candidates may arrive from a potentially unbounded set of collators. This subsystem chooses either 0 or 1 of them per relay parent to second. If it chooses to second a candidate, it sends an appropriate message to the **Candidate Backing** subsystem to generate an appropriate `Statement`. All parablocks which peers have seconded are also sent to the Candidate Backing subsystem for re-validation. As seconded peers are tallied, double-votes are detected. If found, a report is sent to the **Misbehavior Arbitration** subsystem.