Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Commit d0bc8ca

Browse files
drahnrrphmeieremostovLldenauroisbkchr
committed
move paras inherent filtering to runtime (#4028)
* move things around, add filter methods�� * validator keys, modify availability bitfields according to disputes * simplify, keep the filter -> sanitize generic for both usecases * minor * assure tests still work, reduce changeset * integration * start entropy passing * fixins * compile, 1 failing test * filter with coverage * fixins * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Robert Habermeier <[email protected]> * slip of the pen * improve test cases * misc * fix * fixins * test avoid extra into() calls in assert_noop! * chores * ff * test fixup superfluous into call * chore: pfmt * improve apply_block_weight_limit to try to maximize the number of sufficiently backed blocks and add extra bitfields in a round-robin fashion * new code treats the lack of backed candidates as ok * Use vrf based entropy * fixup vrf random * add warn * slip of the pen * fixup * assure ordering * rethink apply_weights * mock * use a closure as predicate check * extract and use DisputedBitfield * chore: simplify * remove stray dbg * chore: fmt * address feedback * fix test, halfway there * stage1 * dbg stuff * make group selection align * fix session index * fix wrongly returned candidates * cleanup * chore fmt * fix ensure check * make good case test work * more tests for bitfields * create sanitize_backed_candidates * fixup tests * update guide * add check referenced in the guide * improve weights code * fmt * fixins * Update roadmap/implementers-guide/src/runtime/inclusion.md Co-authored-by: Zeke Mostov <[email protected]> * compiling + address review * add comments * fix weight calc * address review comments and test failure * fix * fix: condition * Fix random_sel function * Fix overlength block check * Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests * Add benchmarks for code upgrades * Code upgrade bench; Feature gate TestWeightInfo * Try and make CI happier * Feature gate enter test to not(benchmarks) * Make sure no unused imports/fn * refactor, re-use, the beginning * Fix issue with frame benchmarking dep compilation * More precise feature gating for some derives * integrate piece-wise * foo * fixins * chore fmt * fixins * rename const generic * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Zeke Mostov <[email protected]> * Fix compilation * limit to test * remove unused spam slots * spellcheck * remove a tick, fix a typo * Add Code upgrade weights * comment improvements + >= Co-authored-by: Zeke Mostov <[email protected]> * remove another tick * Update runtime/parachains/src/paras_inherent/benchmarking.rs Co-authored-by: Zeke Mostov <[email protected]> * saturating fixins + some spaces * fix * benchmarking - preliminary results * Add training wheels * Refactor some early exit logic for enter * Gracefully handle filtering bitfields & candidates (#4280) This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items. * Refactor inherent data weight limiting logic (#4287) * Apply suggestions from code review * Update runtime/parachains/src/builder.rs Co-authored-by: Zeke Mostov <[email protected]> * Update runtime/parachains/src/builder.rs * Update runtime/parachains/src/paras_inherent.rs * final pass * Run cargo +nightly-2021-10-29 fmt * Update implementors guide with `sanitize_*` & `enter` (#4294) * Make spell check happier * Make wasm runtimes compile with benchmarks enabled (#4303) * comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest * Update weight files for runtime Co-authored-by: Robert Habermeier <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Lldenaurois <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Bastian Köcher <[email protected]>
1 parent bff00af commit d0bc8ca

39 files changed

+3928
-484
lines changed

Cargo.lock

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

parachain/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,4 @@ std = [
3636
"polkadot-core-primitives/std",
3737
"frame-support/std",
3838
]
39+
runtime-benchmarks = []

parachain/src/primitives.rs

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,19 @@ pub use polkadot_core_primitives::BlockNumber as RelayChainBlockNumber;
4141

4242
/// Parachain head data included in the chain.
4343
#[derive(
44-
PartialEq, Eq, Clone, PartialOrd, Ord, Encode, Decode, RuntimeDebug, derive_more::From, TypeInfo,
44+
PartialEq,
45+
Eq,
46+
Clone,
47+
PartialOrd,
48+
Ord,
49+
Encode,
50+
Decode,
51+
RuntimeDebug,
52+
derive_more::From,
53+
TypeInfo,
54+
Default,
4555
)]
46-
#[cfg_attr(feature = "std", derive(Serialize, Deserialize, Default, Hash, MallocSizeOf))]
56+
#[cfg_attr(feature = "std", derive(Serialize, Deserialize, Hash, MallocSizeOf))]
4757
pub struct HeadData(#[cfg_attr(feature = "std", serde(with = "bytes"))] pub Vec<u8>);
4858

4959
impl HeadData {

primitives/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,4 +55,4 @@ std = [
5555
"polkadot-core-primitives/std",
5656
"bitvec/std",
5757
"frame-system/std",
58-
]
58+
]

primitives/src/v0.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -107,8 +107,8 @@ impl MallocSizeOf for ValidatorId {
107107
}
108108

109109
/// Index of the validator is used as a lightweight replacement of the `ValidatorId` when appropriate.
110-
#[derive(Eq, Ord, PartialEq, PartialOrd, Copy, Clone, Encode, Decode, TypeInfo)]
111-
#[cfg_attr(feature = "std", derive(Serialize, Deserialize, Debug, Hash, MallocSizeOf))]
110+
#[derive(Eq, Ord, PartialEq, PartialOrd, Copy, Clone, Encode, Decode, TypeInfo, Debug)]
111+
#[cfg_attr(feature = "std", derive(Serialize, Deserialize, Hash, MallocSizeOf))]
112112
pub struct ValidatorIndex(pub u32);
113113

114114
// We should really get https://github.com/paritytech/polkadot/issues/2403 going ..

primitives/src/v1/mod.rs

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -327,8 +327,8 @@ fn check_collator_signature<H: AsRef<[u8]>>(
327327
}
328328

329329
/// A unique descriptor of the candidate receipt.
330-
#[derive(PartialEq, Eq, Clone, Encode, Decode, TypeInfo)]
331-
#[cfg_attr(feature = "std", derive(Debug, Default, Hash, MallocSizeOf))]
330+
#[derive(PartialEq, Eq, Clone, Encode, Decode, TypeInfo, Default)]
331+
#[cfg_attr(feature = "std", derive(Debug, Hash, MallocSizeOf))]
332332
pub struct CandidateDescriptor<H = Hash> {
333333
/// The ID of the para this is a candidate for.
334334
pub para_id: Id,
@@ -407,8 +407,8 @@ pub struct FullCandidateReceipt<H = Hash, N = BlockNumber> {
407407
}
408408

409409
/// A candidate-receipt with commitments directly included.
410-
#[derive(PartialEq, Eq, Clone, Encode, Decode, TypeInfo)]
411-
#[cfg_attr(feature = "std", derive(Debug, Default, Hash, MallocSizeOf))]
410+
#[derive(PartialEq, Eq, Clone, Encode, Decode, TypeInfo, Default)]
411+
#[cfg_attr(feature = "std", derive(Debug, Hash, MallocSizeOf))]
412412
pub struct CommittedCandidateReceipt<H = Hash> {
413413
/// The descriptor of the candidate.
414414
pub descriptor: CandidateDescriptor<H>,
@@ -509,8 +509,8 @@ impl<H: Encode, N: Encode> PersistedValidationData<H, N> {
509509
}
510510

511511
/// Commitments made in a `CandidateReceipt`. Many of these are outputs of validation.
512-
#[derive(PartialEq, Eq, Clone, Encode, Decode, TypeInfo)]
513-
#[cfg_attr(feature = "std", derive(Debug, Default, Hash, MallocSizeOf))]
512+
#[derive(PartialEq, Eq, Clone, Encode, Decode, TypeInfo, Default)]
513+
#[cfg_attr(feature = "std", derive(Debug, Hash, MallocSizeOf))]
514514
pub struct CandidateCommitments<N = BlockNumber> {
515515
/// Messages destined to be interpreted by the Relay chain itself.
516516
pub upward_messages: Vec<UpwardMessage>,
@@ -534,6 +534,8 @@ impl CandidateCommitments {
534534
}
535535

536536
/// A bitfield concerning availability of backed candidates.
537+
///
538+
/// Every bit refers to an availability core index.
537539
#[derive(PartialEq, Eq, Clone, Encode, Decode, RuntimeDebug, TypeInfo)]
538540
pub struct AvailabilityBitfield(pub BitVec<bitvec::order::Lsb0, u8>);
539541

primitives/src/v1/signed.rs

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,13 @@ use crate::v0::{SigningContext, ValidatorId, ValidatorIndex, ValidatorSignature}
4141
#[derive(Clone, PartialEq, Eq, RuntimeDebug)]
4242
pub struct Signed<Payload, RealPayload = Payload>(UncheckedSigned<Payload, RealPayload>);
4343

44+
impl<Payload, RealPayload> Signed<Payload, RealPayload> {
45+
/// Convert back to an unchecked type.
46+
pub fn into_unchecked(self) -> UncheckedSigned<Payload, RealPayload> {
47+
self.0
48+
}
49+
}
50+
4451
/// Unchecked signed data, can be converted to `Signed` by checking the signature.
4552
#[derive(Clone, PartialEq, Eq, RuntimeDebug, Encode, Decode, TypeInfo)]
4653
pub struct UncheckedSigned<Payload, RealPayload = Payload> {
@@ -253,6 +260,37 @@ impl<Payload: EncodeAs<RealPayload>, RealPayload: Encode> UncheckedSigned<Payloa
253260
Err(())
254261
}
255262
}
263+
264+
/// Sign this payload with the given context and pair.
265+
///
266+
/// # WARNING
267+
/// Only meant for usage in tests and and benchmarks.
268+
pub fn benchmark_sign<H: Encode>(
269+
public: &crate::v0::ValidatorId,
270+
payload: Payload,
271+
context: &SigningContext<H>,
272+
validator_index: ValidatorIndex,
273+
) -> Self {
274+
use application_crypto::RuntimeAppPublic;
275+
let data = Self::payload_data(&payload, context);
276+
let signature = public.sign(&data).unwrap();
277+
278+
Self { payload, validator_index, signature, real_payload: sp_std::marker::PhantomData }
279+
}
280+
281+
/// Immutably access the signature.
282+
///
283+
/// # WARNING
284+
/// Only meant for usage in tests and and benchmarks.
285+
pub fn benchmark_signature(&self) -> ValidatorSignature {
286+
self.signature.clone()
287+
}
288+
289+
/// Set the signature. Only should be used for creating testing mocks.
290+
#[cfg(feature = "std")]
291+
pub fn set_signature(&mut self, signature: ValidatorSignature) {
292+
self.signature = signature
293+
}
256294
}
257295

258296
impl<Payload, RealPayload> From<Signed<Payload, RealPayload>>

roadmap/implementers-guide/src/runtime/inclusion.md

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,13 +45,38 @@ PendingAvailabilityCommitments: map ParaId => CandidateCommitments;
4545
All failed checks should lead to an unrecoverable error making the block invalid.
4646

4747
* `process_bitfields(expected_bits, Bitfields, core_lookup: Fn(CoreIndex) -> Option<ParaId>)`:
48-
1. check that there is at most 1 bitfield per validator and that the number of bits in each bitfield is equal to `expected_bits`.
49-
1. check that there are no duplicates
50-
1. check all validator signatures.
48+
1. call `sanitize_bitfields<true>` and use the sanitized `signed_bitfields` from now on.
49+
1. call `sanitize_backed_candidates<true>` and use the sanitized `backed_candidates` from now on.
5150
1. apply each bit of bitfield to the corresponding pending candidate. looking up parathread cores using the `core_lookup`. Disregard bitfields that have a `1` bit for any free cores.
5251
1. For each applied bit of each availability-bitfield, set the bit for the validator in the `CandidatePendingAvailability`'s `availability_votes` bitfield. Track all candidates that now have >2/3 of bits set in their `availability_votes`. These candidates are now available and can be enacted.
5352
1. For all now-available candidates, invoke the `enact_candidate` routine with the candidate and relay-parent number.
5453
1. Return a list of `(CoreIndex, CandidateHash)` from freed cores consisting of the cores where candidates have become available.
54+
* `sanitize_bitfields<T: crate::inclusion::Config, const CHECK_SIGS: bool>(
55+
unchecked_bitfields: UncheckedSignedAvailabilityBitfields,
56+
disputed_bitfield: DisputedBitfield,
57+
expected_bits: usize,
58+
parent_hash: T::Hash,
59+
session_index: SessionIndex,
60+
validators: &[ValidatorId],
61+
)`:
62+
1. check that `disputed_bitfield` has the same number of bits as the `expected_bits`, iff not return early with an empty vec.
63+
1. each of the below checks is for each bitfield. If a check does not pass the bitfield will be skipped.
64+
1. check that there are no bits set that reference a disputed candidate.
65+
1. check that the number of bits is equal to `expected_bits`.
66+
1. check that the validator index is strictly increasing (and thus also unique).
67+
1. check that the validator bit index is not out of bounds.
68+
1. check the validators signature, iff `CHECK_SIGS=true`.
69+
70+
* `sanitize_backed_candidates<T: crate::inclusion::Config, F: Fn(CandidateHash) -> bool>(
71+
relay_parent: T::Hash,
72+
mut backed_candidates: Vec<BackedCandidate<T::Hash>>,
73+
candidate_has_concluded_invalid_dispute: F,
74+
scheduled: &[CoreAssignment],
75+
) `
76+
1. filter out any backed candidates that have concluded invalid.
77+
1. filter out backed candidates that don't have a matching `relay_parent`.
78+
1. filters backed candidates whom's paraid was scheduled by means of the provided `scheduled` parameter.
79+
5580
* `process_candidates(parent_storage_root, BackedCandidates, scheduled: Vec<CoreAssignment>, group_validators: Fn(GroupIndex) -> Option<Vec<ValidatorIndex>>)`:
5681
1. check that each candidate corresponds to a scheduled core and that they are ordered in the same order the cores appear in assignments in `scheduled`.
5782
1. check that `scheduled` is sorted ascending by `CoreIndex`, without duplicates.
@@ -78,6 +103,7 @@ All failed checks should lead to an unrecoverable error making the block invalid
78103
1. call `Hrmp::prune_hrmp` with the para id of the candiate and the candidate's `hrmp_watermark`.
79104
1. call `Hrmp::queue_outbound_hrmp` with the para id of the candidate and the list of horizontal messages taken from the commitment,
80105
1. Call `Paras::note_new_head` using the `HeadData` from the receipt and `relay_parent_number`.
106+
81107
* `collect_pending`:
82108

83109
```rust

roadmap/implementers-guide/src/runtime/parainherent.md

Lines changed: 41 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,19 @@ OnChainVotes: Option<ScrapedOnChainVotes>,
2929

3030
## Entry Points
3131

32-
* `enter`: This entry-point accepts three parameters: The relay-chain parent block header, [`Bitfields`](../types/availability.md#signed-availability-bitfield) and [`BackedCandidates`](../types/backing.md#backed-candidate).
33-
1. Hash the parent header and make sure that it corresponds to the block hash of the parent (tracked by the `frame_system` FRAME module),
32+
* `enter`: This entry-point accepts one parameter: [`ParaInherentData`](../types/runtime.md#ParaInherentData).
33+
1. Ensure the origin is none.
34+
1. Ensure `Included` is set as `None`.
35+
1. Set `Included` as `Some`.
36+
1. Unpack `ParachainsInherentData` into `signed_bitfields`, `backed_candidates`, `parent_header`, and `disputes`.
37+
1. Hash the parent header and make sure that it corresponds to the block hash of the parent (tracked by the `frame_system` FRAME module).
38+
1. Calculate the `candidate_weight`, `bitfields_weight`, and `disputes_weight`.
39+
1. If the sum of `candidate_weight`, `bitfields_weight`, and `disputes_weight` is greater than the max block weight we do the following with the goal of prioritizing the inclusion of disputes without making it game-able by block authors:
40+
1. clear `bitfields` and set `bitfields_weight` equal to 0.
41+
1. clear `backed_candidates` and set `candidate_weight` equal to 0.
42+
1. invoke `limit_disputes` on the `disputes` with the max block weight iff the disputes weight is greater than the max block weight.
3443
1. Invoke `Disputes::provide_multi_dispute_data`.
35-
1. If `Disputes::is_frozen`, return and set `Included` to `Some(())`.
44+
1. If `Disputes::is_frozen`, return.
3645
1. If there are any concluded disputes from the current session, invoke `Inclusion::collect_disputed` with the disputed candidates. Annotate each returned core with `FreedReason::Concluded`, sort them, and invoke `Scheduler::free_cores` with them.
3746
1. The `Bitfields` are first forwarded to the `Inclusion::process_bitfields` routine, returning a set included candidates and the respective freed cores. Provide the number of availability cores (`Scheduler::availability_cores().len()`) as the expected number of bits and a `Scheduler::core_para` as a core-lookup to the `process_bitfields` routine. Annotate each of these freed cores with `FreedReason::Concluded`.
3847
1. For each freed candidate from the `Inclusion::process_bitfields` call, invoke `Disputes::note_included(current_session, candidate)`.
@@ -48,3 +57,32 @@ OnChainVotes: Option<ScrapedOnChainVotes>,
4857
1. Call `Scheduler::occupied` using the `occupied` core indices of the returned above, first sorting the list of assigned core indices.
4958
1. Call the `Ump::process_pending_upward_messages` routine to execute all messages in upward dispatch queues.
5059
1. If all of the above succeeds, set `Included` to `Some(())`.
60+
61+
62+
* `create_inherent`: This entry-point accepts one parameter: `InherentData`.
63+
1. Invoke [`create_inherent_inner(InherentData)`](#routines), the unit testable logic for filtering and sanitzing the inherent data used when invoking `enter`. Save the result as `inherent_data`.
64+
1. If the `inherent_data` is an `Err` variant, return the `enter` call signature with all inherent data cleared else return the `enter` call signature with `inherent_data` passed in as the `data` param.
65+
66+
# Routines
67+
68+
* `create_inherent_inner(data: &InherentData) -> Option<ParachainsInherentData<T::Header>>`
69+
1. Unpack `InherentData` into its parts, `bitfields`, `backed_candidates`, `disputes` and the `parent_header`. If data cannot be unpacked return `None`.
70+
1. Hash the `parent_header` and make sure that it corresponds to the block hash of the parent (tracked by the `frame_system` FRAME module).
71+
1. Invoke `Disputes::filter_multi_dispute_data` to remove duplicates et al from `disputes`.
72+
1. Run the following within a `with_transaction` closure to avoid side effects (we are essentially replicating the logic that would otherwise happen within `enter` so we can get the filtered bitfields and the `concluded_invalid_disputes` + `scheduled` to use in filtering the `backed_candidates`.):
73+
1. Invoke `Disputes::provide_multi_dispute_data`.
74+
1. Collect `current_concluded_invalid_disputes`, the disputed candidate hashes from the current session that have concluded invalid.
75+
1. Collect `concluded_invalid_disputes`, the disputed candidate hashes from the given `backed_candidates`.
76+
1. Invoke `Inclusion::collect_disputed` with the newly disputed candidates. Annotate each returned core with `FreedReason::Concluded`, sort them, and invoke `Scheduler::free_cores` with them.
77+
1. Collect filtered `bitfields` by invoking [`sanitize_bitfields<false>`](inclusion.md#Routines).
78+
1. Collect `freed_concluded` by invoking `update_pending_availability_and_get_freed_cores` on the filtered bitfields.
79+
1. Collect all `freed` cores by invoking `collect_all_freed_cores` on `freed_concluding`.
80+
1. Invoke `scheduler::Pallet<T>>::clear()`.
81+
1. Invoke `scheduler::Pallet<T>>::schedule` with `freed` and the current block number to create the same schedule of the cores that `enter` will create.
82+
1. Read the new `<scheduler::Pallet<T>>::scheduled()` into `schedule`.
83+
1. From the `with_transaction` closure return `concluded_invalid_disputes`, `bitfields`, and `scheduled`.
84+
1. Invoke `sanitize_backed_candidates` using the `scheduled` return from the `with_transaction` and pass the closure `|candidate_hash: CandidateHash| -> bool { DisputesHandler::concluded_invalid(current_session, candidate_hash) }` for the param `candidate_has_concluded_invalid_dispute`.
85+
1. create a `rng` from `rand_chacha::ChaChaRng::from_seed(compute_entropy::<T>(parent_hash))`.
86+
1. Invoke `limit_disputes` with the max block weight and `rng`, storing the returned weigh in `remaining_weight`.
87+
1. Fill up the remaining of the block weight with backed candidates and bitfields by invoking `apply_weight_limit` with `remaining_weigh` and `rng`.
88+
1. Return `Some(ParachainsInherentData { bitfields, backed_candidates, disputes, parent_header }`.

roadmap/implementers-guide/src/types/runtime.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,15 +116,17 @@ struct HostConfiguration {
116116

117117
Inherent data passed to a runtime entry-point for the advancement of parachain consensus.
118118

119-
This contains 3 pieces of data:
119+
This contains 4 pieces of data:
120120
1. [`Bitfields`](availability.md#signed-availability-bitfield)
121121
2. [`BackedCandidates`](backing.md#backed-candidate)
122122
3. [`MultiDisputeStatementSet`](disputes.md#multidisputestatementset)
123+
4. `Header`
123124

124125
```rust
125126
struct ParaInherentData {
126127
bitfields: Bitfields,
127128
backed_candidates: BackedCandidates,
128129
dispute_statements: MultiDisputeStatementSet,
130+
parent_header: Header
129131
}
130132
```

0 commit comments

Comments
 (0)