Explicit sync API for downloading important, possibly orphaned, forks #3633

arkpar · 2019-09-17T14:01:11Z

Sync services provides an API to download specified block chains explicitly.

rphmeier · 2019-09-17T14:25:52Z

core/network/src/protocol/sync.rs

+		Vec<(PeerId, BlockRequest<B>)>
+	{
+		debug!(target: "sync", "Explicit sync request for block {:?} with {:?}", hash, peers);
+		if self.is_known(&hash) || self.is_already_downloading(&hash) {


I'd say we need something a bit more explicit. The fork-sync requests should be kept around forever until removed or the block is actually known, and the is_already_downloading check should be done periodically. We can imagine a case where is_already_downloading is true at the moment, but then for some reason the node stops downloading that block. Then our requested block still wouldn't be synced.

rphmeier · 2019-09-17T14:26:52Z

core/network/src/protocol/sync.rs

+
+		let block_status = self.client.block_status(&BlockId::Number(number - One::one()))
+			.unwrap_or(BlockStatus::Unknown);
+		if block_status == BlockStatus::InChainPruned {


As a follow-up, there should be a special mode where we just download the headers for this chain if requested, but we don't have the state to process them.

#3639 for a more detailed writeup of ^

rphmeier · 2019-09-17T14:27:49Z

It's also missing the APIs for higher-level code to inform the sync service about which peers it knows has the block. It should have add/remove capability.

arkpar · 2019-09-17T15:00:12Z

It's also missing the APIs for higher-level code to inform the sync service about which peers it knows has the block.

But sync_fork does exactly that.
What's the case for remove? Peer forgets a block?
Sync tries to download all forks that other peers have and we don't. It does not matter if GRANDPA is interested in the block or not. Do you suggest changing that?

rphmeier · 2019-09-17T15:15:42Z

The set may change over time, is what I mean - are multiple calls guaranteed to alter the existing "download session" so that it will sync the fork from the union of the given peer-sets?

arkpar · 2019-09-17T16:13:24Z

No, but I'm going to implement that along with request persistence.

rphmeier · 2019-09-17T16:24:12Z

It does not matter if GRANDPA is interested in the block or not. Do you suggest changing that?

In general, I suggest sync being a little bit smarter about figuring out what "all forks" are based on what GRANDPA is interested in. Even if the fork is no longer active it should constantly be trying to sync it, even if attempts fail.

rphmeier · 2019-09-17T16:24:55Z

core/network/src/protocol/sync.rs

+
+			if number > peer.best_number {
+				peer.best_number = number;
+				peer.best_hash = hash.clone();


why set that to the peer's best block? i don't see this working when there is 1 peer with 2 forks we want to download, if that's a pre-requisite to being able to download those blocks.

best_number is used to track global sync target. best_hash is not used for anything really, other than reporting generic peer info.

arkpar · 2019-09-21T08:00:17Z

Fork download requests are now persistent. Sync will continue trying even if peer reconnects.

rphmeier · 2019-09-21T17:15:16Z

core/network/src/protocol.rs

 				message::Direction::Ascending => id = BlockId::Number(number + One::one()),
 				message::Direction::Descending => {
-					if number.is_zero() {
+					if number.is_one() {


if it is zero, will that cause an error further down the line when parent_hash gives None for the header/block?

parent_hash is not an Option. for the genesis block it is a zero hash.

That change is still wrong though. We should allow downloading the genesis block in principle.

rphmeier · 2019-09-21T17:16:04Z

core/network/src/protocol/sync.rs

 		if block_status == BlockStatus::InChainPruned {
 			trace!(target: "sync", "Refusing to sync ancient block {:?}", hash);
-			return Vec::default();
+			return;


this is also not the only kind of ancient fork we can encounter - what happens / is supposed to happen when the common ancestor is more than one block back?

Sync attempts to download the chain up to the last finalized block, not just one block. There's currently a limit of how many blocks these forks thare downloaded up the chain can be. We can't be allowed to download infinite chains of headers/blocks without validating them.

Sync attempts to download the chain up to the last finalized block, not just one block

I don't understand what you mean. Yes, there can be a practical cutoff somewhere, and I think refusing to download forks beyond finalized_height + constant_allowance would be reasonable (with constant_allowance around 1000).

This allowance needs to be really permissive. Our nodes can afford a few extra MB of RAM if it means that you reduce the likelihood of consensus stalling by several orders of magnitude. And if consensus does stall, I don't really see any issue with downloading "infinitely" long chains as a light client, as long as it's in incremental parts, since there's nothing better the node could be doing and that would at least be trying to unstick consensus. The chains in practice aren't going to be infinitely long, as long as the block production mechanism is sane. If it's insane, then it's not the job of the sync code to work around that.

But the most recent finalized block of the fork can at most be the common ancestor (otherwise it wouldn't be a fork). So we always have to download beyond that.

rphmeier · 2019-09-21T17:17:35Z

core/network/src/protocol/sync.rs

-				if let Some(request) = self.download_unknown_stale(&peer_id, &hash) {
-					requests.push((peer_id, request));
+				if number > peer.best_number {
+					peer.best_number = number;


What is the invariant that sync is meant to uphold?

Is this always supposed to be the peer's highest block, or is it the peer's best chain? We do have some non longest-chain fork choice rules (1. follow finality, 2. aurababous), so they are not always the same thing.

This is just the highest block currently. The sync tracks the longest fork a bit differently from the shorter forks. E.g. it supports parallel block fetch for the longest chain only. It still fetches all forks though so it should not matter which one is actually "best". There will be a refactoring to get rid of the "main" chain notion, but this is out of scope of this PR.

rphmeier · 2019-09-21T17:18:53Z

core/network/src/protocol/sync.rs

+	requests: &HashMap<B::Hash, SyncRequest<B>>,
+	best_num: NumberFor<B>,
+	finalized: &B::Hash,
+	attributs: &message::BlockAttributes,


typo: attributes?

rphmeier · 2019-09-21T17:19:25Z

core/network/src/protocol/sync.rs

+		if !r.peers.contains(id) {
+			continue
+		}
+		if r.number <= best_num {


questionable for the same reason as before - the orphan chain may be longer than the peer's "best" chain because of the fork-choice rule.

The other case (r.number > best_num) is handled in a different code path in peer_block_request.

This is confusing. Why is the logic so scattered? I feel like I need a PhD to understand all the side effects and branches

What I mean is that there's a lot of implicit "but actually" in the interface of this function.

It's an explicit sync request...but actually only returns explicit requests for chains shorter than the best.

So the peer doesn't download that chain...but actually it does, it's covered in another function in another file (peer_block_request defers to needed_blocks)

I'd recommend that the function return a special type when there is implicit context that needs to be passed from the execution of this function into the execution of another, where the reader of the code can clearly see that the handling of some case has been deferred beyond the execution of the function.

This is fundamentally caused by the same issue described here: #3633 (comment)
To be refactored.

rphmeier · 2019-09-21T17:21:33Z

Looks good but needs accounting for fork-choice rules

…explicit

arkpar · 2019-09-24T14:31:40Z

After a chat with @rphmeier we've agreed to address all remaining issues in the following PRs

mxinden

I am still getting familiar with the code, so please interpret some of these comments as questions instead of suggestions.

mxinden · 2019-09-27T09:33:07Z

core/network/src/protocol/sync.rs

+	/// Request syncing for the given block from given set of peers.
+	// This is similar to on_block_announce with unknown parent hash.
+	pub fn sync_fork(&mut self, peers: Vec<PeerId>, hash: &B::Hash, number: NumberFor<B>) {
+		if peers.is_empty() {


Do I understand correctly that calling sync_fork with an empty peers removes it from the sync_requests set? This is not what I would have expected from the function name. How about renaming it to set_sync_request or moving the logic to a separate function cancel_sync_request?

Please ignore this in case this is a pattern within Substrate.

renamed to set_fork_sync_request

mxinden · 2019-09-27T09:47:47Z

core/network/src/protocol/sync.rs

 	}

+	/// Request syncing for the given block from given set of peers.
+	// This is similar to on_block_announce with unknown parent hash.


Suggested change

// This is similar to on_block_announce with unknown parent hash.

/// This is similar to on_block_announce with unknown parent hash.

I feel like this could be a doc comment as well, right?

This comment is about the implementation detail, not the interface.

mxinden · 2019-09-27T10:19:22Z

core/network/src/service.rs

 		Ok(())
 	}

+	/// Adds a `PeerId` and its address as reserved.


Suggested change

/// Adds a `PeerId` and its address as reserved.

This is a copy/paste error from above, right?

…header-chain

Explicit sync API

42fe308

arkpar added the A0-please_review Pull request needs code review. label Sep 17, 2019

arkpar mentioned this pull request Sep 17, 2019

GRANDPA: extend neighbor packet to ensure vote-blocks are picked up by sync #3629

Open

rphmeier reviewed Sep 17, 2019

View reviewed changes

rphmeier added A5-grumble and removed A0-please_review Pull request needs code review. labels Sep 17, 2019

rphmeier reviewed Sep 17, 2019

View reviewed changes

arkpar added A3-in_progress Pull request is in progress. No review needed at this stage. and removed A5-grumble labels Sep 17, 2019

rphmeier changed the title ~~Explicit sync API~~ Explicit sync API for downloading important, possibly orphaned, forks Sep 18, 2019

rphmeier mentioned this pull request Sep 18, 2019

Sync cannot download explicitly requested forks as light client if necessary #3639

Closed

Keep sync requests

97e18d9

arkpar added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Sep 21, 2019

rphmeier reviewed Sep 21, 2019

View reviewed changes

rphmeier added A5-grumble and removed A0-please_review Pull request needs code review. labels Sep 21, 2019

Don't request the finalized block we already have

3815130

arkpar added A0-please_review Pull request needs code review. and removed A5-grumble labels Sep 22, 2019

rphmeier added A5-grumble and removed A0-please_review Pull request needs code review. labels Sep 23, 2019

arkpar added 2 commits September 24, 2019 15:44

Merge branch 'master' of github.com:paritytech/substrate into a-sync-…

7bcffcd

…explicit

Dropping requests & docs

d687a54

arkpar added A0-please_review Pull request needs code review. and removed A5-grumble labels Sep 24, 2019

mxinden reviewed Sep 27, 2019

View reviewed changes

arkpar added 2 commits September 27, 2019 16:50

Renamed a function

19673f7

Merge branch 'master' of github.com:paritytech/substrate into a-sync-…

34a88eb

…header-chain

rphmeier approved these changes Sep 28, 2019

View reviewed changes

rphmeier merged commit 4f850d3 into master Sep 28, 2019

rphmeier deleted the a-sync-explicit branch September 28, 2019 12:02

mxinden mentioned this pull request Oct 3, 2019

core/finality-grandpa: Gossip synced blocks via neighbor packets #3755

Closed

mxinden mentioned this pull request Oct 11, 2019

core/finality-grandpa: Request block sync from network after import timeout #3800

Merged

	// This is similar to on_block_announce with unknown parent hash.
	/// This is similar to on_block_announce with unknown parent hash.

Explicit sync API for downloading important, possibly orphaned, forks #3633

Explicit sync API for downloading important, possibly orphaned, forks #3633

Uh oh!

Conversation

arkpar commented Sep 17, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier commented Sep 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arkpar commented Sep 17, 2019

Uh oh!

rphmeier commented Sep 17, 2019

Uh oh!

arkpar commented Sep 17, 2019

Uh oh!

rphmeier commented Sep 17, 2019

Uh oh!

rphmeier Sep 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arkpar commented Sep 21, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier Sep 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier Sep 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier Sep 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier Sep 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier commented Sep 21, 2019

Uh oh!

arkpar commented Sep 24, 2019

Uh oh!

mxinden left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier commented Sep 17, 2019 •

edited

Loading

rphmeier Sep 17, 2019 •

edited

Loading

rphmeier Sep 21, 2019 •

edited

Loading

rphmeier Sep 23, 2019 •

edited

Loading

rphmeier Sep 23, 2019 •

edited

Loading

rphmeier Sep 23, 2019 •

edited

Loading

arkpar Sep 27, 2019 •

edited

Loading