Handle errors on replaying ledger properly

[carllin]

Problem

Problem 1:

When a slot is marked dead, https://github.com/solana-labs/solana/blob/v0.21.5/ledger/src/blocktree_processor.rs#L509, or when an error occurs on replay: https://github.com/solana-labs/solana/blob/v0.21.5/ledger/src/blocktree_processor.rs#L520
continues.

This means https://github.com/solana-labs/solana/blob/v0.21.5/ledger/src/blocktree_processor.rs#L542-L549 (where the tip of the fork is added to fork_info, never runs for the end of that fork, so this logic doesn't run: https://github.com/solana-labs/solana/blob/v0.21.5/ledger/src/blocktree_processor.rs#L467, and then the parents of that fork don't get added on BankForks construction here: https://github.com/solana-labs/solana/blob/v0.21.5/ledger/src/bank_forks.rs#L146

This causes this slot to be replayed again in ReplayStage, leading to DuplicateSignature errors.

Problem 2:

In this else block: https://github.com/solana-labs/solana/blob/v0.21.5/ledger/src/blocktree_processor.rs#L463-L467
, if any child of a slot P is incomplete, P is added to the fork_info structure even if it isn't the tip of a fork.

For instance:

      A
   /      \
 B       C (incomplete)

Bank A is added to fork_info even though the tip of the fork is B.

Summary of Changes

Fix problems 1) and 2) above with better error handling and restructuring the processing pipeline by always adding a bank to fork_info, and having any successfully replayed children delete the immediate parent from fork_info (thanks @sakridge for the suggestion!)

Add tests for above

Fixes #

[carllin]

@jstarry This is where having the replay and blocktree processor verification be the same would be useful.

If we don't mark this slot as dead here on failure, then ReplayStage will try to replay this slot again and will run into DuplicateSignature errors.

If they were the same implementation, then we know that if this fails here, it will also fail in ReplayStage, so it would be safe to mark the slot as dead here. Otherwise, if the implementations/criteria differ, one validator might mark one as dead, another won't depending on if they replay here on boot or in ReplayStage.

[codecov]

Codecov Report

Merging #7741 into master will increase coverage by <.1%.
The diff coverage is 98%.

@@           Coverage Diff            @@
##           master   #7741     +/-   ##
========================================
+ Coverage    81.7%   81.7%   +<.1%     
========================================
  Files         241     241             
  Lines       50750   50814     +64     
========================================
+ Hits        41489   41564     +75     
+ Misses       9261    9250     -11

[sakridge]

lgtm

[mvines]

Umm,

failures:
--
  | blocktree_processor::tests::test_process_blocktree_with_dead_child
  | blocktree_processor::tests::test_process_blocktree_with_dead_slot
  | blocktree_processor::tests::test_root_with_all_dead_children