v1.18: Use poh grace ticks when new reset bank is pending (backport of #794)

[mergify]

Problem

I've observed that a lot of skipped slots are caused when a leader "A" tries to skip the previous leader "B" when producing its blocks but the previous leader had actually already started broadcasting shreds for its blocks. In this case, there's a race between the forks of leader A and B and it's almost certain that leader B's fork will be confirmed because its block will likely be finished and replayed by the cluster before leader A's block that only just started being produced.

Summary of Changes

Currently when leader A skips all of leader B's blocks, it doesn't use grace ticks when deciding when to start building its block. After this PR, when a leader skips all of the previous leader's blocks and has ticked to its leader slot (without grace ticks) it will first check if it has received any shreds for a potential new reset bank. If it has received some shreds, it will apply grace ticks to wait a bit longer to allow time for the corresponding bank for those shreds to be frozen (or marked dead). If it hasn't received any shreds, it can go ahead with producing its block without waiting for grace ticks as before.

• Added new hidden validator cli arg --delay-leader-block-for-pending-fork which allows validators to opt into this new behavior.
• Added new metric poh_recorder-detected_pending_fork which reports when a pending fork was detected and whether or not the validator yielded

Fixes #

---

This is an automatic backport of pull request #794 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 1c1b4c3 has failed:

On branch mergify/bp/v1.18/pr-794
Your branch is up to date with 'origin/v1.18'.

You are currently cherry-picking commit 1c1b4c3e28.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   core/src/replay_stage.rs
	modified:   poh/src/poh_recorder.rs
	modified:   validator/src/cli.rs

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   core/src/validator.rs
	both modified:   local-cluster/src/validator_configs.rs
	both modified:   validator/src/main.rs

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 94.82759% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 81.6%. Comparing base (d7eb8f9) to head (a518a55).

Additional details and impacted files

@@           Coverage Diff            @@
##            v1.18     #886    +/-   ##
========================================
  Coverage    81.5%    81.6%            
========================================
  Files         827      827            
  Lines      224848   224903    +55     
========================================
+ Hits       183453   183565   +112     
+ Misses      41395    41338    -57     

[jstarry]

@AshwinSekar @carllin I'm still waiting on a review here

[AshwinSekar]

LGTM, sorry for the delay - slipped through my github notifications.

[jstarry]

v1.18 is basically prod now that mainnet beta is upgrading so I'm going to close this

[AshwinSekar]

v1.18 is basically prod now that mainnet beta is upgrading so I'm going to close this

If we want to speed up experimentation, I think it could be worth it to acquire a couple of staked canaries to run with this flag. That would give us a couple months of confidence so that we can make this the default choice in 2.0.0.

[jstarry]

I've been running this on a staked node for months already and it works great

[AshwinSekar]

I've been running this on a staked node for months already and it works great

oh awesome. If that's the case, can we backport the metric only? That should give us enough data to do a comparison between 1.18 and your staked node.

If the comparison looks good we can make this the default choice in 2.0, no flag necessary.