Make ReplayStage own the threadpool for tx replay

[steviez]

Problem

The threadpool used to replay multiple transactions in parallel is
currently global state via a lazy_static definition. Making this pool
owned by ReplayStage will enable subsequent work to make the pool
size configurable on the CLI.

More context in #105

Summary of Changes

Make ReplayStage create the threadpool and pass it down to blockstore_processor

[steviez]

This PR maintains the existing thread-pool size for non-test cases. That being said, I noticed this line that will probably need to be evaluated when the change to reduce this threadpool size comes up

agave/ledger/src/blockstore_processor.rs

Line 450 in 5c1df15

let target_batch_count = get_thread_count() as u64;

[steviez]

These functions as well as ReplayStage are in need of some cleanup. One easy win I think would be packing all the stuff that only gets accessed by the bottom layer of the call-stack (execute_batch()) into a struct and unpacking the struct in the bottom layer instead of passing each individual piece in every function.

That being said, I would like to defer that work after this PR - soon enough, work on this project will be lots of experiments / data collection. At that point, I'll have more "idle" time that I could come back to perform this cleanup if there is nothing else more pressing for me

[bw-solana]

I'll have more "idle" time

you wish

[steviez]

Another pro of this change is that we'll be able to configure this value to be different size for startup and steady-state. That's good because startup involves replaying entire blocks that are already in ledger, so higher amount of parallelism might be warranted vs. steady-state when blocks are gradually streamed over

[bw-solana]

Makes sense. Looks like we're using 1 thread per core already, so opportunity seems to be dialing this down for steady state

[steviez]

so opportunity seems to be dialing this down for steady state

Exactly - already have a good bit of data to support this being the correct decision in #25

[bw-solana]

LGTM

[t-nelson]

should we not loft this even higher so banking stage can use the same pool while replay is idle?

[steviez]

should we not loft this even higher so banking stage can use the same pool while replay is idle?

Maybe ? If I recall correctly, I thought BankingStage started doing some work prior to its' leader slots actually starting. Something like organization of tx's so that it things are primed when the leader slots actually start. If that is correct, sharing the pool could have some adverse effects, ie slowing down replay just prior / slowing down the banking stage "priming".

I can followup with someone in the morning. I also already plan on attempting to use the same pool for entry verification and tx replay; that should be a pretty good gain. And comparatively, BankingStage isn't as abusive on thread counts (can't say the same for the tx ingestion pipeline) so we could also keep things separate for now to claim the low hanging fruit, and then circle back to try to further condense

[steviez]

should we not loft this even higher so banking stage can use the same pool while replay is idle?

Maybe ? If I recall correctly, I thought BankingStage started doing some work prior to its' leader slots actually starting. Something like organization of tx's so that it things are primed when the leader slots actually start. If that is correct, sharing the pool could have some adverse effects, ie slowing down replay just prior / slowing down the banking stage "priming".

I can followup with someone in the morning...

I talked to @apfitzge, and the short answer is yes, there appears to be some work that BankingStage is doing prior to leader slots actually starting. I will have to defer to him to go into more detail, but given that, I'm inclined to push this PR as-is and pursue any combination of resources between ReplayStage and BankingStage in the future

[apfitzge]

I think it could probably be done, but will need to be careful with how we do that. BankingStage still does work before becoming leader:

1. receiving packets from sigverify
2. sanitizing transactions
3. buffering transactions (priority ordered - O(nlogn) operation!)
4. clean buffer - removes txs that were already processed or expired

We probably could make the ConsumeWorkers in the new central-scheduler mode of banking stage use a shared thread-pool but not the scheduler thread itself.
Workers should be sleeping (blocking crossbeam receive) except when we are leader - scheduler thread has sent them work.