Add in metrics for detecting Redundant Pulls #199

gregcusack · 2024-03-12T01:57:37Z

Problem

We had previously added in a metric for tracking gossip push messages through the network in PR: #32725. However, this metric does not account for redundant pull requests.
Redundant Pull: A node receives a message via PullResponse and then receives the same message via Push.
Redundant Pulls prevent us from accurately calculating how well messages are propagating via Push.

Summary of Changes

Add in a metric to report when we receive a Push for a message we already (and first) received via PullResponse

num_push_dups changed to num_push_recv and now tracks the number of times we have received a push message.
add a new metric to cluster_info_crds_stats called num_redundant_pull_responses. It counts the number of times a unique message is received via a Redundant Pull

Identifying redundant Pulls:

Receive a message via PullRequest that successfully updates crds.table, set the num_push_recv of this message to 0. Set num_push_recv to 1 if it is a PushMessage
Receive the same message again but via Push, it will fail to insert. Since the already existing entry has num_push_recv == 0, we know this is a Redundant Pull.
Increment CrdsStats.num_redundant_pull_responses

Calculating fraction of messages received via Redundant Pull:

num_redundant_pull_responses / (num_redundant_pull_responses + all-push)

Monogon Simulation: % of redundant pulls in a 100 validator cluster

We see a mean Redundant Pull percentage of ~0.2%. But it does seem to increase with the number of validators

gregcusack · 2024-03-12T03:22:49Z

gossip/src/crds_gossip_push.rs

+                    Err(CrdsError::RedundantPull) => {
+                        // rewarded for being after Pull but first Push??
+                        received_cache.record(origin, from, /*num_dups:*/ 0);
+                        // still track that it is old message
+                        self.num_old.fetch_add(1, Ordering::Relaxed);
+                    }


Question: should the peer that sends the first push message that triggers the discovery of a redundant pull be treated the same in ReceivedCache as the peer who sends the first push message resulting in a successful update of the crds.table? I would argue the answer is yes they should be treated the same:
Reasoning: a peer shouldn't be punished just because a PullRequest happened to beat it in reporting a datapoint. This also means the second peer would still gets its scored increased in ReceivedCache since NUM_DUPS_THRESHOLD == 2. Even though the peer may be slow it is still faster than any peer sending this same message after it.

Possible argument against: Technically the peer wasn't the first one and this is a duplicate. There may be a reason besides chance that a PullRequest beat the Push.

should the peer that sends the first push message that triggers the discovery of a redundant pull be treated the same in ReceivedCache as the peer who sends the first push message resulting in a successful update of the crds.table?

yes, I think so.

Possible argument against: Technically the peer wasn't the first one and this is a duplicate. There may be a reason besides chance that a PullRequest beat the Push.

I can't think of a good reason here. Basically if there was no outgoing pull-request this value would have been updated from push anyways.

codecov-commenter · 2024-03-12T03:23:47Z

Codecov Report

Attention: Patch coverage is 95.65217% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 81.9%. Comparing base (f8bb98b) to head (1d41d97).
Report is 17 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master     #199     +/-   ##
=========================================
- Coverage    81.9%    81.9%   -0.1%     
=========================================
  Files         837      837             
  Lines      226539   226555     +16     
=========================================
- Hits       185574   185573      -1     
- Misses      40965    40982     +17

gregcusack · 2024-03-12T03:24:20Z

gossip/src/crds.rs

+            // If we received a PushMessage, set
+            // num_push_recv <- 1
+            GossipRoute::PushMessage(_) => 1u8,
+            _ => 0u8,


do we have to pay attention to LocalMessage or PullRequest here? At the very least I think we can ignore PullRequest since we're not updating our crds.table when we receive a PullRequest

we're not updating our crds.table when we receive a PullRequest

We actually upsert the origin of PullRequest into crds table:
https://github.com/anza-xyz/agave/blob/0f1ca20d3/gossip/src/crds_gossip_pull.rs#L296

My initial thought was that we can just leave it zero anyways, but looks like it might break record_redundant_pull below.

An ugly workaround would be define:

num_push_recv: Option<u8>,

Then in VersionedCrdsValue::new:

let num_push_recv = match route { GossipRoute::LocalMessage => None, GossipRoute::PullRequest => None, GossipRoute::PullResponse => Some(0), GossipRoute::PushMessage => Some(1), }

Then Some(0) will accurately identify redundant pulls.

ahh shoot missed that. ya not an ideal option but would work. i'll think...

behzadnouri · 2024-03-12T14:08:53Z

gossip/src/crds.rs

+            // If we received a PushMessage, set
+            // num_push_recv <- 1
+            GossipRoute::PushMessage(_) => 1u8,
+            _ => 0u8,


we're not updating our crds.table when we receive a PullRequest

We actually upsert the origin of PullRequest into crds table:
https://github.com/anza-xyz/agave/blob/0f1ca20d3/gossip/src/crds_gossip_pull.rs#L296

My initial thought was that we can just leave it zero anyways, but looks like it might break record_redundant_pull below.

An ugly workaround would be define:

num_push_recv: Option<u8>,

Then in VersionedCrdsValue::new:

let num_push_recv = match route { GossipRoute::LocalMessage => None, GossipRoute::PullRequest => None, GossipRoute::PullResponse => Some(0), GossipRoute::PushMessage => Some(1), }

Then Some(0) will accurately identify redundant pulls.

behzadnouri · 2024-03-12T14:17:20Z

gossip/src/crds.rs

+                    // If num_push_recv == 0, we know this data was first received
+                    // via PullResponse. This is a redundant pull, meaning we received
+                    // a PushMessage after we had already received the same message
+                    // first via PullResponse


I think this is not true if the value was upserted from GossipRoute::LocalMessage or GossipRoute::PullRequest as in:
https://github.com/anza-xyz/agave/blob/0f1ca20d3/gossip/src/crds_gossip_pull.rs#L296
So this will over count redundant pull messages.

behzadnouri · 2024-03-12T14:28:04Z

gossip/src/crds_gossip_push.rs

                        received_cache.record(origin, from, /*num_dups:*/ usize::MAX);
                        self.num_old.fetch_add(1, Ordering::Relaxed);
                    }
+                    Err(CrdsError::RedundantPull) => {


If we are not distinguishing between CrdsError::DuplicatePush(0) and CrdsError::RedundantPull maybe we don't need to add CrdsError::RedundantPull anyways.
The 0 in CrdsError::DuplicatePush(0) should indicate this was received from push after it was upserted from pull-responses.

oh you are right.

behzadnouri · 2024-03-12T14:29:45Z

gossip/src/crds_gossip_push.rs

+                    Err(CrdsError::RedundantPull) => {
+                        // rewarded for being after Pull but first Push??
+                        received_cache.record(origin, from, /*num_dups:*/ 0);
+                        // still track that it is old message
+                        self.num_old.fetch_add(1, Ordering::Relaxed);
+                    }


should the peer that sends the first push message that triggers the discovery of a redundant pull be treated the same in ReceivedCache as the peer who sends the first push message resulting in a successful update of the crds.table?

yes, I think so.

Possible argument against: Technically the peer wasn't the first one and this is a duplicate. There may be a reason besides chance that a PullRequest beat the Push.

I can't think of a good reason here. Basically if there was no outgoing pull-request this value would have been updated from push anyways.

behzadnouri · 2024-03-12T14:37:44Z

gossip/src/crds.rs

+    fn record_redundant_pull(&mut self) {
+        self.redundant_pull += 1;
+    }


feels a bit unnecessary to add a fn for a one-liner which is only used one place.

ok i can just increment directly

gregcusack · 2024-03-12T21:39:26Z

I ended up wrapping num_push_recv into a struct to make it a little more clear what is happening in the actual crds.insert(..) block. Similar sentiment as to what you had here but without an Option: #199 (comment)

impl NumPushRecv {
    fn new(route: GossipRoute) -> Self {
        let (count, received_first_via_pull_request) = match route {
            GossipRoute::PullRequest => (0, true),
            GossipRoute::PushMessage(_) => (1, false),
            _ => (0, false),
        };
        Self {
            count,
            received_first_via_pull_response: received_first_via_pull_request,
        }
    }

    // If first time message was received via PullResponse and count == 0,
    // we know this data was first received via PullResponse.
    // This is a redundant pull, meaning we received a PushMessage
    // after we had already received the same message first via PullResponse
    fn is_redundant_pull(&self) -> bool {
        self.received_first_via_pull_response && self.count == 0
    }
}

…and CrdsError::DuplicatePush

behzadnouri · 2024-03-13T15:25:57Z

gossip/src/crds.rs

    // Indices of EpochSlots keyed by insert order.
    epoch_slots: BTreeMap<u64 /*insert order*/, usize /*index*/>,
-    // Indices of DuplicateShred keyed by insert order.
+    // Indice of DuplicateShred keyed by insert order.


was this change intentional?

no it was not 🤦

behzadnouri · 2024-03-13T15:30:32Z

gossip/src/cluster_info_metrics.rs

        ),
        ("RestartHeaviestFork-push", crds_stats.push.counts[13], i64),
        ("RestartHeaviestFork-pull", crds_stats.pull.counts[13], i64),
+        ("RedundantPull", crds_stats.redundant_pull, i64),


Can we actually move this next to other pull_response_... metrics?
https://github.com/anza-xyz/agave/blob/81075e60b/gossip/src/cluster_info_metrics.rs#L293-L317
We can name it something like num_redundant_pull_responses.

behzadnouri · 2024-03-13T15:31:43Z

gossip/src/crds.rs

 pub(crate) struct CrdsStats {
    pub(crate) pull: CrdsDataStats,
    pub(crate) push: CrdsDataStats,
+    pub(crate) redundant_pull: i64,


num_redundant_pull_responses is probably more descriptive.
Also maybe add a comment what does this metric mean.

Also u64 is a better type since this value is not meant to be negative.

behzadnouri · 2024-03-13T15:45:28Z

gossip/src/crds.rs

+        let (count, received_first_via_pull_response) = match route {
+            GossipRoute::PullRequest => (0, true),
+            GossipRoute::PushMessage(_) => (1, false),
+            _ => (0, false),


The problem with _ => ... is that if a new variant is added to the GossipRoute enum, then this code silently ignores the new variant which might not be the right thing.
Whereas if you list all the remaining variants explicitly, you get a compile error if a new variant is added to the enum, which would require to explicitly handle that new variant here.
https://rust-lang.github.io/rust-clippy/master/index.html#/wildcard_enum_match_arm

this is a very good point. thanks for the info. will avoid in the future (and fix now)

behzadnouri · 2024-03-13T15:56:59Z

gossip/src/crds.rs

+#[derive(PartialEq, Eq, Debug, Clone)]
+struct NumPushRecv {
+    pub count: u8,
+    pub received_first_via_pull_response: bool,
+}
+
+impl NumPushRecv {
+    fn new(route: GossipRoute) -> Self {
+        let (count, received_first_via_pull_response) = match route {
+            GossipRoute::PullRequest => (0, true),
+            GossipRoute::PushMessage(_) => (1, false),
+            _ => (0, false),
+        };
+        Self {
+            count,
+            received_first_via_pull_response,
+        }
+    }
+
+    // If first time message was received via PullResponse and count == 0,
+    // we know this data was first received via PullResponse.
+    // This is a redundant pull, meaning we received a PushMessage
+    // after we had already received the same message first via PullResponse
+    fn is_redundant_pull(&self) -> bool {
+        self.received_first_via_pull_response && self.count == 0
+    }


I feel like this is a bit overkill/verbose for a metric.
Can we just use an Option<u8> to avoid verbosity for now and revise later if that was not sufficient?

ya that was my worry. no problem.

…ting

gregcusack · 2024-03-13T21:37:24Z

gossip/src/crds.rs

+                        }
+                        None => Some(1),
+                    };
+                    let num_push_dups = num_push_recv.unwrap_or(0).saturating_sub(1);


I believe we could just use unwrap() here instead of unwrap_or(0) since num_push_recv will not be None after the match statement

I think this could be written as:

if entry.num_push_recv == Some(0) { self.stats.lock().unwrap().num_redundant_pull_responses += 1; } let num_push_dups = entry.num_push_recv.unwrap_or_default(); entry.num_push_recv = Some(num_push_dups.saturating_add(1));

gregcusack · 2024-03-13T21:44:54Z

gossip/src/crds.rs

    pub(crate) push: CrdsDataStats,
+    /// number of times a message was first received via a PullResponse
+    /// and that message was later received via a PushMessage
+    pub(crate) num_redundant_pull_responses: u64,


should I make this a Counter or leave it as u64?

u64 seems sufficient.

behzadnouri · 2024-03-14T14:34:28Z

gossip/src/crds.rs

    pub(crate) push: CrdsDataStats,
+    /// number of times a message was first received via a PullResponse
+    /// and that message was later received via a PushMessage
+    pub(crate) num_redundant_pull_responses: u64,


u64 seems sufficient.

behzadnouri · 2024-03-14T14:43:11Z

gossip/src/crds.rs

+                        }
+                        None => Some(1),
+                    };
+                    let num_push_dups = num_push_recv.unwrap_or(0).saturating_sub(1);


I think this could be written as:

if entry.num_push_recv == Some(0) { self.stats.lock().unwrap().num_redundant_pull_responses += 1; } let num_push_dups = entry.num_push_recv.unwrap_or_default(); entry.num_push_recv = Some(num_push_dups.saturating_add(1));

behzadnouri · 2024-03-14T14:45:46Z

gossip/src/crds.rs

-    /// Number of times duplicates of this value are recevied from gossip push.
-    num_push_dups: u8,
+    /// number of times this value is recevied via PushMessage
+    num_push_recv: Option<u8>,


Maybe add a comment here that

None indicates the value was upserted by GossipRoute::{LocalMessage,PullRequest}.

Some(0) indicates the value was upserted by GossipRoute::PullResponse.

Some(k) if k > 0 indicates the value was upserted by GossipRoute::PushMessage with k - 1 push duplicates.

behzadnouri

lgtm

behzadnouri · 2024-03-14T18:41:19Z

I think it would be good to backport this to v1.18 to both get the metrics, and also fix the num-dups accounting for received-cache scoring.

gregcusack · 2024-03-14T19:23:56Z

also fix the num-dups accounting for received-cache scoring.

what do you mean by this? isn't the scoring the same from the ReceivedCache side of things?

mergify · 2024-03-14T19:24:48Z

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.

(cherry picked from commit d49ceb0)

behzadnouri · 2024-03-14T20:56:01Z

also fix the num-dups accounting for received-cache scoring.

what do you mean by this? isn't the scoring the same from the ReceivedCache side of things?

Basically this: #199 (comment)

#251) Add in metrics for detecting Redundant Pulls (#199) (cherry picked from commit d49ceb0) Co-authored-by: Greg Cusack <[email protected]>

gregcusack · 2024-03-14T21:00:38Z

also fix the num-dups accounting for received-cache scoring.

what do you mean by this? isn't the scoring the same from the ReceivedCache side of things?

Basically this: #199 (comment)

ok I see. ya makes sense. thank you!

…-xyz#199) (anza-xyz#251) Add in metrics for detecting Redundant Pulls (anza-xyz#199) (cherry picked from commit d49ceb0) Co-authored-by: Greg Cusack <[email protected]>

only reroute if relayer connected (anza-xyz#123) feat: add client tls config (anza-xyz#121) remove extra val (anza-xyz#129) fix clippy (anza-xyz#130) copy all binaries to docker-output (anza-xyz#131) Ledger tool halts at slot passed to create-snapshot (anza-xyz#118) update program submodule (anza-xyz#133) quick fix for tips and clearing old bundles (anza-xyz#135) update submodule to new program (anza-xyz#136) Improve stake-meta-generator usability (anza-xyz#134) pinning submodule head (anza-xyz#140) Use BundleAccountLocker when handling tip txs (anza-xyz#147) Add metrics for relayer + block engine proxy (anza-xyz#149) Build claim-mev in docker (anza-xyz#141) Rework bundle receiving and add metrics (anza-xyz#152) (anza-xyz#154) update submodule + dev files (anza-xyz#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (anza-xyz#159) update jito-programs submodule (anza-xyz#160) Separate MEV tip related workflow (anza-xyz#161) Add block builder fee protos (anza-xyz#162) fix jito programs (anza-xyz#163) update submodule so autosnapshot exits out of ledger tool early (anza-xyz#164) Pipe through block builder fee (anza-xyz#167) pull in new snapshot code (anza-xyz#171) block builder bug (anza-xyz#172) Pull in new slack autosnapshot submodule (anza-xyz#174) sort stake meta json and use int math (anza-xyz#176) add accountsdb conn submod (anza-xyz#169) Update tip distribution parameters (anza-xyz#177) new submodules (anza-xyz#180) Add buildkite link for jito CI (anza-xyz#183) Fixed broken links to repositories (anza-xyz#184) Changed from ssh to https transfer for clone Seg/update submods (anza-xyz#187) fix tests (anza-xyz#190) rm geyser submod (anza-xyz#192) rm dangling geyser references (anza-xyz#193) fix syntax err (anza-xyz#195) use deterministic req ids in batch calls (anza-xyz#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (anza-xyz#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (anza-xyz#203) Add logging for sol balance desired (anza-xyz#205) * add logging * add logging * update msg * tweak vars update submodule (anza-xyz#204) use efficient data structures when calling batch_simulate_bundles (anza-xyz#206) [JIT-504] Add low balance check in uploading merkle roots (anza-xyz#209) add config to simulate on top of working bank (anza-xyz#211) rm frozen bank check simulate_bundle rpc bugfixes (anza-xyz#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (anza-xyz#210) * add files * switch to include bump update submodule (anza-xyz#217) add amount filter (anza-xyz#218) update autosnapshot (anza-xyz#222) Print TX error in Bundles (anza-xyz#223) add new args to support single relayer and block-engine endpoints (anza-xyz#224) point to new jito-programs submod and invoke updated init tda instruction (anza-xyz#228) fix clippy errors (anza-xyz#230) fix validator start scripts (anza-xyz#232) Point README to gitbook (anza-xyz#237) use packaged cargo bin to build (anza-xyz#239) Add validator identity pubkey to StakeMeta (anza-xyz#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (anza-xyz#240) Seg/tip distributor master (anza-xyz#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (anza-xyz#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (anza-xyz#247) Improve docker caching delay constructing claim mev txs (anza-xyz#253) fix stake meta tests from bb fee (anza-xyz#254) fix tests Buffer bundles that exceed cost model (anza-xyz#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (anza-xyz#260) update Cargo.lock to correct solana versions in jito-programs submodule (anza-xyz#265) fix simulate_bundle client and better error handling (anza-xyz#267) update submod (anza-xyz#272) Preallocate Bundle Cost (anza-xyz#238) fix Dockerfile (anza-xyz#278) Fix Tests (anza-xyz#279) Fix Tests (anza-xyz#281) * fix tests update jito-programs submod (anza-xyz#282) add reclaim rent workflow (anza-xyz#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (anza-xyz#292) Remove security.md (anza-xyz#293) demote frequent relayer_stage-stream_error to warn (anza-xyz#275) account for case where TDA exists but not allocated (anza-xyz#295) implement better retries for tip-distributor workflows (anza-xyz#297) limit number of concurrent rpc calls (anza-xyz#298) Discard Empty Packet Batches (anza-xyz#299) Identity Hotswap (anza-xyz#290) small fixes (anza-xyz#305) Set backend config from admin rpc (anza-xyz#304) Admin Shred Receiver Change (anza-xyz#306) Seg/rm bundle UUID (anza-xyz#309) Fix github workflow to recursively clone (anza-xyz#327) Add recursive checkout for downstream-project-spl.yaml (anza-xyz#341) Use cluster info functions for tpu (anza-xyz#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (anza-xyz#374) Updates bootstrap and start scripts needed for local dev. (anza-xyz#384) Remove Deprecated Cli Args (anza-xyz#387) Master Rebase improve simulate_bundle errors and response (anza-xyz#404) derive Clone on accountoverrides (anza-xyz#416) Add upsert to AccountOverrides (anza-xyz#419) update jito-programs (anza-xyz#430) [JIT-1661] Faster Autosnapshot (anza-xyz#436) Reverts simulate_transaction result calls to upstream (anza-xyz#446) Don't unlock accounts in TransactionBatches used during simulation (anza-xyz#449) first pass at wiring up jito-plugin (anza-xyz#428) [JIT-1713] Fix bundle's blockspace preallocation (anza-xyz#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (anza-xyz#491) [JIT-1710] - Optimize Bundle Consumer Checks (anza-xyz#490) Add Blockhash Metrics to Bundle Committer (anza-xyz#500) add priority fee ix to mev-claim (anza-xyz#520) Update Autosnapshot (anza-xyz#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (anza-xyz#582) Update CI (anza-xyz#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (anza-xyz#585) Fix more release urls (anza-xyz#588) [JIT-1812] Fix blocking mutexs (anza-xyz#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (anza-xyz#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase anza-xyz#594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (anza-xyz#597) Sleep longer waiting for buildkite to start (anza-xyz#598) correctly initialize account overrides (anza-xyz#595) Fix: Ensure set contact info to UDP port instead of QUIC (anza-xyz#603) Add fast replay branch to daily rebase (anza-xyz#607) take a snapshot of all bundle accounts before sim (anza-xyz#13) (anza-xyz#615) update jito-programs submodule Add 2.0 to daily rebase (anza-xyz#626) Export agave binaries during docker build (anza-xyz#627) Buffer bundles that exceed processing time and make the allowed processing time longer (anza-xyz#611) Publish releases to S3 and GCS (anza-xyz#633) Rebase from different repos (anza-xyz#637) Point SECURITY.md to immunefi (anza-xyz#671) Loosen requirements on tip accounts touchable in BankingStage (anza-xyz#683) Separate out broadcast + retransmit shredstream (anza-xyz#703) Add packet flag for staked node (anza-xyz#705) Add auto-rebase to v2.1 (anza-xyz#739) Fix release github (anza-xyz#745) Move block_cost_limit tracking to BankingStage in preparation for SIMD-0207 (anza-xyz#753) Add precompile checks in BundleStage (anza-xyz#787) Add auto-rebase to v2.2 (anza-xyz#818) Add better error handling around missing transaction signatures for bundle id generation (anza-xyz#860) Remove unwrap from authentication (anza-xyz#861) Revert Jito-Solana WorkingBankEntry changes (anza-xyz#873) BP anza-xyz#885: Add libclang to Dockerfile (anza-xyz#886) Remove the tip distributor code (anza-xyz#888) Rebase: Update anchor to not use deprecated crates

only reroute if relayer connected (anza-xyz#123) feat: add client tls config (anza-xyz#121) remove extra val (anza-xyz#129) fix clippy (anza-xyz#130) copy all binaries to docker-output (anza-xyz#131) Ledger tool halts at slot passed to create-snapshot (anza-xyz#118) update program submodule (anza-xyz#133) quick fix for tips and clearing old bundles (anza-xyz#135) update submodule to new program (anza-xyz#136) Improve stake-meta-generator usability (anza-xyz#134) pinning submodule head (anza-xyz#140) Use BundleAccountLocker when handling tip txs (anza-xyz#147) Add metrics for relayer + block engine proxy (anza-xyz#149) Build claim-mev in docker (anza-xyz#141) Rework bundle receiving and add metrics (anza-xyz#152) (anza-xyz#154) update submodule + dev files (anza-xyz#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (anza-xyz#159) update jito-programs submodule (anza-xyz#160) Separate MEV tip related workflow (anza-xyz#161) Add block builder fee protos (anza-xyz#162) fix jito programs (anza-xyz#163) update submodule so autosnapshot exits out of ledger tool early (anza-xyz#164) Pipe through block builder fee (anza-xyz#167) pull in new snapshot code (anza-xyz#171) block builder bug (anza-xyz#172) Pull in new slack autosnapshot submodule (anza-xyz#174) sort stake meta json and use int math (anza-xyz#176) add accountsdb conn submod (anza-xyz#169) Update tip distribution parameters (anza-xyz#177) new submodules (anza-xyz#180) Add buildkite link for jito CI (anza-xyz#183) Fixed broken links to repositories (anza-xyz#184) Changed from ssh to https transfer for clone Seg/update submods (anza-xyz#187) fix tests (anza-xyz#190) rm geyser submod (anza-xyz#192) rm dangling geyser references (anza-xyz#193) fix syntax err (anza-xyz#195) use deterministic req ids in batch calls (anza-xyz#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (anza-xyz#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (anza-xyz#203) Add logging for sol balance desired (anza-xyz#205) * add logging * add logging * update msg * tweak vars update submodule (anza-xyz#204) use efficient data structures when calling batch_simulate_bundles (anza-xyz#206) [JIT-504] Add low balance check in uploading merkle roots (anza-xyz#209) add config to simulate on top of working bank (anza-xyz#211) rm frozen bank check simulate_bundle rpc bugfixes (anza-xyz#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (anza-xyz#210) * add files * switch to include bump update submodule (anza-xyz#217) add amount filter (anza-xyz#218) update autosnapshot (anza-xyz#222) Print TX error in Bundles (anza-xyz#223) add new args to support single relayer and block-engine endpoints (anza-xyz#224) point to new jito-programs submod and invoke updated init tda instruction (anza-xyz#228) fix clippy errors (anza-xyz#230) fix validator start scripts (anza-xyz#232) Point README to gitbook (anza-xyz#237) use packaged cargo bin to build (anza-xyz#239) Add validator identity pubkey to StakeMeta (anza-xyz#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (anza-xyz#240) Seg/tip distributor master (anza-xyz#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (anza-xyz#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (anza-xyz#247) Improve docker caching delay constructing claim mev txs (anza-xyz#253) fix stake meta tests from bb fee (anza-xyz#254) fix tests Buffer bundles that exceed cost model (anza-xyz#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (anza-xyz#260) update Cargo.lock to correct solana versions in jito-programs submodule (anza-xyz#265) fix simulate_bundle client and better error handling (anza-xyz#267) update submod (anza-xyz#272) Preallocate Bundle Cost (anza-xyz#238) fix Dockerfile (anza-xyz#278) Fix Tests (anza-xyz#279) Fix Tests (anza-xyz#281) * fix tests update jito-programs submod (anza-xyz#282) add reclaim rent workflow (anza-xyz#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (anza-xyz#292) Remove security.md (anza-xyz#293) demote frequent relayer_stage-stream_error to warn (anza-xyz#275) account for case where TDA exists but not allocated (anza-xyz#295) implement better retries for tip-distributor workflows (anza-xyz#297) limit number of concurrent rpc calls (anza-xyz#298) Discard Empty Packet Batches (anza-xyz#299) Identity Hotswap (anza-xyz#290) small fixes (anza-xyz#305) Set backend config from admin rpc (anza-xyz#304) Admin Shred Receiver Change (anza-xyz#306) Seg/rm bundle UUID (anza-xyz#309) Fix github workflow to recursively clone (anza-xyz#327) Add recursive checkout for downstream-project-spl.yaml (anza-xyz#341) Use cluster info functions for tpu (anza-xyz#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (anza-xyz#374) Updates bootstrap and start scripts needed for local dev. (anza-xyz#384) Remove Deprecated Cli Args (anza-xyz#387) Master Rebase improve simulate_bundle errors and response (anza-xyz#404) derive Clone on accountoverrides (anza-xyz#416) Add upsert to AccountOverrides (anza-xyz#419) update jito-programs (anza-xyz#430) [JIT-1661] Faster Autosnapshot (anza-xyz#436) Reverts simulate_transaction result calls to upstream (anza-xyz#446) Don't unlock accounts in TransactionBatches used during simulation (anza-xyz#449) first pass at wiring up jito-plugin (anza-xyz#428) [JIT-1713] Fix bundle's blockspace preallocation (anza-xyz#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (anza-xyz#491) [JIT-1710] - Optimize Bundle Consumer Checks (anza-xyz#490) Add Blockhash Metrics to Bundle Committer (anza-xyz#500) add priority fee ix to mev-claim (anza-xyz#520) Update Autosnapshot (anza-xyz#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (anza-xyz#582) Update CI (anza-xyz#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (anza-xyz#585) Fix more release urls (anza-xyz#588) [JIT-1812] Fix blocking mutexs (anza-xyz#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (anza-xyz#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase anza-xyz#594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (anza-xyz#597) Sleep longer waiting for buildkite to start (anza-xyz#598) correctly initialize account overrides (anza-xyz#595) Fix: Ensure set contact info to UDP port instead of QUIC (anza-xyz#603) Add fast replay branch to daily rebase (anza-xyz#607) take a snapshot of all bundle accounts before sim (anza-xyz#13) (anza-xyz#615) update jito-programs submodule Add 2.0 to daily rebase (anza-xyz#626) Export agave binaries during docker build (anza-xyz#627) Buffer bundles that exceed processing time and make the allowed processing time longer (anza-xyz#611) Publish releases to S3 and GCS (anza-xyz#633) Rebase from different repos (anza-xyz#637) Point SECURITY.md to immunefi (anza-xyz#671) Loosen requirements on tip accounts touchable in BankingStage (anza-xyz#683) Separate out broadcast + retransmit shredstream (anza-xyz#703) Add packet flag for staked node (anza-xyz#705) Add auto-rebase to v2.1 (anza-xyz#739) Fix release github (anza-xyz#745) Move block_cost_limit tracking to BankingStage in preparation for SIMD-0207 (anza-xyz#753) Add precompile checks in BundleStage (anza-xyz#787) Add auto-rebase to v2.2 (anza-xyz#818) Add better error handling around missing transaction signatures for bundle id generation (anza-xyz#860) Remove unwrap from authentication (anza-xyz#861) Revert Jito-Solana WorkingBankEntry changes (anza-xyz#873) Add libclang to Dockerfile (anza-xyz#885) Remove the tip distributor code (anza-xyz#888) Rebase: Update anchor to not use deprecated crates Add TLS webpki roots back in (anza-xyz#933) Remove trusted relayer packets (anza-xyz#952) Fix shred retransmit (anza-xyz#954) [v3.0] Automatically use optimal Block Engine region (anza-xyz#983) (Backport: anza-xyz#995) Disable autoconfig (anza-xyz#996)

avoid errors for empty args

only reroute if relayer connected (anza-xyz#123) feat: add client tls config (anza-xyz#121) remove extra val (anza-xyz#129) fix clippy (anza-xyz#130) copy all binaries to docker-output (anza-xyz#131) Ledger tool halts at slot passed to create-snapshot (anza-xyz#118) update program submodule (anza-xyz#133) quick fix for tips and clearing old bundles (anza-xyz#135) update submodule to new program (anza-xyz#136) Improve stake-meta-generator usability (anza-xyz#134) pinning submodule head (anza-xyz#140) Use BundleAccountLocker when handling tip txs (anza-xyz#147) Add metrics for relayer + block engine proxy (anza-xyz#149) Build claim-mev in docker (anza-xyz#141) Rework bundle receiving and add metrics (anza-xyz#152) (anza-xyz#154) update submodule + dev files (anza-xyz#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (anza-xyz#159) update jito-programs submodule (anza-xyz#160) Separate MEV tip related workflow (anza-xyz#161) Add block builder fee protos (anza-xyz#162) fix jito programs (anza-xyz#163) update submodule so autosnapshot exits out of ledger tool early (anza-xyz#164) Pipe through block builder fee (anza-xyz#167) pull in new snapshot code (anza-xyz#171) block builder bug (anza-xyz#172) Pull in new slack autosnapshot submodule (anza-xyz#174) sort stake meta json and use int math (anza-xyz#176) add accountsdb conn submod (anza-xyz#169) Update tip distribution parameters (anza-xyz#177) new submodules (anza-xyz#180) Add buildkite link for jito CI (anza-xyz#183) Fixed broken links to repositories (anza-xyz#184) Changed from ssh to https transfer for clone Seg/update submods (anza-xyz#187) fix tests (anza-xyz#190) rm geyser submod (anza-xyz#192) rm dangling geyser references (anza-xyz#193) fix syntax err (anza-xyz#195) use deterministic req ids in batch calls (anza-xyz#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (anza-xyz#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (anza-xyz#203) Add logging for sol balance desired (anza-xyz#205) * add logging * add logging * update msg * tweak vars update submodule (anza-xyz#204) use efficient data structures when calling batch_simulate_bundles (anza-xyz#206) [JIT-504] Add low balance check in uploading merkle roots (anza-xyz#209) add config to simulate on top of working bank (anza-xyz#211) rm frozen bank check simulate_bundle rpc bugfixes (anza-xyz#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (anza-xyz#210) * add files * switch to include bump update submodule (anza-xyz#217) add amount filter (anza-xyz#218) update autosnapshot (anza-xyz#222) Print TX error in Bundles (anza-xyz#223) add new args to support single relayer and block-engine endpoints (anza-xyz#224) point to new jito-programs submod and invoke updated init tda instruction (anza-xyz#228) fix clippy errors (anza-xyz#230) fix validator start scripts (anza-xyz#232) Point README to gitbook (anza-xyz#237) use packaged cargo bin to build (anza-xyz#239) Add validator identity pubkey to StakeMeta (anza-xyz#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (anza-xyz#240) Seg/tip distributor master (anza-xyz#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (anza-xyz#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (anza-xyz#247) Improve docker caching delay constructing claim mev txs (anza-xyz#253) fix stake meta tests from bb fee (anza-xyz#254) fix tests Buffer bundles that exceed cost model (anza-xyz#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (anza-xyz#260) update Cargo.lock to correct solana versions in jito-programs submodule (anza-xyz#265) fix simulate_bundle client and better error handling (anza-xyz#267) update submod (anza-xyz#272) Preallocate Bundle Cost (anza-xyz#238) fix Dockerfile (anza-xyz#278) Fix Tests (anza-xyz#279) Fix Tests (anza-xyz#281) * fix tests update jito-programs submod (anza-xyz#282) add reclaim rent workflow (anza-xyz#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (anza-xyz#292) Remove security.md (anza-xyz#293) demote frequent relayer_stage-stream_error to warn (anza-xyz#275) account for case where TDA exists but not allocated (anza-xyz#295) implement better retries for tip-distributor workflows (anza-xyz#297) limit number of concurrent rpc calls (anza-xyz#298) Discard Empty Packet Batches (anza-xyz#299) Identity Hotswap (anza-xyz#290) small fixes (anza-xyz#305) Set backend config from admin rpc (anza-xyz#304) Admin Shred Receiver Change (anza-xyz#306) Seg/rm bundle UUID (anza-xyz#309) Fix github workflow to recursively clone (anza-xyz#327) Add recursive checkout for downstream-project-spl.yaml (anza-xyz#341) Use cluster info functions for tpu (anza-xyz#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (anza-xyz#374) Updates bootstrap and start scripts needed for local dev. (anza-xyz#384) Remove Deprecated Cli Args (anza-xyz#387) Master Rebase improve simulate_bundle errors and response (anza-xyz#404) derive Clone on accountoverrides (anza-xyz#416) Add upsert to AccountOverrides (anza-xyz#419) update jito-programs (anza-xyz#430) [JIT-1661] Faster Autosnapshot (anza-xyz#436) Reverts simulate_transaction result calls to upstream (anza-xyz#446) Don't unlock accounts in TransactionBatches used during simulation (anza-xyz#449) first pass at wiring up jito-plugin (anza-xyz#428) [JIT-1713] Fix bundle's blockspace preallocation (anza-xyz#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (anza-xyz#491) [JIT-1710] - Optimize Bundle Consumer Checks (anza-xyz#490) Add Blockhash Metrics to Bundle Committer (anza-xyz#500) add priority fee ix to mev-claim (anza-xyz#520) Update Autosnapshot (anza-xyz#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (anza-xyz#582) Update CI (anza-xyz#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (anza-xyz#585) Fix more release urls (anza-xyz#588) [JIT-1812] Fix blocking mutexs (anza-xyz#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (anza-xyz#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase anza-xyz#594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (anza-xyz#597) Sleep longer waiting for buildkite to start (anza-xyz#598) correctly initialize account overrides (anza-xyz#595) Fix: Ensure set contact info to UDP port instead of QUIC (anza-xyz#603) Add fast replay branch to daily rebase (anza-xyz#607) take a snapshot of all bundle accounts before sim (anza-xyz#13) (anza-xyz#615) update jito-programs submodule Add 2.0 to daily rebase (anza-xyz#626) Export agave binaries during docker build (anza-xyz#627) Buffer bundles that exceed processing time and make the allowed processing time longer (anza-xyz#611) Publish releases to S3 and GCS (anza-xyz#633) Rebase from different repos (anza-xyz#637) Point SECURITY.md to immunefi (anza-xyz#671) Loosen requirements on tip accounts touchable in BankingStage (anza-xyz#683) Separate out broadcast + retransmit shredstream (anza-xyz#703) Add packet flag for staked node (anza-xyz#705) Add auto-rebase to v2.1 (anza-xyz#739) Fix release github (anza-xyz#745) Move block_cost_limit tracking to BankingStage in preparation for SIMD-0207 (anza-xyz#753) Add precompile checks in BundleStage (anza-xyz#787) Add auto-rebase to v2.2 (anza-xyz#818) Add better error handling around missing transaction signatures for bundle id generation (anza-xyz#860) Remove unwrap from authentication (anza-xyz#861) Revert Jito-Solana WorkingBankEntry changes (anza-xyz#873) Add libclang to Dockerfile (anza-xyz#885) Remove the tip distributor code (anza-xyz#888) Rebase: Update anchor to not use deprecated crates Add TLS webpki roots back in (anza-xyz#933) Remove trusted relayer packets (anza-xyz#952) Fix shred retransmit (anza-xyz#954) [v3.0] Automatically use optimal Block Engine region (anza-xyz#983) (Backport: anza-xyz#995) Disable autoconfig (anza-xyz#996)

only reroute if relayer connected (anza-xyz#123) feat: add client tls config (anza-xyz#121) remove extra val (anza-xyz#129) fix clippy (anza-xyz#130) copy all binaries to docker-output (anza-xyz#131) Ledger tool halts at slot passed to create-snapshot (anza-xyz#118) update program submodule (anza-xyz#133) quick fix for tips and clearing old bundles (anza-xyz#135) update submodule to new program (anza-xyz#136) Improve stake-meta-generator usability (anza-xyz#134) pinning submodule head (anza-xyz#140) Use BundleAccountLocker when handling tip txs (anza-xyz#147) Add metrics for relayer + block engine proxy (anza-xyz#149) Build claim-mev in docker (anza-xyz#141) Rework bundle receiving and add metrics (anza-xyz#152) (anza-xyz#154) update submodule + dev files (anza-xyz#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (anza-xyz#159) update jito-programs submodule (anza-xyz#160) Separate MEV tip related workflow (anza-xyz#161) Add block builder fee protos (anza-xyz#162) fix jito programs (anza-xyz#163) update submodule so autosnapshot exits out of ledger tool early (anza-xyz#164) Pipe through block builder fee (anza-xyz#167) pull in new snapshot code (anza-xyz#171) block builder bug (anza-xyz#172) Pull in new slack autosnapshot submodule (anza-xyz#174) sort stake meta json and use int math (anza-xyz#176) add accountsdb conn submod (anza-xyz#169) Update tip distribution parameters (anza-xyz#177) new submodules (anza-xyz#180) Add buildkite link for jito CI (anza-xyz#183) Fixed broken links to repositories (anza-xyz#184) Changed from ssh to https transfer for clone Seg/update submods (anza-xyz#187) fix tests (anza-xyz#190) rm geyser submod (anza-xyz#192) rm dangling geyser references (anza-xyz#193) fix syntax err (anza-xyz#195) use deterministic req ids in batch calls (anza-xyz#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (anza-xyz#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (anza-xyz#203) Add logging for sol balance desired (anza-xyz#205) * add logging * add logging * update msg * tweak vars update submodule (anza-xyz#204) use efficient data structures when calling batch_simulate_bundles (anza-xyz#206) [JIT-504] Add low balance check in uploading merkle roots (anza-xyz#209) add config to simulate on top of working bank (anza-xyz#211) rm frozen bank check simulate_bundle rpc bugfixes (anza-xyz#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (anza-xyz#210) * add files * switch to include bump update submodule (anza-xyz#217) add amount filter (anza-xyz#218) update autosnapshot (anza-xyz#222) Print TX error in Bundles (anza-xyz#223) add new args to support single relayer and block-engine endpoints (anza-xyz#224) point to new jito-programs submod and invoke updated init tda instruction (anza-xyz#228) fix clippy errors (anza-xyz#230) fix validator start scripts (anza-xyz#232) Point README to gitbook (anza-xyz#237) use packaged cargo bin to build (anza-xyz#239) Add validator identity pubkey to StakeMeta (anza-xyz#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (anza-xyz#240) Seg/tip distributor master (anza-xyz#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (anza-xyz#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (anza-xyz#247) Improve docker caching delay constructing claim mev txs (anza-xyz#253) fix stake meta tests from bb fee (anza-xyz#254) fix tests Buffer bundles that exceed cost model (anza-xyz#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (anza-xyz#260) update Cargo.lock to correct solana versions in jito-programs submodule (anza-xyz#265) fix simulate_bundle client and better error handling (anza-xyz#267) update submod (anza-xyz#272) Preallocate Bundle Cost (anza-xyz#238) fix Dockerfile (anza-xyz#278) Fix Tests (anza-xyz#279) Fix Tests (anza-xyz#281) * fix tests update jito-programs submod (anza-xyz#282) add reclaim rent workflow (anza-xyz#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (anza-xyz#292) Remove security.md (anza-xyz#293) demote frequent relayer_stage-stream_error to warn (anza-xyz#275) account for case where TDA exists but not allocated (anza-xyz#295) implement better retries for tip-distributor workflows (anza-xyz#297) limit number of concurrent rpc calls (anza-xyz#298) Discard Empty Packet Batches (anza-xyz#299) Identity Hotswap (anza-xyz#290) small fixes (anza-xyz#305) Set backend config from admin rpc (anza-xyz#304) Admin Shred Receiver Change (anza-xyz#306) Seg/rm bundle UUID (anza-xyz#309) Fix github workflow to recursively clone (anza-xyz#327) Add recursive checkout for downstream-project-spl.yaml (anza-xyz#341) Use cluster info functions for tpu (anza-xyz#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (anza-xyz#374) Updates bootstrap and start scripts needed for local dev. (anza-xyz#384) Remove Deprecated Cli Args (anza-xyz#387) Master Rebase improve simulate_bundle errors and response (anza-xyz#404) derive Clone on accountoverrides (anza-xyz#416) Add upsert to AccountOverrides (anza-xyz#419) update jito-programs (anza-xyz#430) [JIT-1661] Faster Autosnapshot (anza-xyz#436) Reverts simulate_transaction result calls to upstream (anza-xyz#446) Don't unlock accounts in TransactionBatches used during simulation (anza-xyz#449) first pass at wiring up jito-plugin (anza-xyz#428) [JIT-1713] Fix bundle's blockspace preallocation (anza-xyz#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (anza-xyz#491) [JIT-1710] - Optimize Bundle Consumer Checks (anza-xyz#490) Add Blockhash Metrics to Bundle Committer (anza-xyz#500) add priority fee ix to mev-claim (anza-xyz#520) Update Autosnapshot (anza-xyz#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (anza-xyz#582) Update CI (anza-xyz#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (anza-xyz#585) Fix more release urls (anza-xyz#588) [JIT-1812] Fix blocking mutexs (anza-xyz#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (anza-xyz#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase anza-xyz#594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (anza-xyz#597) Sleep longer waiting for buildkite to start (anza-xyz#598) correctly initialize account overrides (anza-xyz#595) Fix: Ensure set contact info to UDP port instead of QUIC (anza-xyz#603) Add fast replay branch to daily rebase (anza-xyz#607) take a snapshot of all bundle accounts before sim (anza-xyz#13) (anza-xyz#615) update jito-programs submodule Add 2.0 to daily rebase (anza-xyz#626) Export agave binaries during docker build (anza-xyz#627) Buffer bundles that exceed processing time and make the allowed processing time longer (anza-xyz#611) Publish releases to S3 and GCS (anza-xyz#633) Rebase from different repos (anza-xyz#637) Point SECURITY.md to immunefi (anza-xyz#671) Loosen requirements on tip accounts touchable in BankingStage (anza-xyz#683) Separate out broadcast + retransmit shredstream (anza-xyz#703) Add packet flag for staked node (anza-xyz#705) Add auto-rebase to v2.1 (anza-xyz#739) Fix release github (anza-xyz#745) Move block_cost_limit tracking to BankingStage in preparation for SIMD-0207 (anza-xyz#753) Add precompile checks in BundleStage (anza-xyz#787) Add auto-rebase to v2.2 (anza-xyz#818) Add better error handling around missing transaction signatures for bundle id generation (anza-xyz#860) Remove unwrap from authentication (anza-xyz#861) Revert Jito-Solana WorkingBankEntry changes (anza-xyz#873) Add libclang to Dockerfile (anza-xyz#885) Remove the tip distributor code (anza-xyz#888) Rebase: Update anchor to not use deprecated crates Add TLS webpki roots back in (anza-xyz#933) Remove trusted relayer packets (anza-xyz#952) Fix shred retransmit (anza-xyz#954) Add daily v3.0 rebase (anza-xyz#972) [Master] Automatically use optimal Block Engine region (anza-xyz#974) Disable autoconfig (anza-xyz#995) Make shredstream optional (anza-xyz#997)

gregcusack mentioned this pull request Mar 12, 2024

add in metrics for detecting redundant pulls #139

Closed

gregcusack marked this pull request as ready for review March 12, 2024 03:07

gregcusack commented Mar 12, 2024

View reviewed changes

gregcusack requested a review from behzadnouri March 12, 2024 03:28

behzadnouri reviewed Mar 12, 2024

View reviewed changes

gregcusack force-pushed the redundant-pull-metrics-v3 branch from 9f3f240 to 0a21125 Compare March 12, 2024 21:37

gregcusack added 2 commits March 12, 2024 21:44

switch to num_push_recv and add metric for redundant pull

04ef20b

add in CrdsError::RedundantPull, update handling of new num_push_recv…

de00d18

…and CrdsError::DuplicatePush

gregcusack force-pushed the redundant-pull-metrics-v3 branch from 0a21125 to 642c89e Compare March 12, 2024 21:45

change up num_push_recv and use struct

90707c3

gregcusack force-pushed the redundant-pull-metrics-v3 branch from 642c89e to 90707c3 Compare March 12, 2024 21:55

gregcusack requested a review from behzadnouri March 12, 2024 21:57

behzadnouri reviewed Mar 13, 2024

View reviewed changes

gregcusack force-pushed the redundant-pull-metrics-v3 branch 2 times, most recently from 7e4849b to 1a1ca54 Compare March 13, 2024 20:29

change num_push_recv handling and implementation, change metric repor…

6f7bd30

…ting

gregcusack force-pushed the redundant-pull-metrics-v3 branch from 1a1ca54 to 6f7bd30 Compare March 13, 2024 21:34

gregcusack commented Mar 13, 2024

View reviewed changes

gregcusack requested a review from behzadnouri March 13, 2024 21:38

gregcusack commented Mar 13, 2024

View reviewed changes

behzadnouri reviewed Mar 14, 2024

View reviewed changes

clarify num_push_recv, refactor redundant pull logic

1d41d97

gregcusack requested a review from behzadnouri March 14, 2024 16:42

behzadnouri approved these changes Mar 14, 2024

View reviewed changes

gregcusack merged commit d49ceb0 into anza-xyz:master Mar 14, 2024

gregcusack deleted the redundant-pull-metrics-v3 branch March 14, 2024 18:22

gregcusack added the v1.18 label Mar 14, 2024

mergify bot pushed a commit that referenced this pull request Mar 14, 2024

Add in metrics for detecting Redundant Pulls (#199)

4cb9fec

(cherry picked from commit d49ceb0)

mergify bot mentioned this pull request Mar 14, 2024

v1.18: Add in metrics for detecting Redundant Pulls (backport of #199) #251

Merged

willhickey mentioned this pull request Mar 28, 2024

v1.18 commits - please ignore #475

Closed

codebender828 pushed a commit to codebender828/agave that referenced this pull request Oct 3, 2024

Add in metrics for detecting Redundant Pulls (anza-xyz#199)

274382f

OliverNChalk pushed a commit to OliverNChalk/agave that referenced this pull request Nov 11, 2025

Fix up invalidator scripts (anza-xyz#199)

291ccc1

avoid errors for empty args

Add in metrics for detecting Redundant Pulls #199

Add in metrics for detecting Redundant Pulls #199

Uh oh!

Conversation

gregcusack commented Mar 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Summary of Changes

Identifying redundant Pulls:

Calculating fraction of messages received via Redundant Pull:

Monogon Simulation: % of redundant pulls in a 100 validator cluster

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Mar 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregcusack commented Mar 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

behzadnouri left a comment

Choose a reason for hiding this comment

Uh oh!

behzadnouri commented Mar 14, 2024

Uh oh!

gregcusack commented Mar 12, 2024 •

edited

Loading

codecov-commenter commented Mar 12, 2024 •

edited

Loading

gregcusack commented Mar 12, 2024 •

edited

Loading