add in metrics for detecting redundant pulls

[gregcusack]

Problem

We had previously added in a metric for tracking gossip push messages through the network in PR: #32725. However, this metric does not account for redundant pull requests.
Redundant Pull: A node receives a message via PullResponse and then receives the same message via Push.
Redundant Pulls prevent us from accurately calculating how well messages are propagating via Push.

Summary of Changes

Add in metric to report when a node receives a NEW message via PullResponse (gossip_crds_sample_pull).
Add in a metric to report when a node receives a message via Push but fails to insert (gossip_crds_sample_fail).

Identifying redundant Pulls:

1. Get message signatures reported in gossip_crds_sample_pull
2. Get message signatures reported in gossip_crds_sample_fail
3. Take the intersection of signatures from (1) and (2)
4. The intersection of these sets results in all messages that were first received via Pull and then received via Push (aka a Redundant Pull)

Simulation Results

In a 100 node simulation, I saw Redundant Pulls occur somewhat frequently. This indicates Redundant Pulls may be the reason for the discrepancy between the simulated Push coverage and measured Push coverage

Possible Issues

1. Adding in more metrics to an already heavily used metrics server. Could possibly remove these metrics once we get data we need.

[behzadnouri]

• Get message signatures reported in gossip_crds_sample_pull

• Get message signatures reported in gossip_crds_sample_fail

• Take the intersection of signatures from (1) and (2)

• The intersection of these sets results in all messages that were first received via Pull and then received via Push (aka a Redundant Pull)

So this requires a lot of offline processing.
I was thinking of something simpler and not restricted to sampled messages.

Basically, we can change num_push_dups here:
https://github.com/anza-xyz/agave/blob/adefcbbb4/gossip/src/crds.rs#L130-L131
to be just num_push. So

• a newly inserted value from GossipRoute::PushMessage will initialize this value to 1.
• a newly inserted value from GossipRoute::PullResponse will initialize this value to 0.
• (have to think what to do about other GossipRoute::* cases).

(also have to accordingly update a lot of other places which use num_push_dups).

Then if you receive the same value again from GossipRoute::PushMessage:
https://github.com/anza-xyz/agave/blob/adefcbbb4/gossip/src/crds.rs#L304-L308
but num_push is zero then that indicates a redundant pull, and we can either add a metric to CrdsStats to record that or return a new error message to the caller of crds.insert to process that at the call-site.

I would suggest lets hold on to this pr for now, but first implement above simpler approach in a separate pr and lets see how that looks like.

[gregcusack]

I was thinking of something simpler and not restricted to sampled messages.

My only concern here is that in a simple test, there appeared to be a lot of redundant pulls, so it is possible this would be very heavy on metrics server. But I can give it a shot and run some tests with your suggested changes and see what we see. If it's too much we can then sample.

[gregcusack]

Closing in favor of PR: #139