Reprovide Sweep

[guillaumemichel]

Supersedes #1082

Reprovide Sweep

Problem

Reproviding many keys to the DHT one by one is inefficient, because it requires a GetClosestPeers (or GCP) request for every key.

Current state

Currently, reprovides are managed in boxo/provider. Every ReprovideInterval (22h in Amino DHT), all keys matching the reprovide strategy are reprovided at once. The process is slightly different depending on whether the accelerated DHT client is enabled.

Default DHT client

All the keys are reprovided sequentially, using the go-libp2p-kad-dht Provide() method. This operation consists in finding the k closest peers to the given key, and then request them all to store the associated provider record.

The process is expensive because it requires a GCP for each key (opening approx. 20-30 connections). Timeouts due to unreachable peers make this process very long, resulting in a mean of ~10s in provide time (source: probelab.io 2025-06-13).

With 10 seconds per provide, a node using this process could reprovide less than 8'000 keys over the reprovide interval of 22h (using a single thread).

Accelerated DHT client (fullrt)

The accelerated DHT client periodically (every 1h) crawls the DHT swarm to cache the addresses of all discovered peers. It allows it to skip the GCP during the provide request, since it already knows the k closest peers and the associated multiaddrs.

Hence, the accelerated DHT client is able to provide much more keys during the reprovide interval compared with the default DHT client. However, crawling the DHT swarm is an expensive operation (networking, memory), and since all the keys are reprovided at once, the node will experience a bust period until all keys are reprovided.

Ideally, nodes wouldn't have to crawl the swarm to reprovide content, and the reprovide operation could be smoothed over time to avoid a bust during which the libp2p node is incapable of performing other actions.

Pooling Reprovides

If there are more keys to be reprovided than the number of nodes in the DHT swarm divided by the replication factor (k), then it means that there are at least two keys that will be provided to the exact same set of peers. This means that the number of GCP is less than the number of keys to reprovide.

For the Amino DHT, containing ~10k DHT servers and having a replication factor of 20, pooling reprovides becomes efficient starting from 500 keys.

Reprovide Sweep

The current process of reproviding all keys at once is bad because it creates a bust. In order to smooth the reprovide process, we can sweep the keyspace from left to right, in order to cover all peers over time. This consists of exploring keyspace regions, corresponding to a set of peers that are close to each other in the Kademlia XOR distance metric.

⚠️ The Kademlia keyspace in NOT linear

A keyspace region is explored using a few (typically 2-4 GCP) to discover all the peers it contains. A keyspace region can be identified by a Kademlia identifier prefix, the kademlia identifiers of all peers within this region start with the region's prefix.

Once a region is fully explored, all the keys matching the keyspace region's prefix can be allocated to this set of peers. No additional GCP is needed.

Pull Requests

Done

• tests: GCP result ordering #1097
• provider: keystore #1096
• Provider interface #1093
• provider: SweepingProvider interface #1098
• refactor(providers): migrate providers/ to records/ #1094
• provider: Connectivity Checker #1099
• Keyspace (previously helpers, so not renamed in all PRs) -> provider: helpers pacakge rename #1111
  • provider: key helpers #1101
  • provider: add ShortestCoveredPrefix helper #1102
  • provider: trie items listing helpers #1103
  • provider: helpers trie find prefix #1104
  • provider: find subtrie helper #1105
  • provider: NextNonEmptyLeaf trie helper #1106
  • provider: PruneSubtrie helper #1107
  • provider: trie allocation helper #1108
  • provider: trie region helpers #1109
  • provider: helpers pacakge rename #1111
• provider: adding provide and reprovide queue #1114
• provider: network operations #1115
• provider: ProvideStatus interface #1110
• provider: schedule prefix length #1116
• provider: schedule #1117
• provider: handleProvide #1118
• provider: swarm exploration #1120
• provider: batch provide #1121
• provider: batch reprovide #1122
• provider: catchup pending work #1123
• provider: options #1124
• provide: handle reprovide #1125
• provider: daemon #1126
• provider: integration tests #1127
• feat(crawler): configurable protocol messenger #1128
• provider: refresh schedule #1131
• dual: provider #1132
• provider: minor fixes #1133
• provider: connectivity state machine #1135
• provider: use synctest for testing time #1136
• DON'T MERGE: provider: keystore benchmark #1139
• keystore: revamp #1142
• provider: Resettable KeyStore #1146
• keystore: remove mutex #1147
• buffered provider #1149
• provider: minor fixes #1150
• keystore: rename #1151

Ready for review (PRs must be merged in this order)

Ready for review (nice-to-have)

Issues

• provider: optimize keystore operations #1112
• provider: StartProviding shouldn't block #1145

Optional improvements

• provider: stats #1144
• ipfs provide status <cid>
• Resume operations on restart
• Limit ProviderQueue and ReproviderQueue size
• Timeout management
• provider package readme

Admin

Depends on:

• feat: generic find PeerID with CPL go-libp2p-kbucket#145
• fix: bitstr comparison panic probe-lab/go-libdht#17

Need new release of:

• go-libdht
• go-libp2p-kbucket

Closes #824

Part of ipshipyard/roadmaps#6, ipshipyard/roadmaps#7, ipshipyard/roadmaps#8