provider: connectivity state machine

[guillaumemichel]

Connectivity State Machine

Benefits

If a node isn't bootstrapped, or is offline for a prolonged period, new provide operations should error rather than adding cids to the queue, and consuming memory.

It helps in the case the client is offline when adding a lot of data, or in the case of the LAN DHT if no DHT server is on the LAN.

State Machine

Connectivity package:

// State Machine starting in OFFLINE state (when `Start()` is called)
//  1. OFFLINE state:
//     - Calls `checkFunc` with exponential backoff until node is found ONLINE.
//     - Calls to `TriggerCheck()` are ignored while OFFLINE.
//     - When `checkFunc` returns true, state changes to ONLINE and
//     `onOnline()` callback is called.
//  2. ONLINE state:
//     - Calls to `TriggerCheck()` will call `checkFunc` only if at least
//     `onlineCheckInterval` has passed since the last check.
//     - If `TriggerCheck()` returns false, switch state to DISCONNECTED.
//  3. DISCONNECTED state:
//     - Calls `checkFunc` with exponential backoff until node is found ONLINE.
//     - Calls to `TriggerCheck()` are ignored while DISCONNECTED.
//     - When `checkFunc` returns true, state changes to ONLINE and
//     `onOnline()` callback is called.
//     - After `offlineDelay` has passed in DISCONNECTED state, state changes
//     to OFFLINE and `onOffline()` callback is called.

What the callbacks do:

• onOnline():
  • make a few GCP requests to estimate the prefix length to be used with the schedule
  • refresh schedule with keys that may have been added while the node was OFFLINE
  • resume any pending work (provide and reprovide queue)
• onOffline():
  • clear provide queue
  • invalidate cached prefix length (set to -1) to signal that ProvideOnce, StartProviding, RefreshSchedule and AddToSchedule should return an error.

It feels like this PR is adding complexity :(

Notes:

• This PR changes the Provide interface by returning errors for ProvideOnce, StartProviding, and StopProviding (as well as RefreshSchedule and AddToSchedule).
  • This seems fine since the operation remains async
  • Error is returned only if supplied keys cannot be queued
  • Returning errors is better than silently failing
• connectivity tests
  • AI generated
  • exponential backoff isn't tested
  • tests will be rewritten when migrating to quartz

To do before merging:

• Test in feat: opt-in new Sweep provide system ipfs/kubo#10834

[gammazero]

Only a few minor comments.