feat: iroh-gossip (#1149)

* feat: iroh-gossip initial commit

This commit imports iroh-gossip into the repo. See n0-computer/iroh-sync#1 for the prehistory.

* From that original PR comes my IO-less implementation of the gossip protocol
  (Plumtree broadcast trees and HyParView cluster membership). The
  implementations are quite strict according to the papers
  [1](https://asc.di.fct.unl.pt/~jleitao/pdf/dsn07-leitao.pdf) and
  [2](https://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf). If you
  find any deviations that are not documented, please report, this is
  likely a bug.
* The combined protocol for Plumtree and Hyparview is additionally namespaced
  with a 32-byte topic ID to allow multiple broadcast trees (one per
  topic) to coexist. This means: Each topic is a seperate broadcast tree
  with seperate swarm membership and active connections (connections to
  the same peers are still shared on the network layer)

This IO-less protocol implementation is in the `proto` module.

This PR *also* adds my work from the past week: A `net` module that provides networking
for the IO-less protocol impl, through the `MagicEndpoint` from iroh-net, with a (I think) quite straightforward actor implementation.

The `net` module has a smoke test, and the PR also includes a `chat` example that adds signed messages on the trustless base layer.

What this mostly needs is, I think, more testing and validation of the
protocol implementation. For this to be practical, I think we should
formulate some base usecase scenarios.

The `proto` module has a very simple simulator and some tests that can
take environment variables to increase the cluster size. They passed
with 10.000 nodes. They also report some stats. The numbers look OK-ish,
but I'd love to compare them properly with either other implementations
or estimations from the original paper or other resoures. However, those
are hard to find.

* fix(gossip): make util pub

* fix(gossip): better naming of cli arguments in chat example

* chore: make clippy happy

* feat: transmit opaque PeerData on join, forward and neighbor

... and wire it up in net module to transmit your endpoint addresses

* example: improve gossip chat example

* chore: clippy

* fix(gossip): update dep version after rebase

* fix: changes after rebase

* change(gossip): cleanup and better debug impls

* fix(gossip): disconnect handling

* change: do not error on connect without addrs

Before MagicEndpoint::connect would error if called without addrs or
derp_region. This commit changes this by only warning and not bailing.
This allows to connect to peers by peer id only if addresses or a derp_region
were provided previously.

The downside is that we lose a valuable error case. A more proper fix
will need to move the check if a peer is dialable into the actor.

* fix: adapt for changes in iroh-net, and use genawaiter

* chore: clippy

* docs: document public items in iroh-gossip

also renames mod.rs files to modname.rs

* tests: move gossip simulator into tests module

* tests: make simulation tests work with top level protocol state

* change: use derive_more::Debug

* feat: add command to leave a topic

* docs: document default plumtree config

* docs: default values for hyparview

* docs: add high-level description of iroh-gossip protocol

* test/refactor: add unit test for plumtree optimization step

* refactor: improve chat example and address review

* fix: do not stop actor if conn closes before event is processed

* docs: more links

* fix: changes after rebase

* change: remove unneeded serialize in iroh-net added earlier

* fix: make tests work and cleanup some namings

* chore: clippy

* refactor: improve and cleanup chat example

* refactor: remove unneeded InEvent variant in plumtree

* refactor: improve gossip example further

* docs for le people and le me

* fmt

* docs: plumtree

* docs: hyparview

* feat: validate gossip message ids on receive

* chore: cleanup

* feat: metrics, and backport fixes from sync-gossip-bytes branch

* feat: expose which peer sent us a gossip message

* fix: postcard is required

* fix: only graft lazy peers in optimize_tree

* chore: remove leftover comment

* docs(hyparview): improve with divma's suggestions

* refactor: impl IntoIterator for IndexSet

* fix(hyparview): only trigger ForwardJoin if peer is not yet active

* fix(hyparview): reply to shuffle if active_view.len() is 1 or 0

* refactor: rename methods for clarity

* fix: do not auto-join topics on external requests

* docs: typos

* docs: fix

* refactor: rename GossipHandle to just Gossip

* refactor: less code, more derive

* docx: fix

* refactor: add util module for base32 parse/fmt

* refactor: make PeerData optional

* refactor: do not use shift_remove in indexset, not needed

* test: add unittests for IndexSet

* test: TimerMap and base32 utils

* refactor: rename trait PeerAddress to PeerId an generic PA to PI

* feat: evict messages after some time

* chore: clippy & fmt

* fix: renames in doc comments

* fix: use rand_chacha in tests for predictable cross-arch results

* chore: fmt

* chore: remove unneeded line

* test: try to fix cross-arch rngs

* test: always recreate RNG for cross-arch compat

* fix: change after rebase on main

* chore: remove debug leftovers

---------

Co-authored-by: Diva M <[email protected]>

Author: Frando

Cargo.lock

@@ -3,6 +3,7 @@ members = [
   "iroh",
   "iroh-net",
   "iroh-bytes",
+  "iroh-gossip",
   "iroh-metrics",
 ]