-
Notifications
You must be signed in to change notification settings - Fork 512
feat: address and proto books #590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
a1617e6
feat: address and proto books
vasco-santos 8dab477
chore: apply suggestions from code review
vasco-santos 72dea96
chore: minor fixes and initial tests added
vasco-santos d3e4ad2
chore: integrate new peer-store with code using adapters for other mo…
vasco-santos eb27b8d
chore: do not use peerstore.put on get-peer-info
vasco-santos f938535
chore: apply suggestions from code review
vasco-santos 1662065
chore: add new peer store tests
vasco-santos 7dcc78e
chore: apply suggestions from code review
vasco-santos File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next
Next commit
feat: address and proto books
- Loading branch information
commit a1617e6793e603fee4ea3bcba26bac017018e7f4
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,156 @@ | ||
| # Peerstore | ||
|
|
||
| WIP | ||
| Libp2p's Peerstore is responsible for keeping an updated register with the relevant information of the known peers. It should gather environment changes and be able to take decisions and notice interested parties of relevant changes. The Peerstore comprises four main components: `addressBook`, `keyBook`, `protocolBook` and `metadataBook`. These book components have similar characteristics with the `Javascript Map` implementation. | ||
|
|
||
| The PeerStore needs to manage the high level operations on its inner books, have a job runner to trigger other books runners for data trimming or computations. Moreover, the peerStore should be responsible for noticing interested parties of relevant events, through its Event Emitter. | ||
|
|
||
| ## Peers Environment | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| #### Sense | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| Several libp2p subsystems will perform operations, which will gather relevant information about peers. Some operations might not have this as an end goal, but can also gather important data. | ||
|
|
||
| In a libp2p node life, it will discover peers the existance of peers through its discovery protocols. In a typical discovery protocol, an address of the peer is discovered combined with its peer id. Once this happens, the `PeerStore` should collect this information for future (or immediate) usage by other subsystems. When the information is stored, the `PeerStore` should inform interested parties of the peer discovered (`peer` event). | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| Taking into account a different scenario, a peer might perform/receive a dial request to/from a unkwown peer. In such a scenario, the `PeerStore` must store the peer's multiaddr once a connection is established. | ||
|
|
||
| (NOTE: this should be removed later) | ||
| (currently we silently put it to the peerStore, without emitting events, as this logic exists in the `onConnected` callback from the upgrader. This way, we are never emitting the `peer` event when inbound connections happen, or a unkwown peer is dialed. Should we differentiate this?) | ||
|
|
||
| After a connection is established with a peer, the Identify Service will act on this connection. A stream is created and peers exchange their information (listenMuldiaddrs and running protocols). Once this information is obtained, the PeerStore can collect the new data. In this specific case, we have a guarantee that this data is complete and updated, so the data stored in the PeerStore should be replaced (older and outdated data should disappear). However, if the recorded `multiaddrs` or `protocols` have changed, interested parties must be informed via `change:multiaddrs` or `change:protocols` events. | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| In the background, the Identify Service is also waiting for new protocols to be started by the peer. If a new protocol is started, the `identify-push` message is sent to all the connected peers, so that their PeerStore can be updated with the new protocol and relevant parties are noticed. As the `identify-push` also sends complete and updated information, the data in the PeerStore is replaced. | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| On different context, it is also possible to gather relevant information for the peers. For instance, in `dht` operations, nodes can exchanges data of peers they know as part of the `dht` operation. In this case, we can get information from a peer that we already know. As a consequence, the `PeerStore` should act accordingly and not replace the data it owns, but just try to merge it the discovered data is new. For example, discovered a new address of a peer. | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| #### Act | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| When the `PeerStore` data is updated, this information might be important for different parties. | ||
|
|
||
| `js-libp2p` keeps a topology of peers for each protocol a node is running. This way, once a protocol is supported for a peer, the topology of that protocol should be informed that a new peer may be used and the subsystem can decide if it should open a new stream it that peer or not. | ||
|
|
||
| Every time a peer needs to dial another peer, it is essential that it knows the multiaddrs used by the peer, in order to perform a successful dial to a peer. The same is true for pinging a peer. | ||
|
|
||
| ## PeerStore implementation | ||
|
|
||
| (Note: except for highlighting the APIs functionallity, they should be better formally described on `API.md` file) | ||
|
|
||
| #### API: | ||
|
|
||
| Access to its underlying books: | ||
|
|
||
| - `peerStore.protoBook.*` | ||
| - `peerStore.addressBook.*` | ||
|
|
||
| High level operations: | ||
|
|
||
| - `peerStore.set(peerId, data, options)` or `events` | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| High level set which should be able to identify the type of data received and forward to the appropriate book sets. More than a bridge, this aims to allow the combination of multiple data storage as follows: | ||
|
|
||
| `data = { multiaddrs: [...], protocols: [...] }` | ||
|
|
||
| ---- (should be removed / re-written, but important for design decisions) | ||
|
|
||
| One aspect that we need to consider is wether we should add information to every book, even if we don't have any relevant information for it. For instance, if we just discover a peer via a typical discovery service, we will have the `peerId` and an array of `multiaddr`. When we do `peerStore.set()`, should we also do `protoBook.set()` with an empty list of protocols? I don't see any advantage on adding to the remaining ones. | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| **IMPORTANT:** This is one of the biggest design decisions to make (set vs events). The programmatic API is the easiest solution but it can provide users an API that they sould not need. If we go on an event-based approach, the `peerStore` should receive all the relevant subsystems (discovery, identifyService, ...) and sense the environment (via events) to gather the information that would need to be sent via the API. Thile the latest seems the best solution, it is the more complex one to implement, as we would ideally have an interface that those subsystems would need to implement and each time we have a new subsystem that needs to add data to the peerStore, we might need to update the `peer-store` codebase (or have a good set of abstractions). | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| It is also important pointing out that users would be able to use `peerStore.protoBook.*`, so eventually we should move those into `peerStore._ protoBook.*` if we do not intend them to use it. | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| --- | ||
|
|
||
| - `peerStore.get(peerId, options)` | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| Get the information of a provided peer. The amount of information that we want can be customized with the following options, which are true by default: | ||
|
|
||
| ```js | ||
| { | ||
| address: true, | ||
| proto: true, | ||
| key: true, | ||
| metadata: true | ||
| } | ||
| ``` | ||
|
|
||
| - `peerStore.delete(peerId, [data])` | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| Deletes the provided peer from every book. If data is provided, just remove the data from the books. The data should be provided as follows: | ||
|
|
||
| ```js | ||
| { | ||
| address: [], | ||
| proto: [], | ||
| key: [], | ||
| metadata: [] | ||
| } | ||
| ``` | ||
|
|
||
| - `peerStore.peers(options)` | ||
|
|
||
| Get an array of all the peers, as well as their information. The information intended can be customized with the following options, which are true by default: | ||
|
|
||
| ```js | ||
| { | ||
| address: true, | ||
| proto: true, | ||
| key: true, | ||
| metadata: true | ||
| } | ||
| ``` | ||
|
|
||
| ## Address Book | ||
|
|
||
| The `addressBook` keeps the known multiaddrs of a peer. The multiaddrs of each peer are not a constant and the Address book must have this into consideration. | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| `Map<string, multiaddrInfo>` | ||
|
|
||
| A `peerId.toString()` identifier mapping to a `multiaddrInfo` object, which should have the following structure: | ||
|
|
||
| ```js | ||
| { | ||
| multiaddr: , | ||
| validity: , | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| confidence: | ||
| } | ||
| ``` | ||
|
|
||
| **Note:** except for multiaddr namings, the other properties are placeholders for now and might not be as described in the future milestones. | ||
|
|
||
| - `addressBook.set()` | ||
| - `addressBook.get()` | ||
| - `getMultiaddrsForPeer()` | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - `addressBook.has()` | ||
| - `addressBook.delete()` | ||
| - `addressBook.peers()` | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| It is important pointing out that the API methods which return arrays of data (`set`, `get`, `getMultiaddrsForPeer`) shuld return the `multiaddr` property of the `multiaddrInfo` and not the entire `multiaddrInfo` as the remaining data should be used internally. Should we consider having two datastructure instead? | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| Further API methods will probably be added in the context of multiaddr `ttl` and multiaddr confidence. | ||
|
|
||
| **Not Yet Implemented**: Multiaddr Confidence | ||
|
|
||
| ## Key Book | ||
|
|
||
| The `keyBook` tracks the keys of the peers. | ||
|
|
||
| **Not Yet Implemented** | ||
|
|
||
| ## Protocol Book | ||
|
|
||
| The `protoBook` holds the identifiers of the protocols supported by each peer. The protocols supported by each peer are dynamic and will change over time. | ||
|
|
||
| `Map<string, Set<string>>` | ||
|
|
||
| A `peerId.toString()` identifier mapping to a `Set` of protocol identifier strings. | ||
|
|
||
| - `protoBook.set()` | ||
| - `protoBook.get()` | ||
| - `protoBook.has()` | ||
| - `protoBook.delete()` | ||
| - `protoBook.supports()` | ||
vasco-santos marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - `protoBook.peers()` | ||
|
|
||
| ## Metadata Book | ||
|
|
||
| **Not Yet Implemented** | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.