A fast Ethereum consensus layer crawler. Discovers nodes via discv5 and, optionally, dials them over libp2p to collect beacon chain Status, MetaData, and Ping responses.
The discv5 stage runs many independent crawl instances in parallel, each with its own keypair, UDP socket, and session cache. Sharing only the discovered-node set, the instances avoid contention on go-ethereum's per-listener rate limits and saturate outbound bandwidth instead.
| Path | Purpose |
|---|---|
cmd/turbocrawl |
CLI entry point (cobra). |
discv5 |
Multi-instance discv5 crawler, bootnodes, config. |
dialer |
libp2p dialer for Eth2 Req/Resp (Status v1/v2, MetaData v0/v1/v2, Ping v1). |
The discv5 crawler spins up N independent listeners (defaults to one per CPU, capped). Each listener has its own ECDSA key, enode.LocalNode, session cache, and 16MB-buffered UDP socket. A shared sync.Map tracks discovered and queried node IDs. A 1M-entry channel feeds work to the listeners; each listener runs random target lookups against its own routing table and forwards new ENRs back to the shared set.
When --dial is set, every discovered node carrying an eth2 ENR entry is handed to the libp2p dialer. The dialer establishes a connection (QUIC preferred, TCP fallback), negotiates the matching protocol version per peer, and records Status, MetaData, and Ping results.
go build ./cmd/turbocrawl
# Discovery only, mainnet, 60s
./turbocrawl
# Discovery + libp2p dial, holesky, JSON logs
./turbocrawl --network holesky --dial --log-format jsonUseful flags:
-n, --network mainnet | holesky | sepolia
-t, --timeout crawl duration (default 60s)
-w, --workers discovery workers (0 = NumCPU * 256)
-d, --dial dial discovered Eth2 nodes via libp2p
--dial-workers libp2p dial workers (default 64)
--prefer-quic prefer QUIC over TCP (default true)
-k, --key private key file (random if omitted)
Licensed under either of Apache License 2.0 or MIT license at your option. Unless you explicitly state otherwise, any contribution submitted for inclusion shall be dual licensed as above, without additional terms or conditions.