Skip to content

roadmap.md

Latest commit

 

History

History
337 lines (263 loc) · 11.4 KB

roadmap.md

File metadata and controls

337 lines (263 loc) · 11.4 KB

Roadmap

Vision

Build a reusable, production-grade blockchain indexing platform that provides:

  • Canonical on-chain data (source of truth)
  • Derived protocol-level insights (modular & extensible)
  • High-performance APIs for downstream applications

This repo is NOT an intelligence system (Argus). This is the data foundation layer.

Architecture

Core Principle

Separate:

  • Truth Layer (Core Indexer) — raw, canonical blockchain data
  • Derived Layer (Protocol Modules) — decoded, interpretable events

System Layers

1. Core Indexer Layer (Always Present)

  • Block ingestion
  • Transactions & receipts
  • Logs
  • ERC-20 transfers
  • ERC-721 ownership
  • ERC-1155 balances
  • NFT metadata pipeline
  • Data integrity & reconciliation
  • Read models (holdings, stats, summaries)
  • Public API with Swagger docs

2. Protocol Decoder Layer (Modular & Pluggable)

  • DEX swaps (Uniswap V2 implemented)
  • Lending activity
  • NFT marketplace events
  • Bridges / cross-chain
  • Vault interactions
  • Custom contracts

Design Principles

  1. Truth vs Derived — protocol data is always rebuildable, reorg-safe, and decoupled from ingestion
  2. Modularity — each protocol is an isolated module implementing ProtocolDecoder interface
  3. Replayability — backfills, reorg handling, and full rebuilds supported
  4. Performance First — partition large tables, composite indexes, cursor pagination, read models

Phase 1 — Foundation

Status: COMPLETE

  • Block ingestion pipeline
  • Transactions + receipts
  • Logs ingestion
  • ERC-20 transfer decoding
  • Token metadata discovery (name, symbol, decimals via RPC)
  • Checkpoint-based resumable sync
  • Confirmation depth (default 6 blocks)
  • Idempotent processing (INSERT ON CONFLICT DO NOTHING)
  • Backfill system with pause/resume/progress tracking
  • Reorg detection (parent hash validation) + rollback + audit trail
  • Partitioned tables (transactions, receipts, logs, token_transfers, nft_transfers)
  • Partition manager service (auto-creates partitions ahead of sync)
  • Composite indexes for address lookups (from_address, to_address + block_number DESC)
  • Read models: address_summaries, token_stats
  • MetricsService (counters, gauges, rate tracking, error recording)
  • Admin status/metrics/checkpoints/reorgs endpoints
  • Search endpoint (tx hash, block number/hash, address)
  • Response DTOs with Swagger decorators
  • Validation pipes (class-validator/class-transformer)
  • Feature-based NestJS module organization
  • 87 integration + unit tests

Phase 2 — NFT Support

Status: COMPLETE

ERC-721

  • Transfer decoding (4-topic Transfer logs)
  • Contract standard detection (ERC-165 supportsInterface probing)
  • Contract standard persistence (contract_standards table)
  • erc721_ownership table — PK (token_address, token_id), exactly one owner
  • Ownership updates: mint/transfer/burn
  • Reorg-safe rollback of ownership

ERC-1155

  • TransferSingle decoding (ABI decode from data field)
  • TransferBatch decoding (expand arrays into individual rows)
  • erc1155_balances table — PK (token_address, token_id, owner_address)
  • Balance tracking: increment/decrement with quantity
  • Burn: delete row when balance reaches 0

NFT Metadata

  • nft_token_metadata table with fetch status (PENDING/FETCHING/SUCCESS/FAILED/RETRYABLE)
  • NftMetadataService — tokenURI fetch, IPFS/Arweave/data URI normalization
  • Queue-backed metadata worker (Bull NFT_METADATA queue)
  • Fire-and-forget from decoder — never blocks ingestion

NFT Read Models

  • address_nft_holdings — fast lookup of NFTs held by address
  • nft_contract_stats — collection-level transfer counts, holder counts
  • Incremental updates during decode/backfill
  • Cursor pagination on all NFT transfer history endpoints

NFT Reconciliation

  • NftReconciliationService — validate + rebuild all derived tables
  • Rebuild order: nft_transfers → ownership/balances → holdings → stats
  • Admin endpoints: POST /admin/nfts/reconcile, GET /admin/nfts/validate
  • Drift detection: ownership mismatches, holdings gaps, stats errors
  • dryRun mode for validation without mutation

NFT API

  • GET /nfts/collections/:address/transfers (cursor paginated)
  • GET /nfts/collections/:address/tokens/:tokenId (metadata + owners)
  • GET /nfts/collections/:address/tokens/:tokenId/transfers
  • GET /nfts/collections/:address/tokens/:tokenId/owners
  • GET /addresses/:address/nfts (from holdings read model)
  • GET /addresses/:address/nft-transfers (cursor paginated)

Phase 3 — Protocol Decoder Framework

Status: COMPLETE

  • ProtocolDecoder interface (decodeBlock, rollbackFrom, rebuild)
  • ProtocolRegistryService (register, decodeBlock, rollbackFrom)
  • Self-registration via onModuleInit
  • protocol_contracts table (cross-protocol contract registry)
  • dex_pairs table (pair metadata: token0, token1, factory)
  • dex_swaps table (normalized swap events)
  • Integrated into decode worker queue processor
  • Integrated into backfill runner (inline decode)
  • Reorg rollback for protocol-derived tables
  • Protocol API: GET /protocols/dex/swaps, /pairs, /addresses/:address/dex-swaps

Uniswap V2

  • Swap event decoding (topic0 = 0xd78ad95f...)
  • Pair detection: memory cache → DB → RPC probe (token0/token1/factory)
  • Persistence in dex_swaps + dex_pairs + protocol_contracts
  • Idempotent (orIgnore on unique tx_hash + log_index)
  • Mint / Burn (LP) events
  • Sync event for reserve tracking

Phase 4 — Tier 1 Protocols

ERC-20 Approvals — COMPLETE

  • Approval(address indexed owner, address indexed spender, uint256 value) decoding
  • token_approvals table (historical events, unique on tx_hash + log_index)
  • token_allowances_current table — PK (token_address, owner_address, spender_address)
  • Erc20ApprovalDecoderService — 3-topic Approval logs, value in data
  • Inline decode in backfill runner
  • Reorg rollback for approvals + allowances
  • API: GET /tokens/:address/allowance/:owner/:spender
  • API: GET /addresses/:address/approvals (paginated history)
  • API: GET /addresses/:address/allowances (current state)

Uniswap V3 — COMPLETE

  • Swap(address,address,int256,int256,uint160,uint128,int24) decoding
  • Pool detection: memory cache → DB (dex_pairs) → RPC probe (token0/token1/fee/factory)
  • Reuse dex_swaps with protocol_name = UNISWAP_V3
  • Signed amount mapping: positive → amountIn, negative → amountOut
  • Self-registering via ProtocolDecoder + onModuleInit
  • Inline decode in backfill runner
  • Idempotent (orIgnore on unique tx_hash + log_index)
  • Liquidity position tracking (optional, future)

NFT Marketplaces — COMPLETE

Seaport (OpenSea):

  • OrderFulfilled event decoding (nested SpentItem/ReceivedItem struct arrays)
  • nft_sales table (collection, tokenId, seller, buyer, paymentToken, totalPrice)
  • Sale direction detection: offer NFT + consideration ETH/ERC20 = sell, vice versa = buy
  • Price extraction: sum consideration payments to seller

Blur:

  • OrdersMatched event decoding (nested Order/Input struct pairs)
  • Reuses nft_sales table with protocol_name = BLUR
  • Seller/buyer determination from sell/buy order sides + maker/taker roles

Shared:

  • Self-registering via ProtocolDecoder + onModuleInit
  • Reorg rollback for both protocols
  • API: GET /nfts/collections/:address/sales (cursor paginated)
  • API: GET /addresses/:address/nft-sales (cursor paginated)

Phase 5 — Tier 2 Protocols

Lending: Aave — COMPLETE

  • lending_events table (generic, reusable for Compound/Aave/etc.)
  • AaveDecoder — Deposit/Supply, Withdraw, Borrow, Repay, LiquidationCall
  • Supports both Aave V2 (Deposit) and V3 (Supply) event signatures
  • Liquidation decoding with collateralAsset, debtToCover, liquidatedCollateral, liquidator
  • Self-registering via ProtocolDecoder + onModuleInit
  • Inline in backfill via protocol registry
  • Reorg rollback
  • API: GET /protocols/lending/events?protocolName=&eventType=&assetAddress= (cursor paginated)
  • API: GET /addresses/:address/lending (cursor paginated)

Lending: Compound — COMPLETE

  • CompoundDecoder — Mint (→DEPOSIT), Redeem (→WITHDRAW), Borrow, RepayBorrow, LiquidateBorrow
  • Reuses lending_events table with protocol_name = COMPOUND
  • Compound events use non-indexed params (all in data) — different decode pattern from Aave
  • Self-registering, backfill-integrated, reorg-safe

ERC-4626 Vaults — COMPLETE

  • Erc4626Decoder — Deposit and Withdraw events
  • Reuses lending_events table with protocol_name = ERC4626
  • Self-registering, backfill-integrated, reorg-safe

Bridges

  • LayerZero
  • Hop
  • Stargate
  • Native L2 bridges

Phase 6 — Advanced Protocols

Curve

  • Swaps
  • Liquidity events

Balancer

  • Multi-token pool support

GMX (Perps)

  • Position lifecycle
  • Liquidations

ENS

  • Name registration
  • Transfers

Phase 7 — Meta / Infrastructure

Aggregators

  • 1inch
  • 0x
  • Paraswap Note: Requires multi-call reconstruction

Safe (Gnosis Safe)

  • Transaction execution
  • Module interactions

ERC-4337

  • UserOperation decoding

Future Infrastructure

Multi-chain support

  • Chain-aware entity design (chain_id column or separate schemas)
  • Multiple chain provider instances
  • Per-chain workers and checkpoints

Internal transactions / traces

  • debug_traceBlockByNumber or trace_block
  • internal_transactions table
  • Requires archive node

Contract verification

  • Store verified contract ABIs
  • Auto-decode all events for verified contracts

Frontend Explorer — COMPLETE

  • Next.js 15 with App Router + Tailwind CSS v4
  • Home page: stats dashboard + latest blocks + search bar
  • Block list page: block table (age, miner, gas, base fee)
  • Block detail page: header fields + transaction table
  • Transaction detail page: status, from/to, value, gas, token transfers, event logs
  • Address page with 6 tabs: Transactions, Tokens, NFTs, DEX Swaps, Allowances, Lending
  • Token detail page: symbol, decimals, standard, recent transfers
  • NFT collection page: transfer history table
  • NFT token detail page: metadata image, description, owners, transfer history
  • Search: auto-redirect to block/tx/address
  • Pagination component (Previous/Next with page numbers)
  • Dark theme, responsive, server-rendered
  • Real-time updates via WebSocket

Out of Scope (Handled by Argus)

Do NOT include:

  • Wallet intelligence scoring
  • Smart money detection
  • Alerting systems
  • Trading strategies
  • Machine learning models
  • Cross-wallet behavioral analysis

Implementation Priority

Difficulty Protocols
Easy ERC-20 Approvals, ERC-4626
Medium Uniswap V3, Seaport, Aave
Hard Aggregators, Router-based swaps

Execution Order

  1. Core stability (Phase 1) DONE
  2. NFT support (Phase 2) DONE
  3. Protocol framework + Uniswap V2 (Phase 3) DONE
  4. ERC-20 Approvals + Uniswap V3 DONE
  5. NFT marketplaces (Seaport + Blur) DONE
  6. Lending: Aave + Compound DONE
  7. ERC-4626 Vaults DONE
  8. Bridges ← NEXT
  9. Advanced protocols