Distributed Key-Value Store

A distributed key-value store built with Go and gRPC, featuring Raft consensus for high availability and consistency.

Features

• Distributed Architecture: Multi-node cluster with automatic leader election
• Raft Consensus: Strong consistency guarantees through Raft protocol
• gRPC API: High-performance RPC communication
• CRUD Operations: Get, Set, Delete, and List operations
• Cluster Management: Join/leave cluster operations
• Health Monitoring: Built-in health checks
• TTL Support: Time-to-live for key expiration

Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Node 1        │    │   Node 2        │    │   Node 3        │
│   (Leader)      │◄──►│   (Follower)    │◄──►│   (Follower)    │
│                 │    │                 │    │                 │
│ ┌─────────────┐ │    │ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │   gRPC API  │ │    │ │   gRPC API  │ │    │ │   gRPC API  │ │
│ └─────────────┘ │    │ └─────────────┘ │    │ └─────────────┘ │
│ ┌─────────────┐ │    │ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │ Raft Engine │ │    │ │ Raft Engine │ │    │ │ Raft Engine │ │
│ └─────────────┘ │    │ └─────────────┘ │    │ └─────────────┘ │
│ ┌─────────────┐ │    │ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │   Storage   │ │    │ │   Storage   │ │    │ │   Storage   │ │
│ └─────────────┘ │    │ └─────────────┘ │    │ └─────────────┘ │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Quick Start

Prerequisites

• Go 1.21 or later
• Protocol Buffers compiler (protoc)

Installation

1. Clone the repository:

git clone <repository-url>
cd distributed-kv-store

2. Install dependencies and generate code:

make setup

Running the Cluster

1. Start the first node (will become leader):

./kv-server --node-id=node1 --address=localhost:8080 --data-dir=./data/node1

2. Start additional nodes:

./kv-server --node-id=node2 --address=localhost:8081 --data-dir=./data/node2 --join=localhost:8080
./kv-server --node-id=node3 --address=localhost:8082 --data-dir=./data/node3 --join=localhost:8080

3. Use the client to interact with the cluster:

./kv-client --server=localhost:8080 set key1 value1
./kv-client --server=localhost:8080 get key1
./kv-client --server=localhost:8080 list-keys

API Reference

gRPC Service Methods

• Get(key) - Retrieve a value by key
• Set(key, value, ttl) - Store a key-value pair with optional TTL
• Delete(key) - Remove a key-value pair
• ListKeys(prefix, limit) - List keys with optional prefix filter
• JoinCluster(node_id, address) - Add a new node to the cluster
• LeaveCluster(node_id) - Remove a node from the cluster
• GetClusterInfo() - Get information about cluster nodes
• HealthCheck() - Check node health status

Configuration

The server can be configured via command-line flags or environment variables:

• --node-id: Unique identifier for this node
• --address: Address to bind the gRPC server
• --data-dir: Directory for Raft logs and storage
• --join: Address of an existing node to join
• --log-level: Logging level (debug, info, warn, error)

Building

make build          # Build all binaries
make server         # Build server only
make client         # Build client only
make test           # Run tests
make clean          # Clean build artifacts