Skip to content

DOCKER_SETUP.md

Latest commit

 

History

History
504 lines (363 loc) · 9.6 KB

DOCKER_SETUP.md

File metadata and controls

504 lines (363 loc) · 9.6 KB

Docker Setup Guide

Home > Docs > Installation > Docker Setup

This guide provides detailed information about Docker setup and configuration for EverMemOS.

Table of Contents

Overview

EverMemOS uses Docker Compose to manage four essential services:

  • MongoDB - Primary database
  • Elasticsearch - Keyword search engine
  • Milvus - Vector database
  • Redis - Cache service

All services are defined in docker-compose.yml and can be started with a single command.

Services Overview

Service Host Port Container Port Purpose Memory Usage
MongoDB 27017 27017 Primary database for storing memory cells and profiles ~500MB
Elasticsearch 19200 9200 Keyword search engine (BM25) ~2GB
Milvus 19530 19530 Vector database for semantic retrieval ~1GB
Redis 6379 6379 Cache service ~100MB

Total Memory Requirements: Approximately 4GB minimum

Quick Start

Start All Services

docker-compose up -d

The -d flag runs containers in detached mode (background).

Check Service Status

docker-compose ps

Expected output:

NAME                STATUS              PORTS
mongodb             Up 2 minutes        0.0.0.0:27017->27017/tcp
elasticsearch       Up 2 minutes        0.0.0.0:19200->9200/tcp
milvus-standalone   Up 2 minutes        0.0.0.0:19530->19530/tcp
redis               Up 2 minutes        0.0.0.0:6379->6379/tcp

View Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f mongodb
docker-compose logs -f elasticsearch

Stop Services

docker-compose down

To also remove volumes (⚠️ deletes all data):

docker-compose down -v

Service Configuration

MongoDB

Configuration:

  • Version: 7.0
  • Default credentials: admin / memsys123
  • Port: 27017

Connection String:

mongodb://admin:memsys123@localhost:27017

Collections:

  • memcells - Atomic memory units
  • episodes - Episodic memories
  • profiles - User profiles
  • preferences - User preferences
  • And more...

See MongoDB Guide for detailed database information.

Elasticsearch

Configuration:

  • Version: 8.x
  • Security: Disabled (for local development)
  • Port: 19200 (mapped from container port 9200)

Connection URL:

http://localhost:19200

Test Connection:

curl http://localhost:19200

Indices:

  • Memory cells indexed for keyword search
  • BM25 algorithm for relevance ranking

Milvus

Configuration:

  • Version: 2.4+
  • Standalone deployment
  • Port: 19530

Connection:

from pymilvus import connections
connections.connect(host="localhost", port="19530")

Collections:

  • Vector embeddings for semantic search
  • Multiple index types supported

Redis

Configuration:

  • Version: 7.x
  • Port: 6379
  • No password (for local development)

Connection:

redis-cli -h localhost -p 6379 ping

Usage:

  • Caching frequently accessed data
  • Session management
  • Temporary data storage

Port Mapping

Understanding Port Mapping

Format: HOST_PORT:CONTAINER_PORT

When connecting from your local machine, always use the HOST PORT.

Default Port Mapping

Service Host Port Container Port URL from Host
MongoDB 27017 27017 localhost:27017
Elasticsearch 19200 9200 http://localhost:19200
Milvus 19530 19530 localhost:19530
Redis 6379 6379 localhost:6379

Custom Port Configuration

To use different host ports, edit docker-compose.yml:

services:
  mongodb:
    ports:
      - "27018:27017"  # Use port 27018 on host

  elasticsearch:
    ports:
      - "9200:9200"  # Use standard port 9200 on host

After changing ports, update your .env file accordingly:

MONGODB_URI=mongodb://admin:memsys123@localhost:27018
ELASTICSEARCH_URL=http://localhost:9200

Volume Management

Data Persistence

All services use Docker volumes to persist data:

# List volumes
docker volume ls

# Inspect a volume
docker volume inspect evermemos_mongodb_data
docker volume inspect evermemos_elasticsearch_data
docker volume inspect evermemos_milvus_data
docker volume inspect evermemos_redis_data

Backup Data

MongoDB:

# Backup
docker exec mongodb mongodump --username admin --password memsys123 --authenticationDatabase admin --out /tmp/backup

# Copy backup out of container
docker cp mongodb:/tmp/backup ./mongodb_backup

Elasticsearch:

# Take snapshot (requires snapshot repository setup)
curl -X PUT "http://localhost:19200/_snapshot/my_backup/snapshot_1?wait_for_completion=true"

Restore Data

MongoDB:

# Copy backup into container
docker cp ./mongodb_backup mongodb:/tmp/backup

# Restore
docker exec mongodb mongorestore --username admin --password memsys123 --authenticationDatabase admin /tmp/backup

Clear All Data

⚠️ Warning: This will delete all memory data!

docker-compose down -v
docker-compose up -d

Service Management

Restart Services

# Restart all services
docker-compose restart

# Restart specific service
docker-compose restart mongodb

Update Services

# Pull latest images
docker-compose pull

# Recreate containers with new images
docker-compose up -d

Scale Services (Advanced)

Some services can be scaled:

# Not typically needed for local development
docker-compose up -d --scale redis=2

Resource Limits

Edit docker-compose.yml to set memory/CPU limits:

services:
  elasticsearch:
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: '2'

Troubleshooting

Services Won't Start

Check Docker Status:

docker info

Check Logs:

docker-compose logs

Common Issues:

  • Port conflicts: Another service using the same port
  • Insufficient memory: Docker doesn't have enough RAM allocated
  • Permission issues: Docker doesn't have permission to create volumes

Port Conflicts

Find what's using a port:

# macOS/Linux
lsof -i :27017
lsof -i :19200

# Or use netstat
netstat -an | grep 27017

Solutions:

  • Stop conflicting service
  • Change host port in docker-compose.yml

Out of Memory

Symptoms:

  • Elasticsearch crashes
  • Milvus becomes unresponsive
  • Services restart repeatedly

Solutions:

  1. Increase Docker Memory (Docker Desktop):

    • Docker Desktop > Preferences > Resources
    • Increase memory to 8GB or more

  2. Reduce Elasticsearch Heap: Edit docker-compose.yml:

    elasticsearch:
  environment:
    - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

  3. Close other applications to free up memory

Connection Refused

Check service is running:

docker-compose ps

Verify port mapping:

docker-compose port mongodb 27017

Test connectivity:

# MongoDB
telnet localhost 27017

# Elasticsearch
curl http://localhost:19200

# Redis
redis-cli -h localhost -p 6379 ping

Elasticsearch Yellow/Red Status

Check cluster health:

curl http://localhost:19200/_cluster/health?pretty

Common cause: Single-node cluster (expected for local development)

Set to single-node mode:

elasticsearch:
  environment:
    - discovery.type=single-node

Advanced Configuration

Custom docker-compose.yml

Create docker-compose.override.yml for custom configuration:

version: '3.8'
services:
  mongodb:
    environment:
      - MONGO_INITDB_ROOT_USERNAME=myuser
      - MONGO_INITDB_ROOT_PASSWORD=mypassword

This file is automatically merged with docker-compose.yml.

Network Configuration

Services communicate on a shared network:

networks:
  evermemos-network:
    driver: bridge

Environment Variables

Pass environment variables to services:

services:
  mongodb:
    environment:
      - MONGO_INITDB_ROOT_USERNAME=${MONGO_USER:-admin}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGO_PASSWORD:-memsys123}

Then create .env.docker:

MONGO_USER=admin
MONGO_PASSWORD=secretpassword

Health Checks

Add health checks to ensure services are ready:

services:
  mongodb:
    healthcheck:
      test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
      interval: 10s
      timeout: 5s
      retries: 5

Production Considerations

⚠️ Note: The default docker-compose.yml is for local development only.

For production deployment:

  1. Change default passwords
  2. Enable authentication for all services
  3. Use external volumes for backups
  4. Configure SSL/TLS
  5. Set resource limits
  6. Use managed services (AWS, Azure, GCP) when possible
  7. Implement monitoring and alerting

See Also