test: Request Migration Docs and E2E vLLM Tests (#2177)

Signed-off-by: Jacky <[email protected]>
Co-authored-by: Neelay Shah <[email protected]>

components/backends/llama_cpp/README.md

@@ -13,16 +13,10 @@ python -m dynamo.llama_cpp --model-path /data/models/Qwen3-0.6B-Q8_0.gguf [args]
 
 ## Request Migration
 
-In a [Distributed System](#distributed-system), a request may fail due to connectivity issues between the Frontend and the Backend.
+You can enable [request migration](../../../docs/architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker:
 
-The Frontend will automatically track which Backends are having connectivity issues with it and avoid routing new requests to the Backends with known connectivity issues.
-
-For ongoing requests, there is a `--migration-limit` flag which can be set on the Backend that tells the Frontend how many times a request can be migrated to another Backend should there be a loss of connectivity to the current Backend.
-
-For example,
 ```bash
 python3 -m dynamo.llama_cpp ... --migration-limit=3
 ```
-indicates a request to this model may be migrated up to 3 times to another Backend, before failing the request, should the Frontend detects a connectivity issue to the current Backend.
 
-The migrated request will continue responding to the original request, allowing for a seamless transition between Backends, and a reduced overall request failure rate at the Frontend for enhanced user experience.
+This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/architecture/request_migration.md) documentation for details on how this works.

components/backends/sglang/README.md

@@ -143,19 +143,13 @@ When using MoE models, you can also use the our implementation of the native SGL
 
 ## Request Migration
 
-In a [Distributed System](#distributed-system), a request may fail due to connectivity issues between the Frontend and the Backend.
+You can enable [request migration](../../../docs/architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker:
 
-The Frontend will automatically track which Backends are having connectivity issues with it and avoid routing new requests to the Backends with known connectivity issues.
-
-For ongoing requests, there is a `--migration-limit` flag which can be set on the Backend that tells the Frontend how many times a request can be migrated to another Backend should there be a loss of connectivity to the current Backend.
-
-For example,
 ```bash
 python3 -m dynamo.sglang ... --migration-limit=3
 ```
-indicates a request to this model may be migrated up to 3 times to another Backend, before failing the request, should the Frontend detects a connectivity issue to the current Backend.
 
-The migrated request will continue responding to the original request, allowing for a seamless transition between Backends, and a reduced overall request failure rate at the Frontend for enhanced user experience.
+This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/architecture/request_migration.md) documentation for details on how this works.
 
 ## Advanced Examples
 

components/backends/trtllm/README.md

@@ -263,19 +263,13 @@ Dynamo with TensorRT-LLM supports two methods for transferring KV cache in disag
 
 ## Request Migration
 
-In a [Distributed System](#distributed-system), a request may fail due to connectivity issues between the Frontend and the Backend.
+You can enable [request migration](../../../docs/architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker:
 
-The Frontend will automatically track which Backends are having connectivity issues with it and avoid routing new requests to the Backends with known connectivity issues.
-
-For ongoing requests, there is a `--migration-limit` flag which can be set on the Backend that tells the Frontend how many times a request can be migrated to another Backend should there be a loss of connectivity to the current Backend.
-
-For example,
 ```bash
 python3 -m dynamo.trtllm ... --migration-limit=3
 ```
-indicates a request to this model may be migrated up to 3 times to another Backend, before failing the request, should the Frontend detects a connectivity issue to the current Backend.
 
-The migrated request will continue responding to the original request, allowing for a seamless transition between Backends, and a reduced overall request failure rate at the Frontend for enhanced user experience.
+This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/architecture/request_migration.md) documentation for details on how this works.
 
 ## Client
 

components/backends/vllm/README.md

@@ -235,16 +235,10 @@ The [documentation](https://docs.vllm.ai/en/v0.9.2/configuration/serve_args.html
 
 ## Request Migration
 
-In a [Distributed System](#distributed-system), a request may fail due to connectivity issues between the Frontend and the Backend.
+You can enable [request migration](../../../docs/architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker:
 
-The Frontend will automatically track which Backends are having connectivity issues with it and avoid routing new requests to the Backends with known connectivity issues.
-
-For ongoing requests, there is a `--migration-limit` flag which can be set on the Backend that tells the Frontend how many times a request can be migrated to another Backend should there be a loss of connectivity to the current Backend.
-
-For example,
 ```bash
 python3 -m dynamo.vllm ... --migration-limit=3
 ```
-indicates a request to this model may be migrated up to 3 times to another Backend, before failing the request, should the Frontend detects a connectivity issue to the current Backend.
 
-The migrated request will continue responding to the original request, allowing for a seamless transition between Backends, and a reduced overall request failure rate at the Frontend for enhanced user experience.
+This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/architecture/request_migration.md) documentation for details on how this works.

components/backends/vllm/src/dynamo/vllm/handlers.py

@@ -106,6 +106,7 @@ def cleanup(self):
 
     async def generate(self, request):
         request_id = str(uuid.uuid4().hex)
+        logger.debug(f"New Request ID: {request_id}")
 
         prompt = TokensPrompt(prompt_token_ids=request["token_ids"])
 
@@ -164,6 +165,8 @@ def __init__(self, component, engine, default_sampling_params):
 
     async def generate(self, request):
         request_id = request["request_id"]
+        logger.debug(f"New Prefill Request ID: {request_id}")
+
         prompt = TokensPrompt(prompt_token_ids=request["token_ids"])
         sampling_params = msgspec.convert(request["sampling_params"], SamplingParams)
 

docs/architecture/request_migration.md

@@ -0,0 +1,114 @@
+# Request Migration Architecture
+
+This document describes how Dynamo implements request migration to handle worker failures gracefully during LLM text generation. Request migration allows in-progress requests to continue on different workers when the original worker becomes unavailable, providing fault tolerance and improved user experience.
+
+## Overview
+
+Request migration is implemented through a Migration operator that sits in the LLM processing pipeline between the Backend operator and the service backend. When a worker fails during request processing, the migration system preserves the partial generation state and recreates the request on a new worker to continue from where the previous worker left off.
+
+## Architecture Components
+
+### Migrator
+
+The migration system is integrated into the LLM processing pipeline between the frontend preprocessing and the actual service backends. This positioning allows it to intercept all communication flows and manage failure scenarios transparently.
+
+Key responsibilities:
+- Intercepts all requests and responses flowing through the pipeline
+- Detects worker failure scenarios through error pattern matching
+- Manages retry logic with configurable migration limits
+- Tracks partial response state for seamless continuation
+
+### Migration Limit Configuration
+
+Each model can be configured with a migration limit parameter that specifies the maximum number of times a request can be migrated to another worker:
+
+- Default behavior: no migration allowed
+- Can be set independently for different engine types
+- Applicable to LLM worker nodes that perform inference
+- Allows engines to override user-specified limits for compatibility
+
+## Token State Tracking and Request Migration
+
+The core of the migration system is the ability to preserve and continue partial generations through token state management. This ensures that when a worker fails mid-generation, the new worker can seamlessly continue from the exact point of failure.
+
+### Token Accumulation Process
+
+When a request is being processed and responses are flowing back from a worker, the migration system tracks every token that has been successfully generated:
+
+1. **Initial Request State**: The system starts with the original preprocessed request containing the initial prompt tokens.
+
+2. **Response Tracking**: As each response arrives from the worker, the migration system extracts the newly generated tokens and appends them to the request's token sequence. This creates accumulates all tokens that have been generated.
+
+3. **Token Count Management**: The system also updates the remaining token budget to reflect the number of tokens already generated, ensuring that the total generation stays within the originally requested limits.
+
+### Migration Trigger Scenarios
+
+The migration system handles two distinct failure scenarios:
+
+#### 1. New Request Migration (Initial Connection Failure)
+
+**Scenario**: Worker is unreachable when creating the initial connection.
+
+**Error Pattern**: Communication system reports chosen worker instance is unavailable.
+
+**Migration Process**:
+- Detects connection failure during initial stream setup
+- Decrements migration retry count
+- Attempts to create a new stream with the original request
+- No partial state to preserve since generation hasn't started
+
+#### 2. Ongoing Request Migration (Mid-Stream Disconnection)
+
+**Scenario**: Connection lost during active generation after partial responses have been received.
+
+**Error Pattern**: Stream termination detected before generation completion.
+
+**Migration Process**:
+
+1. **Failure Detection**: The system detects the stream disconnection through error monitoring.
+
+2. **State Preservation**: At this point, the request's token sequence contains both the original prompt tokens and all successfully generated tokens from the failed worker.
+
+3. **New Stream Creation**: A fresh stream is created with the accumulated request state, ensuring the new worker has complete context.
+
+4. **Continuation**: The new worker receives the request with the full token context and continues generation from the exact point where the previous worker left off.
+
+### Seamless Token Flow and Request State Evolution
+
+From the client's perspective, the token stream appears continuous and uninterrupted. The client receives tokens from the first worker until failure occurs, then seamlessly continues receiving tokens from the backup worker without any indication of the underlying migration.
+
+The request state evolves dynamically during processing. Initially, the request contains only the original prompt tokens. As generation proceeds, each successfully generated token is appended to the request's token sequence, creating a growing record of the complete conversation context.
+
+When a migration occurs, this accumulated state is transferred to the new worker, which uses it to reconstruct the complete context. The new worker then continues generation as if it had been processing the request from the beginning, but starting from the current position in the sequence.
+
+The migration is transparent because:
+1. No tokens are lost or duplicated during the transition
+2. The new worker has complete context via the accumulated token sequence
+3. Generation continues from the exact failure point
+4. Response streaming maintains consistent format and timing
+
+This token accumulation mechanism ensures that migrations are truly seamless, preserving all computational work and maintaining generation quality across worker transitions.
+
+## Benefits
+
+1. **Fault Tolerance**: System continues operating during individual worker failures
+2. **Resource Efficiency**: Partial generations are preserved rather than restarted
+3. **Seamless User Experience**: Users experience no interruption during worker failures
+4. **Configurable Behavior**: Migration limits allow tuning based on deployment requirements
+5. **No Token Loss**: Complete preservation of generation state across migrations
+
+## Design Considerations
+
+The migration system is designed with several important architectural considerations:
+
+**Engine Compatibility**: Different LLM engines may have varying capabilities for handling migrated requests. The system allows engines to override migration settings to ensure compatibility and correctness.
+
+**Multi-Model Support**: Since a frontend may serve multiple models simultaneously, migration limits can be configured at the engine level, providing flexibility for different model types with varying reliability characteristics.
+
+**State Management**: The system carefully tracks not only token sequences but also metadata such as remaining token budgets, stop conditions, and sampling parameters to ensure complete state preservation.
+
+**Error Handling**: The migration system distinguishes between different types of failures and applies appropriate recovery strategies for each scenario.
+
+## Operational Impact
+
+Request migration fundamentally changes how the system handles failures, moving from a "fail-fast" approach to a "graceful degradation" model. This architectural shift enables higher availability and better resource utilization while maintaining the same external API contract for clients.

docs/guides/backend.md

@@ -71,6 +71,7 @@ The `model_type` can be:
 - `model_name`: The name to call the model. Your incoming HTTP requests model name must match this. Defaults to the hugging face repo name, the folder name, or the GGUF file name.
 - `context_length`: Max model length in tokens. Defaults to the model's set max. Only set this if you need to reduce KV cache allocation to fit into VRAM.
 - `kv_cache_block_size`: Size of a KV block for the engine, in tokens. Defaults to 16.
+- `migration_limit`: Maximum number of times a request may be [migrated to another Instance](../architecture/request_migration.md). Defaults to 0.
 
 See `components/backends` for full code examples.
 

docs/guides/dynamo_run.md

@@ -211,19 +211,13 @@ The KV-aware routing arguments:
 
 ### Request Migration
 
-In a [Distributed System](#distributed-system), a request may fail due to connectivity issues between the HTTP Server and the Worker Engine.
+In a [Distributed System](#distributed-system), you can enable [request migration](../architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker:
 
-The HTTP Server will automatically track which Worker Engines are having connectivity issues with it and avoid routing new requests to the Engines with known connectivity issues.
-
-For ongoing requests, there is a `--migration-limit` flag which can be set on the Worker Engines that tells the HTTP Server how many times a request can be migrated to another Engine should there be a loss of connectivity to the current Engine.
-
-For example,
 ```bash
-dynamo-run in=dyn://... out=vllm ... --migration-limit=3
+dynamo-run in=dyn://... out=<engine> ... --migration-limit=3
 ```
-indicates a request to this model may be migrated up to 3 times to another Engine, before failing the request, should the HTTP Server detects a connectivity issue to the current Engine.
 
-The migrated request will continue responding to the original request, allowing for a seamless transition between Engines, and a reduced overall request failure rate at the HTTP Server for enhanced user experience.
+This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../architecture/request_migration.md) documentation for details on how this works.
 
 ## Full usage details
 

tests/fault_tolerance/README.md

@@ -0,0 +1,80 @@
+# Fault Tolerance Tests
+
+This directory contains end-to-end tests for Dynamo's fault tolerance capabilities.
+
+## Tests
+
+### `test_request_migration.py`
+
+Tests worker fault tolerance with migration support using the `test_request_migration_vllm` function. This test:
+
+0. Downloads the DeepSeek-R1-Distill-Llama-8B model from HuggingFace if not already cached
+1. Starts a Dynamo frontend using `python -m dynamo.frontend` with round-robin routing
+2. Starts 2 workers sequentially using `python3 -m dynamo.vllm` with specific configuration:
+   - Model: `deepseek-ai/DeepSeek-R1-Distill-Llama-8B`
+   - `--enforce-eager`, `--gpu-memory-utilization 0.45`
+   - `--max-model-len 8192`, `--migration-limit 3`
+3. Waits for both workers to be fully ready (looking for "Reading Events from" messages)
+4. Sends a test request ("Who are you?", 100 tokens) to determine which worker handles requests
+5. Determines primary/backup worker roles based on round-robin routing and log analysis
+6. Sends a long completion request ("Tell me a long long long story about yourself?", 8000 tokens) in a separate thread
+7. Waits 0.5 seconds, then kills the primary worker using SIGKILL process group termination
+8. Verifies the request completes successfully despite the worker failure (with 240s timeout)
+9. Checks that the frontend logs contain "Stream disconnected... recreating stream..." indicating migration occurred
+
+## Prerequisites
+
+- vLLM backend installed (`pip install ai-dynamo-vllm`)
+- NATS and etcd services running (provided by `runtime_services` fixture)
+- Access to DeepSeek-R1-Distill-Llama-8B model (automatically downloaded from HuggingFace)
+- Sufficient GPU memory (test uses 0.45 GPU memory utilization)
+
+## Running the Tests
+
+To run the fault tolerance tests:
+
+```bash
+# Run all fault tolerance tests
+pytest /workspace/tests/fault_tolerance
+
+# Run specific test with verbose output
+pytest /workspace/tests/fault_tolerance/test_request_migration.py::test_request_migration_vllm -v
+
+# Run with specific markers
+pytest -m "e2e and vllm" /workspace/tests/fault_tolerance
+
+# Run with debug logging
+pytest /workspace/tests/fault_tolerance/test_request_migration.py::test_request_migration_vllm -v -s
+```
+
+## Test Markers
+
+- `@pytest.mark.e2e`: End-to-end test
+- `@pytest.mark.vllm`: Requires vLLM backend
+- `@pytest.mark.gpu_1`: Requires single GPU access
+- `@pytest.mark.slow`: Known to be slow (due to model loading and inference)
+
+## Environment Variables
+
+- `DYN_LOG`: Set to `debug` or `trace` for verbose logging (automatically set to `debug` by worker processes)
+- `CUDA_VISIBLE_DEVICES`: Control which GPUs are used for testing
+
+## Expected Test Duration
+
+The test typically takes 2-3 minutes to complete, including:
+- Model download/loading time (if not cached) - can take 1-2 minutes for first run
+- Worker startup and registration
+- Request processing and response validation
+- Worker failure simulation and migration
+- Cleanup
+
+## Troubleshooting
+
+If tests fail:
+
+1. Check that NATS and etcd services are running
+2. Verify vLLM backend is properly installed
+3. Ensure sufficient GPU memory is available (test requires ~45% GPU memory)
+4. Check internet connectivity for model download from HuggingFace
+5. Review test logs for specific error messages
+6. Verify that the DeepSeek-R1-Distill-Llama-8B model can be accessed