feat: FT Request Cancellation feature and test for 0.5.0

[kthui]

Overview:

This is a side branch for checking in all the smaller Request Cancellation features and fixes and tests, and have everything merged into main before the next 0.5.0 code freeze.

Details:

Features and Fixes:

• Relay cancellation signal between streams over migration layer.
• Relay context_id between streams over migration layer.
• Add debug print on received --migration-limit on mdc.
• Allow Python outgoing calls to optionally supply a context.
• vLLM to abort on context stop / killed.

Tests:

• Python unit tests on incoming and outgoing context cancellation.
• E2E vLLM test on request abort and disagg decode request abort.

Where should the reviewer start?

Recommend starting with /docs updates, and then the followings:

Python cancellation support:

1. Look into /lib/bindings/python/rust/tests/test_cancellation on how Python can cancel / be notified on cancellation.
2. Look into /lib/bindings/python/rust/context.rs and /lib/bindings/python/rust/lib.rs on the implementation.

Rust child and parent context:

1. Look into /lib/runtime/src/engine.rs on how a child context is linked to a parent context.
2. Look into /lib/runtime/src/pipeline/context.rs for an example on how the interface at engine.rs is implemented.

vLLM abort on cancel:

1. Look into /components/backends/vllm/src/dynamo/vllm/handlers.py on how requests are aborted in vLLM on cancel.
2. Look into /tests/fault_tolerance/test_request_cancellation.py for the E2E tests on cancellation.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

N/A

Summary by CodeRabbit

• New Features

  • Introduced request cancellation across generation and streaming, including prefill/decode workflows.
  • Added hierarchical context propagation to abort in-flight requests promptly.
  • Python SDK: new Context class and optional context parameter on client methods to monitor/control cancellation.
  • Improved cancellation logging for better observability.

• Documentation

  • New Request Cancellation architecture doc.
  • Backend and runtime guides updated with cancellation usage and HTTP disconnect behavior.
  • Fault-tolerance docs expanded with cancellation scenarios.

• Tests

  • Added unit, integration, and end-to-end cancellation tests, including vLLM and disaggregated prefill/decode setups.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[michaelfeil]

looks correct on a high level, something we would use. left some questions.

[grahamking]

We should address the Mutex in a follow-up PR.

[kthui]

We should address the Mutex in a follow-up PR.

Yes, DIS-569.

[keivenchang]

Thanks for answering the questions and thanks for this feat!