Retrying nemotron-parse API calls receiving 408 timeouts

[jamesbraza]

When serving nemotron-parse using vllm==0.14.1 on Modal with an Nvidia H200 with a FastAPI proxy to make vLLM match Nvidia NIM, we see 408 status codes coming back sometimes.

litellm==1.81.1 casts these 408 to litellm.exceptions.TimeoutError: https://github.com/BerriAI/litellm/blob/v1.81.1-nightly/litellm/exceptions.py#L243

This PR moves us to also retry those 408 errors.

---

Note

Expands retry logic for _call_nvidia_api to handle 408 Request Timeout surfaced by LiteLLM.

• Adds _is_litellm_timeout_with_408 and updates Tenacity decorator to retry_if_exception(_is_litellm_timeout_with_408) alongside TimeoutError
• Imports http and retry_if_exception to support new condition

^{Written by Cursor Bugbot for commit 954774a. Configure here.}

[Copilot]

Pull request overview

This PR adds retry logic to handle 408 (Request Timeout) status codes from the nemotron-parse API when served via vLLM on Modal. According to the PR description, litellm version 1.81.1 casts these 408 responses to litellm.exceptions.Timeout exceptions, and the new retry logic ensures these timeout errors are retried along with the existing rate limit timeouts.

Changes:

• Added a helper function to detect litellm timeout exceptions with 408 status codes
• Extended the retry decorator on _call_nvidia_api to retry on inference timeouts (408) in addition to rate limit timeouts

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.