Skip to content

Fixing 60-sec wait for retrying nemotron-parse API calls' 408 timeouts #1277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jamesbraza merged 1 commit into from
Jan 27, 2026
Merged

Fixing 60-sec wait for retrying nemotron-parse API calls' 408 timeouts #1277

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions packages/paper-qa-nemotron/src/paperqa_nemotron/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,14 +364,16 @@ async def _call_nvidia_api(
before_sleep=before_sleep_log(logger, logging.WARNING),
)
@retry(
retry=(
retry_if_exception_type(TimeoutError) # Hitting rate limits
| retry_if_exception(_is_litellm_timeout_with_408) # Inference timeout
),
retry=retry_if_exception_type(TimeoutError), # Hitting rate limits
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=2, min=GLOBAL_RATE_LIMITER_TIMEOUT),
before_sleep=before_sleep_log(logger, logging.WARNING),
)
@retry(
retry=retry_if_exception(_is_litellm_timeout_with_408), # Inference timeout
stop=stop_after_attempt(3),
before_sleep=before_sleep_log(logger, logging.WARNING),
)
Comment on lines 366 to +376
Copy link

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stacking multiple tenacity @retry decorators makes the retry semantics hard to reason about and can lead to more attempts and/or waits than intended (e.g., an exception that satisfies both retry predicates can be retried by the inner decorator and then again by the outer one, potentially re-introducing the long wait_exponential(..., min=GLOBAL_RATE_LIMITER_TIMEOUT) delay you’re trying to avoid for 408s). Consider collapsing this back to a single @retry with a combined retry= predicate and a wait= callable that chooses the backoff based on the caught exception (or otherwise ensure the TimeoutError retry explicitly excludes the 408 case).

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

@jamesbraza jamesbraza Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a bad idea, but YAGNI for now

async def _call_nvidia_api(
image: "np.ndarray",
tool_name: NemotronParseToolName,
Expand Down
Loading