-
Notifications
You must be signed in to change notification settings - Fork 7
feat: implement retry mechanism for log processing #136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens when the log never succeeds? Maybe because an agent token is no longer valid. Judging by the code, logstream will attempt to send it to coder Forever
l.logger.Error(ctx, "max retries exceeded", | ||
slog.F("retryCount", rs.retryCount), | ||
slog.F("maxRetries", l.maxRetries)) | ||
rs.exhausted = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will be kept in memory forever now right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will be kept in memory until the pod is deleted.
…efault of 15 retries for log send failures.
Closes #119
This PR adds error handling and retry functionality to the log streaming by adding exponential backoff retry logic for failed log operations.
Changes
PostLogSource
andAgent dRPC client
failures to handle connection issuesWith this PR, users will see more reliable logs from pod startup in their Coder workspace, including pod events before container startup that were previously not visible.