Bug 1891143: pkg/cvo/metrics: Abandon child goroutines after shutdownContext expires #477
Conversation
Apparently there's something in the HTTPS server goroutine that can hang up even if we've called Shutdown() on the server [1]. Defend against that with a safety valve to abandon stuck goroutines if shutdownContext expires. Also pivot to resultChannel and asyncResult, so we can get names for the collected channels (and more easily identify the stuck channels by elimination), following the pattern set by cc1921d (pkg/start: Release leader lease on graceful shutdown, 2020-08-03, openshift#424). [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1891143#c1
openshift-ci-robot
added
the
bugzilla/severity-unspecified
|
@wking: This pull request references Bugzilla bug 1891143, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
the
bugzilla/invalid-bug
openshift-ci-robot
added
the
approved
|
/bugzilla refresh |
openshift-ci-robot
removed
the
bugzilla/severity-unspecified
|
@wking: This pull request references Bugzilla bug 1891143, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
the
bugzilla/severity-high
|
/bugzilla refresh |
|
@wking: This pull request references Bugzilla bug 1891143, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
bugzilla/valid-bug
|
/assign @jottofar |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jottofar, wking The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/test unit |
|
John pruned some leaked GCP service accounts. /retest |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
2 similar comments
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
We have doubled our GCP service account quota. /retest |
|
@wking: All pull requests linked via external trackers have merged: Bugzilla bug 1891143 has been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
wking
deleted the
abandon-child-goroutines-after-metrics-shutdown-context
branch
|
/test e2e-metal-assisted |
|
Bug VERIFIED in 4.7. /cherrypick release-4.6 |
|
@wking: new pull request created: #480 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
7 participants
Apparently there's something in the HTTPS server goroutine that can hang up even if we've called
Shutdown()on the server. Defend against that with a safety valve to abandon stuck goroutines ifshutdownContextexpires.Also pivot to
resultChannelandasyncResult, so we can get names for the collected channels (and more easily identify the stuck channels by elimination), following the pattern set by cc1921d (#424).