OCPBUGS-62227: bump telemetry series limit to 1000 #30302
Conversation
| averageSeriesLimit = 850 | ||
| default: | ||
| averageSeriesLimit = 780 | ||
| averageSeriesLimit = 1000 |
There was a problem hiding this comment.
let's revert 52c9e9a
and have back a unique 1000 limit for avg_over_time for all clusters
cc @stbenjam, was the change in 52c9e9a made to allow more fine-grained (potentially lower) limits for managed clusters in the future?
We’re suggesting setting the limit to 1000 across all clusters. This would act as a good safeguard in case the average bursts a lot all at once (currently +200 series).
Incrementally raising the limit isn’t sustainable for us (just did that 3 months ago #29975 (comment)). While we can point out which metrics started emitting more series, we can’t really judge whether that’s acceptable, so we always just end up raising the limits.
We can also debate the usefulness of the test itself, but that’s a separate discussion.
|
/retest-required |
|
Job Failure Risk Analysis for sha: f993b85
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: f993b85
New tests seen in this PR at sha: f993b85
|
|
e2e-aws-ovn-microshift-serial failed since the case is not related to telemetry series limit, I think we could skip the job |
|
thanks! |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: juzhao, machine424 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/verified by @juzhao |
openshift-ci-robot
added
the
verified
|
@juzhao: This PR has been marked as verified by In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/retitle OCPBUGS-62227: bump telemetry series limit to 1000 |
openshift-ci
bot
changed the title
openshift-ci-robot
added
jira/severity-moderate
|
@juzhao: This pull request references Jira Issue OCPBUGS-62227, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
openshift-ci-robot
added
the
jira/valid-bug
|
@juzhao: This pull request references Jira Issue OCPBUGS-62227, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
removed
the
jira/invalid-bug
|
@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: juzhao. Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/skip |
|
/override ci/prow/e2e-aws-ovn-microshift-serial |
|
@juzhao: juzhao unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:openshift: openshift-release-oversight openshift-staff-engineers openshift-sustaining-engineers. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test e2e-aws-ovn-microshift-serial |
|
/skip |
|
/retest-required |
1 similar comment
|
/retest-required |
|
@juzhao: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Job Failure Risk Analysis for sha: 7c50620
|
openshift-merge-bot
bot
merged commit 44be851
into
openshift:main
|
@juzhao: Jira Issue Verification Checks: Jira Issue OCPBUGS-62227 Jira Issue OCPBUGS-62227 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Fix included in accepted release 4.21.0-0.nightly-2025-09-27-154726 |
5 participants
see from OCPBUGS-62227, case "Alerts shouldn't exceed the series limit of total series sent via telemetry from each cluster" failed on e2e-aws-ovn-techpreview job
this PR bumped the limit to 1000 to tolerate more series added to telemetry in the future