Bug 1829923: etcd-quorum-guard: update containers section in deployment.yaml #1818

kikisdeliveryservice · 2020-06-11T18:22:21Z

Made the following changes as settings were not getting
picked up:

move failure threshold/timeout to nest under readinessProbe
move and correct initialDelaySeconds and periodSeconds
reorder container sub-elements to nest image under name

ironcladlou · 2020-06-11T18:22:59Z

/retitle Bug 1829923: etcd-quorum-guard: move failure threshold/timeout to under readiness

openshift-ci-robot · 2020-06-11T18:23:17Z

@kikisdeliveryservice: This pull request references Bugzilla bug 1829923, which is invalid:

expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1829923: etcd-quorum-guard: move failure threshold/timeout to under readiness

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-06-11T18:23:28Z

@kikisdeliveryservice: This pull request references Bugzilla bug 1829923, which is invalid:

expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1829923: etcd-quorum-guard: move failure threshold/timeout to under readiness

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ironcladlou · 2020-06-11T18:24:06Z

/bugzilla refresh

openshift-ci-robot · 2020-06-11T18:24:10Z

@ironcladlou: This pull request references Bugzilla bug 1829923, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.6.0) matches configured target release for branch (4.6.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kikisdeliveryservice · 2020-06-11T18:26:23Z

cc: @wking

openshift-ci-robot · 2020-06-11T18:33:03Z

@kikisdeliveryservice: This pull request references Bugzilla bug 1829923, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.6.0) matches configured target release for branch (4.6.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1829923: etcd-quorum-guard: move failure threshold/timeout to readinessProbe

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hexfusion · 2020-06-11T19:01:31Z

/test e2e-aws

kikisdeliveryservice · 2020-06-11T19:01:46Z

e2e-aws seems to have some issues..

level=error msg="When applying changes to module.vpc.aws_vpc.new_vpc[0], provider" level=error msg="\"registry.terraform.io/-/aws\" produced an unexpected new value for was" level=error msg="present, but now absent."

/test e2e-aws

hexfusion · 2020-06-11T19:27:04Z

LGTM in principle, let's confirm we see expected results.

kikisdeliveryservice · 2020-06-11T19:53:30Z

LGTM in principle, let's confirm we see expected results.

cool after this is confirmed we can cherrypick to 4.5

ironcladlou · 2020-06-11T20:06:33Z

https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/1818/pull-ci-openshift-machine-config-operator-master-e2e-gcp-upgrade/1271147848536166400/artifacts/e2e-gcp-upgrade/pods.json

"failureThreshold": 3,
"initialDelaySeconds": 5,
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 1

Hmmm...

kikisdeliveryservice · 2020-06-11T20:26:09Z

i think the containers section is off maybe?

kikisdeliveryservice · 2020-06-11T20:27:13Z

other platforms are doing it in this format:

machine-config-operator/manifests/vsphere/coredns.yaml

Line 51 in 6452d05

containers:

Made the following changes as settings were not getting picked up: - move failure threshold/timeout to nest under readinessProbe - move and correct initialDelaySeconds and periodSeconds - reorder container sub-elements to nest image under name

wking · 2020-06-11T23:17:25Z

/lgtm
/hold

Pull the hold when the results look good :)

openshift-ci-robot · 2020-06-11T23:19:34Z

@kikisdeliveryservice: This pull request references Bugzilla bug 1829923, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.6.0) matches configured target release for branch (4.6.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1829923: etcd-quorum-guard: move failure threshold/timeout to readinessProbe

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking · 2020-06-11T23:24:46Z

/lgtm

openshift-ci-robot · 2020-06-11T23:25:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kikisdeliveryservice, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kikisdeliveryservice]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-06-11T23:34:28Z

@kikisdeliveryservice: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-scaleup-rhel7	`9d62f94`	link	`/test e2e-aws-scaleup-rhel7`
ci/prow/e2e-metal-ipi	`9d62f94`	link	`/test e2e-metal-ipi`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

kikisdeliveryservice · 2020-06-12T02:08:23Z

reconfirming in this e2e-aws run:
Expect deployment values from PR ftr:
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
timeoutSeconds: 3

And also:
e2e-aws pods.json check:

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/1818/pull-ci-openshift-machine-config-operator-master-e2e-aws/1271220314893717504/artifacts/e2e-aws/gather-extra/pods.json | jq -r '.items[] | select(.metadata.name | contains("quorum-guard")).spec.containers[] | select(.name == "guard").readinessProbe | {initialDelaySeconds, periodSeconds, failureThreshold, timeoutSeconds} | tostring'
{"initialDelaySeconds":5,"periodSeconds":5,"failureThreshold":3,"timeoutSeconds":3}
{"initialDelaySeconds":5,"periodSeconds":5,"failureThreshold":3,"timeoutSeconds":3}
{"initialDelaySeconds":5,"periodSeconds":5,"failureThreshold":3,"timeoutSeconds":3}

e2e-gcp-op pods.json check:
$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/1818/pull-ci-openshift-machine-config-operator-master-e2e-gcp-op/1271220325798907904/artifacts/e2e-gcp-op/pods.json | jq -r '.items[] | select(.metadata.name | contains("quorum-guard")).spec.containers[] | select(.name == "guard").readinessProbe | {initialDelaySeconds, periodSeconds, failureThreshold, timeoutSeconds} | tostring'
{"initialDelaySeconds":5,"periodSeconds":5,"failureThreshold":3,"timeoutSeconds":3}
{"initialDelaySeconds":5,"periodSeconds":5,"failureThreshold":3,"timeoutSeconds":3}
{"initialDelaySeconds":5,"periodSeconds":5,"failureThreshold":3,"timeoutSeconds":3}

kikisdeliveryservice · 2020-06-12T02:09:15Z

rhel7 and metal-ipi seem to have RBAC issues

/skip

kikisdeliveryservice · 2020-06-12T02:14:54Z

Since I've confirmed this 2x am lifting hold to merge

/hold cancel

openshift-ci-robot · 2020-06-12T02:18:50Z

@kikisdeliveryservice: All pull requests linked via external trackers have merged: openshift/machine-config-operator#1818, openshift/machine-config-operator#1797. Bugzilla bug 1829923 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1829923: etcd-quorum-guard: update containers section in deployment.yaml

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kikisdeliveryservice · 2020-06-12T02:19:55Z

/cherry-pick release-4.5

openshift-cherrypick-robot · 2020-06-12T02:20:01Z

@kikisdeliveryservice: #1818 failed to apply on top of branch "release-4.5":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	install/0000_80_machine-config-operator_07_etcdquorumguard_deployment.yaml
Falling back to patching base and 3-way merge...
Auto-merging install/0000_80_machine-config-operator_07_etcdquorumguard_deployment.yaml
CONFLICT (content): Merge conflict in install/0000_80_machine-config-operator_07_etcdquorumguard_deployment.yaml
Patch failed at 0001 etcd-quorum-guard: update containers section in deployment.yaml

Details

In response to this:

/cherry-pick release-4.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kikisdeliveryservice · 2020-06-12T02:42:25Z

Backports for 4.4 & 4.5:
#1819
#1820

Also fixed the BZs to link correctly to the open PRs 😄

@ironcladlou @hexfusion PTAL at the above backports

ironcladlou · 2020-06-15T13:03:18Z

I don't see the right values in https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/1818/pull-ci-openshift-machine-config-operator-master-e2e-gcp-upgrade/1271220339220680704/artifacts/e2e-gcp-upgrade/pods.json

But I do see them in https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/1818/pull-ci-openshift-machine-config-operator-master-e2e-aws/1271220314893717504/artifacts/e2e-aws/gather-extra/pods.json

Which is somewhat concerning. I don't know how to explain the discrepancy yet.

I'm running an upgrade test against #1819 to exercise 4.4 -> 4.5 and see what happens.

wking · 2020-06-15T13:20:19Z

CVO code looks like it is missing some properties. Not sure yet if that is intentional or not, but seems like a CVO bug to me.

ironcladlou · 2020-06-15T13:52:30Z

@wking

CVO code looks like it is missing some properties. Not sure yet if that is intentional or not, but seems like a CVO bug to me.

Interesting... that could explain the upgrade results. Still waiting on my Y upgrade run to confirm the issue.

ironcladlou · 2020-06-15T14:32:05Z

Wanted to test a Y upgrade but cluster-bot didn't like that this PR was merged so I went with #1819:

test upgrade 4.4 openshift/machine-config-operator#1819

https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-launch-aws/1272513653010075648/artifacts/launch/pods.json

"failureThreshold": 3,
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 1

ironcladlou · 2020-06-15T14:59:19Z

openshift/cluster-version-operator#383

openshift-ci-robot requested review from cgwalters and yuqi-zhang June 11, 2020 18:22

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 11, 2020

kikisdeliveryservice changed the title ~~etcd-quorum-guard: move failure threshold/timeout to under readiness …~~ etcd-quorum-guard: move failure threshold/timeout to under readiness Jun 11, 2020

openshift-ci-robot changed the title ~~etcd-quorum-guard: move failure threshold/timeout to under readiness~~ Bug 1829923: etcd-quorum-guard: move failure threshold/timeout to under readiness Jun 11, 2020

openshift-ci-robot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jun 11, 2020

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jun 11, 2020

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 11, 2020

kikisdeliveryservice force-pushed the fix-quorum-guard branch from de94602 to 4d11568 Compare June 11, 2020 18:30

kikisdeliveryservice changed the title ~~Bug 1829923: etcd-quorum-guard: move failure threshold/timeout to under readiness~~ Bug 1829923: etcd-quorum-guard: move failure threshold/timeout to readinessProbe Jun 11, 2020

kikisdeliveryservice requested review from hexfusion, ironcladlou and wking and removed request for cgwalters and yuqi-zhang June 11, 2020 19:02

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 11, 2020

openshift-ci-robot assigned wking Jun 11, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 11, 2020

kikisdeliveryservice force-pushed the fix-quorum-guard branch from 2763493 to 9d62f94 Compare June 11, 2020 23:18

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jun 11, 2020

kikisdeliveryservice changed the title ~~Bug 1829923: etcd-quorum-guard: move failure threshold/timeout to readinessProbe~~ Bug 1829923: etcd-quorum-guard: update containers section in deployment.yaml Jun 11, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 11, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 12, 2020

openshift-merge-robot merged commit 22cc559 into openshift:master Jun 12, 2020

ironcladlou mentioned this pull request Jun 15, 2020

Bug 1847672: Expand supported set of probe field mutations openshift/cluster-version-operator#383

Merged

This was referenced Jun 17, 2020

[release-4.5] Bug 1845636: etcd-quorum-guard: update containers section in deployment.yaml #1819

Merged

[release-4.4] Bug 1845635: etcd-quorum-guard: update containers section in deployment.yaml #1820

Merged

Bug 1829923: etcd-quorum-guard: update containers section in deployment.yaml #1818

Bug 1829923: etcd-quorum-guard: update containers section in deployment.yaml #1818

Uh oh!

Conversation

kikisdeliveryservice commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ironcladlou commented Jun 11, 2020

Uh oh!

openshift-ci-robot commented Jun 11, 2020

Uh oh!

openshift-ci-robot commented Jun 11, 2020

Uh oh!

ironcladlou commented Jun 11, 2020

Uh oh!

openshift-ci-robot commented Jun 11, 2020

Uh oh!

kikisdeliveryservice commented Jun 11, 2020

Uh oh!

openshift-ci-robot commented Jun 11, 2020

Uh oh!

hexfusion commented Jun 11, 2020

Uh oh!

kikisdeliveryservice commented Jun 11, 2020

Uh oh!

hexfusion commented Jun 11, 2020

Uh oh!

kikisdeliveryservice commented Jun 11, 2020

Uh oh!

ironcladlou commented Jun 11, 2020

Uh oh!

kikisdeliveryservice commented Jun 11, 2020

Uh oh!

kikisdeliveryservice commented Jun 11, 2020

Uh oh!

wking commented Jun 11, 2020

Uh oh!

openshift-ci-robot commented Jun 11, 2020

Uh oh!

wking commented Jun 11, 2020

Uh oh!

openshift-ci-robot commented Jun 11, 2020

Uh oh!

openshift-ci-robot commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kikisdeliveryservice commented Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kikisdeliveryservice commented Jun 12, 2020

Uh oh!

kikisdeliveryservice commented Jun 12, 2020

Uh oh!

openshift-ci-robot commented Jun 12, 2020

Uh oh!

kikisdeliveryservice commented Jun 12, 2020

Uh oh!

openshift-cherrypick-robot commented Jun 12, 2020

Uh oh!

kikisdeliveryservice commented Jun 12, 2020

Uh oh!

ironcladlou commented Jun 15, 2020

Uh oh!

wking commented Jun 15, 2020

Uh oh!

ironcladlou commented Jun 15, 2020

Uh oh!

ironcladlou commented Jun 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ironcladlou commented Jun 15, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kikisdeliveryservice commented Jun 11, 2020 •

edited

Loading

openshift-ci-robot commented Jun 11, 2020 •

edited

Loading

kikisdeliveryservice commented Jun 12, 2020 •

edited

Loading

ironcladlou commented Jun 15, 2020 •

edited

Loading