Skip to content

pkg/cvo/sync_worker: TaskNodeComplete events during updates #435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wking wants to merge 2 commits into from
Closed

pkg/cvo/sync_worker: TaskNodeComplete events during updates #435

wking wants to merge 2 commits into from

Conversation

@wking
Copy link
Member

@wking wking commented Aug 13, 2020

Only during updates, because:

  • Install-time is a free-for-all, where the CVO doesn't block on anything. This would be a lot of "node complete" noise about nodes where we had only attempted to push manifests, and that's unlikely to be what event-readers expect TaskNodeComplete to imply.

  • Reconcile-time hopefully has very few instances where the CVO needs to stomp on changes, block on a recently Available=False operator, etc. Eventing on each completed TaskNode would be lots of noise without much interesting signal.

During updates, we have the structured graph and blocking TaskNodes described here, and the flow through that graph is what the events from this commit will help shed light on. You could also achieve this by preserving logs from the CVO pods as they are repositioned throughout an update, but we don't have tooling in CI to do that conveniently today.

The hardcoded name and namespace for cvoObjectRef isn't great (for example, it won't work in pkg/start/start_integration_test.go , where the ClusterVersion's name and namespace are random). But it's the pattern we've used since we started eventing in 475e71f (#411), so I'm recylcing it for now.

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 13, 2020
@deads2k
Copy link
Contributor

deads2k commented Aug 13, 2020

A later refinement to find a clusteroperator (if one exists) and put it in the message and as a relatedObject would help readability in the future. This is much better than where we are today.

/lgtm
/hold

holding in case you want someone else to review.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 13, 2020
@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 13, 2020
@wking wking force-pushed the manifest-block-node-completion-events-on-update branch from 698a759 to f66fc41 Compare August 13, 2020 19:06
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Aug 13, 2020
@wking wking force-pushed the manifest-block-node-completion-events-on-update branch from f66fc41 to 04e0add Compare August 13, 2020 19:17
@jottofar
Copy link
Contributor

jottofar commented Aug 13, 2020

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 13, 2020
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Aug 13, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, jottofar, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@deads2k
Copy link
Contributor

deads2k commented Aug 13, 2020

looks like you found someone.

/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 13, 2020
@wking
Copy link
Member Author

wking commented Aug 13, 2020

/hold

We want to confirm that we like the results before this lands ;)

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 13, 2020
@deads2k
Copy link
Contributor

deads2k commented Aug 17, 2020

/retest

@wking
Copy link
Member Author

wking commented Aug 17, 2020

Peeking into the running job, I see:

$ oc -n openshift-cluster-version logs cluster-version-operator-7b87bb4cf-d84dj
...
I0817 20:53:06.402574       1 task_graph.go:486] Running 1 on worker 15
E0817 20:53:06.402661       1 runtime.go:78] Observed a panic: runtime.boundsError{x:-1, y:0, signed:true, code:0x0} (runtime error: index out of range [-1])
goroutine 272 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x185a8c0, 0xc000ee70e0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x185a8c0, 0xc000ee70e0)
	/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).apply.func2(0x1c0cf20, 0xc000d85380, 0x0, 0x0, 0x0, 0x2, 0x2)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:709 +0x777
github.com/openshift/cluster-version-operator/pkg/payload.RunGraph.func3(0xc00191a700, 0xc0003a5440, 0xc00011f050, 0xc0003a55c0, 0x1c0cf20, 0xc000d85380, 0xf)
	/go/src/github.com/openshift/cluster-version-operator/pkg/payload/task_graph.go:487 +0xea
created by github.com/openshift/cluster-version-operator/pkg/payload.RunGraph
	/go/src/github.com/openshift/cluster-version-operator/pkg/payload/task_graph.go:477 +0x32a
panic: runtime error: index out of range [-1] [recovered]
	panic: runtime error: index out of range [-1]

goroutine 272 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x185a8c0, 0xc000ee70e0)
	/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).apply.func2(0x1c0cf20, 0xc000d85380, 0x0, 0x0, 0x0, 0x2, 0x2)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:709 +0x777
github.com/openshift/cluster-version-operator/pkg/payload.RunGraph.func3(0xc00191a700, 0xc0003a5440, 0xc00011f050, 0xc0003a55c0, 0x1c0cf20, 0xc000d85380, 0xf)
	/go/src/github.com/openshift/cluster-version-operator/pkg/payload/task_graph.go:487 +0xea
created by github.com/openshift/cluster-version-operator/pkg/payload.RunGraph
	/go/src/github.com/openshift/cluster-version-operator/pkg/payload/task_graph.go:477 +0x32a

So we do need to work out clearer naming for the nodes with no tasks. Or not event on those taskless nodes completing. Or something.

@deads2k
Copy link
Contributor

deads2k commented Aug 19, 2020

So we do need to work out clearer naming for the nodes with no tasks. Or not event on those taskless nodes completing. Or something.

not eventing on taskless nodes seems reasonable.

@wking wking force-pushed the manifest-block-node-completion-events-on-update branch from 04e0add to 4a88428 Compare August 19, 2020 20:20
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Aug 19, 2020
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Aug 19, 2020

New changes are detected. LGTM label has been removed.

@wking
Copy link
Member Author

wking commented Aug 19, 2020

I have local work in progress to name task nodes, but yeah, skipping events for nodes with no tasks is nice and easy, so pivoted to that with 04e0add -> 4a88428.

@wking
Copy link
Member Author

wking commented Aug 19, 2020

update:

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/435/pull-ci-openshift-cluster-version-operator-master-e2e-upgrade/1296180435792957440/artifacts/e2e-upgrade/events.json | jq -r '.items[] | select(.metadata.namespace == "openshift-cluster-version" and .reason == "TaskNodeComplete") | .timePrefix = if .firstTimestamp == null or .firstTimestamp == "null" then .eventTime else .firstTimestamp + " - " + .lastTimestamp + " (" + (.count | tostring) + ")" end | .timePrefix + " " + .message'
2020-08-19T21:15:01Z - 2020-08-19T21:15:01Z (1) synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:15:01Z - 2020-08-19T21:15:01Z (1) synchronized task node ending in customresourcedefinition "securitycontextconstraints.security.openshift.io" (15 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:15:01Z - 2020-08-19T21:15:01Z (1) synchronized task node ending in customresourcedefinition "rolebindingrestrictions.authorization.openshift.io" (11 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:15:01Z - 2020-08-19T21:15:01Z (1) synchronized task node ending in customresourcedefinition "clusterresourcequotas.quota.openshift.io" (14 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:15:01Z - 2020-08-19T21:15:01Z (1) synchronized task node ending in customresourcedefinition "proxies.config.openshift.io" (13 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:21:08Z - 2020-08-19T21:44:17Z (2) synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:21:08Z - 2020-08-19T21:27:42Z (2) synchronized task node ending in customresourcedefinition "clusterresourcequotas.quota.openshift.io" (14 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:21:08Z - 2020-08-19T21:21:08Z (1) synchronized task node ending in customresourcedefinition "rolebindingrestrictions.authorization.openshift.io" (11 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:21:08Z - 2020-08-19T21:27:42Z (2) synchronized task node ending in customresourcedefinition "securitycontextconstraints.security.openshift.io" (15 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:21:08Z - 2020-08-19T21:35:07Z (35) (combined from similar events): synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:27:48Z - 2020-08-19T21:27:48Z (1) synchronized task node ending in clusteroperator "kube-scheduler" (111 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:28Z - 2020-08-19T21:49:28Z (1) synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:28Z - 2020-08-19T21:49:28Z (1) synchronized task node ending in customresourcedefinition "rolebindingrestrictions.authorization.openshift.io" (11 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:28Z - 2020-08-19T21:49:28Z (1) synchronized task node ending in customresourcedefinition "clusterresourcequotas.quota.openshift.io" (14 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:28Z - 2020-08-19T21:49:28Z (1) synchronized task node ending in customresourcedefinition "securitycontextconstraints.security.openshift.io" (15 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:28Z - 2020-08-19T21:49:28Z (1) synchronized task node ending in customresourcedefinition "proxies.config.openshift.io" (13 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:32Z - 2020-08-19T21:49:32Z (1) synchronized task node ending in clusteroperator "kube-apiserver" (93 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:33Z - 2020-08-19T21:49:33Z (1) synchronized task node ending in clusteroperator "kube-controller-manager" (102 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:34Z - 2020-08-19T21:49:34Z (1) synchronized task node ending in clusteroperator "kube-scheduler" (111 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:35Z - 2020-08-19T21:49:35Z (1) synchronized task node ending in clusteroperator "config-operator" (121 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"
2020-08-19T21:49:37Z - 2020-08-19T21:49:37Z (1) (combined from similar events): synchronized task node ending in clusteroperator "machine-api" (159 of 606) on update to version="4.6.0-0.ci.test-2020-08-19-202618-ci-op-gqk5jv4d" image="registry.svc.ci.openshift.org/ci-op-gqk5jv4d/release@sha256:bc5535d6c2028074451c23201a184d3ba4a195a52f703a9ea3e276d064669f4a"

Not super-chatty (good?), but also looks a bit sparse compared to the number of nodes I expected...

@wking
Copy link
Member Author

wking commented Aug 21, 2020

I've pushed 3462a13 with some hack attempts to disable event aggregation, so we can get a fuller picture.

@wking wking force-pushed the manifest-block-node-completion-events-on-update branch from 3462a13 to a5e8827 Compare August 21, 2020 04:15
@wking
Copy link
Member Author

wking commented Aug 25, 2020

update:

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/435/pull-ci-openshift-cluster-version-operator-master-e2e-upgrade/1296662283056844800/artifacts/e2e-upgrade/events.json | jq -r '.items[] | select(.metadata.namespace == "openshift-cluster-version" and .reason == "TaskNodeComplete") | .timePrefix = if .firstTimestamp == null or .firstTimestamp == "null" then .eventTime else .firstTimestamp + " - " + .lastTimestamp + " (" + (.count | tostring) + ")" end | .timePrefix + " " + .message'
2020-08-21T04:55:35Z - 2020-08-21T04:55:35Z (1) synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T04:55:35Z - 2020-08-21T04:55:35Z (1) synchronized task node ending in customresourcedefinition "clusterresourcequotas.quota.openshift.io" (14 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T04:55:35Z - 2020-08-21T04:55:35Z (1) synchronized task node ending in customresourcedefinition "securitycontextconstraints.security.openshift.io" (15 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T04:55:35Z - 2020-08-21T04:55:35Z (1) synchronized task node ending in customresourcedefinition "rolebindingrestrictions.authorization.openshift.io" (11 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T04:55:35Z - 2020-08-21T04:55:35Z (1) synchronized task node ending in customresourcedefinition "proxies.config.openshift.io" (13 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:01:43Z - 2020-08-21T05:15:28Z (3) synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:01:43Z - 2020-08-21T05:15:28Z (3) synchronized task node ending in customresourcedefinition "rolebindingrestrictions.authorization.openshift.io" (11 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:01:43Z - 2020-08-21T05:15:28Z (3) synchronized task node ending in customresourcedefinition "clusterresourcequotas.quota.openshift.io" (14 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:01:43Z - 2020-08-21T05:15:28Z (3) synchronized task node ending in customresourcedefinition "securitycontextconstraints.security.openshift.io" (15 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:01:43Z - 2020-08-21T05:15:28Z (3) synchronized task node ending in customresourcedefinition "proxies.config.openshift.io" (13 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:01:47Z - 2020-08-21T05:15:37Z (3) synchronized task node ending in clusteroperator "kube-apiserver" (93 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:03:55Z - 2020-08-21T05:15:38Z (3) synchronized task node ending in clusteroperator "kube-controller-manager" (102 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:04:39Z - 2020-08-21T05:15:38Z (3) synchronized task node ending in clusteroperator "kube-scheduler" (111 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:05:00Z - 2020-08-21T05:15:39Z (3) synchronized task node ending in clusteroperator "config-operator" (121 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:06:16Z - 2020-08-21T05:15:39Z (3) synchronized task node ending in clusteroperator "openshift-apiserver" (169 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:08:22Z - 2020-08-21T05:15:41Z (2) synchronized task node ending in clusteroperator "machine-api" (159 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:09:41Z - 2020-08-21T05:15:42Z (2) synchronized task node ending in clusteroperator "kube-storage-version-migrator" (180 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:09:42Z - 2020-08-21T05:15:45Z (2) synchronized task node ending in serviceaccount "openshift-service-catalog-removed/openshift-service-catalog-apiserver-remover" (349 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:09:46Z - 2020-08-21T05:15:45Z (2) synchronized task node ending in serviceaccount "openshift-service-catalog-removed/openshift-service-catalog-controller-manager-remover" (352 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:09:58Z - 2020-08-21T05:15:49Z (2) synchronized task node ending in clusteroperator "storage" (346 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:04Z - 2020-08-21T05:15:56Z (2) synchronized task node ending in clusteroperator "csi-snapshot-controller" (247 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:10Z - 2020-08-21T05:15:52Z (2) synchronized task node ending in clusteroperator "authentication" (207 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:15Z - 2020-08-21T05:15:52Z (2) synchronized task node ending in clusteroperator "node-tuning" (301 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:17Z - 2020-08-21T05:15:50Z (2) synchronized task node ending in clusteroperator "machine-approver" (290 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:24Z - 2020-08-21T05:15:59Z (2) synchronized task node ending in servicemonitor "openshift-insights/insights-operator" (425 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:27Z - 2020-08-21T05:15:56Z (2) synchronized task node ending in clusteroperator "ingress" (280 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:34Z - 2020-08-21T05:15:59Z (2) synchronized task node ending in clusteroperator "operator-lifecycle-manager-packageserver" (448 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:36Z - 2020-08-21T05:16:00Z (2) synchronized task node ending in consolelink "openshift-learning-portal" (403 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:46Z - 2020-08-21T05:16:00Z (2) synchronized task node ending in rolebinding "openshift-cluster-samples-operator/prometheus-k8s" (338 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:47Z - 2020-08-21T05:15:51Z (2) synchronized task node ending in clusteroperator "openshift-controller-manager" (311 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:48Z - 2020-08-21T05:15:58Z (2) synchronized task node ending in prometheusrule "openshift-image-registry/image-registry-operator-alerts" (265 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:10:52Z - 2020-08-21T05:15:56Z (2) synchronized task node ending in credentialsrequest "openshift-cloud-credential-operator/cloud-credential-operator-s3" (196 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:11:14Z - 2020-08-21T05:15:53Z (2) synchronized task node ending in clusteroperator "monitoring" (366 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:11:22Z - 2020-08-21T05:15:57Z (2) synchronized task node ending in role "openshift-marketplace/openshift-marketplace-metrics" (462 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:11:43Z - 2020-08-21T05:15:56Z (2) synchronized task node ending in clusteroperator "service-ca" (471 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:12:58Z - 2020-08-21T05:15:58Z (2) synchronized task node ending in clusterrole "cluster-autoscaler-operator:cluster-reader" (229 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:11Z - 2020-08-21T05:24:11Z (1) synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:11Z - 2020-08-21T05:24:11Z (1) synchronized task node ending in customresourcedefinition "securitycontextconstraints.security.openshift.io" (15 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:11Z - 2020-08-21T05:24:11Z (1) synchronized task node ending in customresourcedefinition "clusterresourcequotas.quota.openshift.io" (14 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:11Z - 2020-08-21T05:24:11Z (1) synchronized task node ending in customresourcedefinition "rolebindingrestrictions.authorization.openshift.io" (11 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:11Z - 2020-08-21T05:24:11Z (1) synchronized task node ending in customresourcedefinition "proxies.config.openshift.io" (13 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:16Z - 2020-08-21T05:24:16Z (1) synchronized task node ending in clusteroperator "kube-apiserver" (93 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:17Z - 2020-08-21T05:24:17Z (1) synchronized task node ending in clusteroperator "kube-scheduler" (111 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:17Z - 2020-08-21T05:24:17Z (1) synchronized task node ending in clusteroperator "kube-controller-manager" (102 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:18Z - 2020-08-21T05:24:18Z (1) synchronized task node ending in clusteroperator "openshift-apiserver" (169 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:19Z - 2020-08-21T05:24:19Z (1) synchronized task node ending in clusteroperator "config-operator" (121 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:20Z - 2020-08-21T05:24:20Z (1) synchronized task node ending in clusteroperator "machine-api" (159 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:21Z - 2020-08-21T05:24:21Z (1) synchronized task node ending in clusteroperator "kube-storage-version-migrator" (180 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:23Z - 2020-08-21T05:24:23Z (1) synchronized task node ending in serviceaccount "openshift-service-catalog-removed/openshift-service-catalog-apiserver-remover" (349 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:24Z - 2020-08-21T05:24:24Z (1) synchronized task node ending in serviceaccount "openshift-service-catalog-removed/openshift-service-catalog-controller-manager-remover" (352 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:28Z - 2020-08-21T05:24:28Z (1) synchronized task node ending in clusteroperator "storage" (346 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:29Z - 2020-08-21T05:24:29Z (1) synchronized task node ending in clusteroperator "machine-approver" (290 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:30Z - 2020-08-21T05:24:30Z (1) synchronized task node ending in clusteroperator "openshift-controller-manager" (311 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:31Z - 2020-08-21T05:24:31Z (1) synchronized task node ending in clusteroperator "node-tuning" (301 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:32Z - 2020-08-21T05:24:32Z (1) synchronized task node ending in clusteroperator "authentication" (207 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:32Z - 2020-08-21T05:24:32Z (1) synchronized task node ending in clusteroperator "monitoring" (366 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:35Z - 2020-08-21T05:24:35Z (1) synchronized task node ending in clusteroperator "ingress" (280 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:35Z - 2020-08-21T05:24:35Z (1) synchronized task node ending in credentialsrequest "openshift-cloud-credential-operator/cloud-credential-operator-s3" (196 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:35Z - 2020-08-21T05:24:35Z (1) synchronized task node ending in clusteroperator "csi-snapshot-controller" (247 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:35Z - 2020-08-21T05:24:35Z (1) synchronized task node ending in clusteroperator "service-ca" (471 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:36Z - 2020-08-21T05:24:36Z (1) synchronized task node ending in role "openshift-marketplace/openshift-marketplace-metrics" (462 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:37Z - 2020-08-21T05:24:37Z (1) synchronized task node ending in clusterrole "cluster-autoscaler-operator:cluster-reader" (229 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:37Z - 2020-08-21T05:24:37Z (1) synchronized task node ending in prometheusrule "openshift-image-registry/image-registry-operator-alerts" (265 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:38Z - 2020-08-21T05:24:38Z (1) synchronized task node ending in clusteroperator "operator-lifecycle-manager-packageserver" (448 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:38Z - 2020-08-21T05:24:38Z (1) synchronized task node ending in servicemonitor "openshift-insights/insights-operator" (425 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:39Z - 2020-08-21T05:24:39Z (1) synchronized task node ending in rolebinding "openshift-cluster-samples-operator/prometheus-k8s" (338 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:39Z - 2020-08-21T05:24:39Z (1) synchronized task node ending in consolelink "openshift-learning-portal" (403 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:23Z - 2020-08-21T05:27:23Z (1) synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:23Z - 2020-08-21T05:27:23Z (1) synchronized task node ending in customresourcedefinition "securitycontextconstraints.security.openshift.io" (15 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:23Z - 2020-08-21T05:27:23Z (1) synchronized task node ending in customresourcedefinition "rolebindingrestrictions.authorization.openshift.io" (11 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:23Z - 2020-08-21T05:27:23Z (1) synchronized task node ending in customresourcedefinition "clusterresourcequotas.quota.openshift.io" (14 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:23Z - 2020-08-21T05:27:23Z (1) synchronized task node ending in customresourcedefinition "proxies.config.openshift.io" (13 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:28Z - 2020-08-21T05:27:28Z (1) synchronized task node ending in clusteroperator "kube-apiserver" (93 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:29Z - 2020-08-21T05:27:29Z (1) synchronized task node ending in clusteroperator "kube-controller-manager" (102 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:29Z - 2020-08-21T05:27:29Z (1) synchronized task node ending in clusteroperator "kube-scheduler" (111 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:31Z - 2020-08-21T05:27:31Z (1) synchronized task node ending in clusteroperator "config-operator" (121 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:27:32Z - 2020-08-21T05:27:32Z (1) synchronized task node ending in clusteroperator "machine-api" (159 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"

@wking wking force-pushed the manifest-block-node-completion-events-on-update branch from a5e8827 to ec7298e Compare August 25, 2020 20:28
@wking wking force-pushed the manifest-block-node-completion-events-on-update branch from ec7298e to afa6dc9 Compare August 25, 2020 20:33
@wking
Copy link
Member Author

wking commented Aug 25, 2020

Trying to understand the version="" bits from my previous comment:

...events with version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5"...
2020-08-21T04:55:35Z - 2020-08-21T04:55:35Z (1) synchronized task node ending in customresourcedefinition "proxies.config.openshift.io" (13 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:01:43Z - 2020-08-21T05:15:28Z (3) synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
...events with version=""...
2020-08-21T05:12:58Z - 2020-08-21T05:15:58Z (2) synchronized task node ending in clusterrole "cluster-autoscaler-operator:cluster-reader" (229 of 606) on update to version="" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
2020-08-21T05:24:11Z - 2020-08-21T05:24:11Z (1) synchronized task node ending in service "openshift-cluster-version/cluster-version-operator" (10 of 606) on update to version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5" image="registry.svc.ci.openshift.org/ci-op-km0flzx5/release@sha256:c71058dae2d6014a7d0150a7385bdb07cc415125000bda98a82089e184f34200"
...events with version="4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5"...

History of that update job (which included an A->B->A rollback):

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/435/pull-ci-openshift-cluster-version-operator-master-e2e-upgrade/1296662283056844800/artifacts/e2e-upgrade/clusterversion.json | jq -r '.items[].status.history[] | .startedTime + " " + .completionTime + " " + .version + " " + .state + " " + (.verified | tostring)'
2020-08-21T05:32:44Z 2020-08-21T06:08:58Z 4.6.0-0.ci.test-2020-08-21-041717-ci-op-km0flzx5 Completed false
2020-08-21T04:55:30Z 2020-08-21T05:32:44Z 4.6.0-0.ci.test-2020-08-21-041733-ci-op-km0flzx5 Partial false
2020-08-21T04:21:58Z 2020-08-21T04:54:03Z 4.6.0-0.ci.test-2020-08-21-041717-ci-op-km0flzx5 Completed false

Possibly we need backfilling like this for these events as well...

@wking
Copy link
Member Author

wking commented Aug 25, 2020

/retest

@wking
Copy link
Member Author

wking commented Aug 26, 2020

Hmm, Generation doesn't seem to be the sync cycle counter after all. Ah well. Still, the failure events are useful:

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/435/pull-ci-openshift-cluster-version-operator-master-e2e-upgrade/1298394890048114688/artifacts/e2e-upgrade/events.json | jq -r '.items[] | select(.metadata.namespace == "openshift-cluster-version" and .reason == "ApplyFailed") | .timePrefix = if .firstTimestamp == null or .firstTimestamp == "null" then .eventTime else .firstTimestamp + " - " + .lastTimestamp + " (" + (.count | tostring) + ")" end | .timePrefix + " " + .message'
2020-08-25T23:52:14Z - 2020-08-25T23:52:14Z (1) failed to apply update to version="4.6.0-0.ci.test-2020-08-25-230422-ci-op-5idnhksv" image="registry.svc.ci.openshift.org/ci-op-5idnhksv/release@sha256:1988f16452c02c968b2ec212e1c3f924ed15d67f05540e5897e4f3c7e1e3ce89" generation=2: Cluster operator kube-apiserver is still updating
2020-08-25T23:58:19Z - 2020-08-25T23:58:19Z (1) failed to apply update to version="" image="registry.svc.ci.openshift.org/ci-op-5idnhksv/release@sha256:1988f16452c02c968b2ec212e1c3f924ed15d67f05540e5897e4f3c7e1e3ce89" generation=2: Some cluster operators are still updating: machine-api, openshift-apiserver
2020-08-26T00:04:46Z - 2020-08-26T00:04:46Z (1) failed to apply update to version="" image="registry.svc.ci.openshift.org/ci-op-5idnhksv/release@sha256:1988f16452c02c968b2ec212e1c3f924ed15d67f05540e5897e4f3c7e1e3ce89" generation=2: Cluster operator network is still updating
2020-08-26T00:12:10Z - 2020-08-26T00:20:49Z (2) failed to apply update to version="" image="registry.svc.ci.openshift.org/ci-op-5idnhksv/release@sha256:1988f16452c02c968b2ec212e1c3f924ed15d67f05540e5897e4f3c7e1e3ce89" generation=2: Cluster operator machine-config is still updating
2020-08-26T00:23:34Z - 2020-08-26T00:23:34Z (1) failed to apply update to version="4.6.0-0.ci.test-2020-08-25-230422-ci-op-5idnhksv" image="registry.svc.ci.openshift.org/ci-op-5idnhksv/release@sha256:1988f16452c02c968b2ec212e1c3f924ed15d67f05540e5897e4f3c7e1e3ce89" generation=2: Cluster operator openshift-apiserver is reporting a failure: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver

wking added 2 commits September 2, 2020 14:19
@wking
pkg/cvo/sync_worker: TaskNodeComplete events during updates
bad839c 
Only during updates, because:

* Install-time is a free-for-all, where the CVO doesn't block on
  anything.  This would be a lot of "node complete" noise about nodes
  where we had only attempted to push manifests, and that's unlikely
  to be what event-readers expect TaskNodeComplete to imply.

* Reconcile-time hopefully has very few instances where the CVO needs
  to stomp on changes, block on a recently Available=False operator,
  etc.  Eventing on each completed TaskNode would be lots of noise
  without much interesting signal.

During updates, we have the structured graph and blocking TaskNodes
described in docs/user/reconciliation.md, and the flow through that
graph is what the events from this commit will help shed light on.
You could also achieve this by preserving logs from the CVO pods as
they are repositioned throughout an update, but we don't have tooling
in CI to do that conveniently today.

The hardcoded name and namespace for cvoObjectRef isn't great (for
example, it won't work in pkg/start/start_integration_test.go , where
the ClusterVersion's name and namespace are random).  But it's the
pattern we've used since we started eventing in 475e71f (emit
events for each new payload, 2020-07-21, openshift#411), so I'm recylcing it
for now.

Also log ApplyFailed events when we fail in apply, to remove some of
the guesswork in determining what manifest(s) had trouble.
@wking
WIP: Block event aggregation
c9705ab
@wking wking force-pushed the manifest-block-node-completion-events-on-update branch from afa6dc9 to c9705ab Compare September 2, 2020 21:20
@wking
Copy link
Member Author

wking commented Sep 2, 2020

Pivoted from generation= to attempt= and rebased onto master with afa6dc9 -> c9705ab.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Sep 2, 2020

@wking: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-upgrade c9705ab link /test e2e-upgrade
ci/prow/e2e c9705ab link /test e2e

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

wking added a commit to wking/cluster-version-operator that referenced this pull request Dec 1, 2020
@wking
pkg/payload/task_graph: Require firstIncompleteNode to have tasks
6b1470b 
Our graph has some nodes without tasks, e.g. nodes which create a
choke point between two sets of parallel nodes.  Ideally we give all
nodes descriptive names [1], but until we have that, require at least
one entry in Tasks before we store a node as firstIncompleteNode.
This avoids the chance of panicking during the subsequent [2]:

  fmt.Errorf("%d incomplete task nodes, beginning with %s", incompleteCount, firstIncompleteNode.Tasks[0])

[1]: openshift#435
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1903382
@wking wking mentioned this pull request Dec 1, 2020
Merged
@openshift-merge-robot
Copy link
Contributor

openshift-merge-robot commented Dec 9, 2020

@wking: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-agnostic-upgrade c9705ab link /test e2e-agnostic-upgrade
ci/prow/e2e-agnostic c9705ab link /test e2e-agnostic
ci/prow/e2e-agnostic-operator c9705ab link /test e2e-agnostic-operator

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/cluster-version-operator that referenced this pull request Feb 2, 2021
@wking
pkg/payload/task_graph: Require firstIncompleteNode to have tasks
8254fc1 
Our graph has some nodes without tasks, e.g. nodes which create a
choke point between two sets of parallel nodes.  Ideally we give all
nodes descriptive names [1], but until we have that, require at least
one entry in Tasks before we store a node as firstIncompleteNode.
This avoids the chance of panicking during the subsequent [2]:

  fmt.Errorf("%d incomplete task nodes, beginning with %s", incompleteCount, firstIncompleteNode.Tasks[0])

[1]: openshift#435
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1903382
@openshift-bot
Copy link
Contributor

openshift-bot commented Mar 10, 2021

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci-robot openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 10, 2021
@openshift-bot
Copy link
Contributor

openshift-bot commented Apr 9, 2021

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci-robot openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 9, 2021
@openshift-bot
Copy link
Contributor

openshift-bot commented May 9, 2021

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

@openshift-ci openshift-ci bot closed this May 9, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 9, 2021

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jottofar jottofar Awaiting requested review from jottofar

@sdodson sdodson Awaiting requested review from sdodson

Assignees

@deads2k deads2k

@jottofar jottofar

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@wking @deads2k @jottofar @openshift-ci-robot @openshift-merge-robot @openshift-bot