Bug 1771810: Add e2e to verify kso metrics accessible #191

sallyom · 2019-12-02T01:39:23Z

example #2 for using library-go/test/metrics functions. This PR adds an e2e to verify cluster-kube-scheduler-operator metrics are accessible.

see openshift/library-go#602

openshift-ci-robot · 2019-12-02T01:39:26Z

@sallyom: This pull request references Bugzilla bug 1771810, which is invalid:

expected the bug to target the "4.4.0" release, but it targets "4.3.0" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1771810: Add e2e to verify kso metrics accessible

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ingvagabund · 2019-12-02T09:31:12Z

This PR adds an e2e to verify cluster-kube-scheduler-operator metrics are accessible.

Is it required to access the metrics through kubectl? Or, is the goal of the PR to verify (using any method) the metrics are accessible? If so, would it be easier to access the metrics from the metrics endpoint? (I am just asking, I have never consumed the metrics other than directly from a component while running in the same binary).

sallyom · 2019-12-02T15:28:16Z

This PR adds an e2e to verify cluster-kube-scheduler-operator metrics are accessible.

Is it required to access the metrics through kubectl? Or, is the goal of the PR to verify (using any method) the metrics are accessible? If so, would it be easier to access the metrics from the metrics endpoint? (I am just asking, I have never consumed the metrics other than directly from a component while running in the same binary).

The goal is to access metrics and also verify cluster CA -this will eventually be used to verify successful cert rotation- what I landed upon (after days of trial and error) is to access metrics via internal service url with curl cmd secured via cluster CA (generated by service-ca-operator/annotated configmap). I also had a version using the external route, and go code instead of curl - but I could not utilize service CA - only way that worked was to set InsecureSkipVerify in TLSconfig - but exec-ing in a pod using internal url works perfectly. I'm open to using anything that works, though!!

sallyom · 2019-12-02T19:50:10Z

/test e2e-aws-operator

ingvagabund · 2019-12-03T11:19:23Z

I'm open to using anything that works, though!!

I have zero experience in this area. Though, as long as it's used in tests, I have no objection. Are you aware of other components/operator that test the same? Or, are you pioneering it right now? :)

s-urbaniak · 2019-12-03T12:59:34Z

@sallyom @ingvagabund hello from the monitoring team! :-) 👋 @paulfantom asked me to help out here, so don't hesitate also to ping me on #forum-monitoring in slack.

I do recommend that you access your metrics via the exposed Thanos routes (which internally forward to Prometheus). That way you will verify that your metrics are being successfully scraped and hence also verifies the RBAC setup towards your scraping targets. I would not advise to check metrics in your local metrics endpoint only.

Regarding querying metrics you could do the approach done here or create your own http client (as we did) after finding out the external route and token: https://github.com/openshift/cluster-monitoring-operator/blob/8e5623597b94d2f62c38bc427c23a820a01f002f/test/e2e/framework/route_client.go#L47-L76

sallyom · 2019-12-03T13:46:42Z

Regarding querying metrics you could do the approach done here or create your own http client (as we did) after finding out the external route and token: https://github.com/openshift/cluster-monitoring-operator/blob/8e5623597b94d2f62c38bc427c23a820a01f002f/test/e2e/framework/route_client.go#L47-L76

thanks @s-urbaniak yes, I was under impression that a TLSconfig secured via CAbundle was required, but as it turns out it is not - so, I am going w/ the external prom url /route client - i had that working a few days ago but only w/ InsecureSkipVerify - and I thought we needed to avoid that but alas after much discussion we do not - so I'm back to using the route (very similar to the route_client setup you pointed to). That will live in library-go/test/library/metrics/query.go.

s-urbaniak · 2019-12-03T14:47:10Z

@sallyom sounds good, yes we are also "InsecureSkipVerify'ing" the TLS connection via the exposed routes right now.

sallyom · 2020-01-07T18:56:24Z

/bugzilla refresh

openshift-ci-robot · 2020-01-07T18:56:30Z

@sallyom: This pull request references Bugzilla bug 1771810, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sallyom · 2020-01-14T14:19:28Z

/retest

soltysh

/lgtm
/approve
/hold
for nit resolution

soltysh · 2020-01-16T12:11:22Z

Makefile

 $(call add-bindata,v4.1.0,./bindata/v4.1.0/...,bindata,v410_00_assets,pkg/operator/v410_00_assets/bindata.go)

 e2e: GO_TEST_PACKAGES :=./test/e2e
+e2e: GO_TEST_FLAGS += -v


I don't think we need that verbosity, if the test fails, we'll get it, otherwise it's not needed.

soltysh · 2020-01-16T12:13:04Z

test/e2e/scheduler_test.go

+	}
+
+	// do something with result, eventually this will be tested b4 and after rotation to ensure values differ
+	fmt.Printf("result from prometheus query `scheduler_scheduling_duration_seconds_sum`: %v", response)


Maybe change this to t.Logf

sallyom · 2020-01-29T21:56:56Z

/retest

soltysh

/lgtm
/hold cancel

openshift-ci-robot · 2020-01-31T08:41:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sallyom, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-01-31T10:43:44Z

@sallyom: All pull requests linked via external trackers have merged. Bugzilla bug 1771810 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1771810: Add e2e to verify kso metrics accessible

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Dec 2, 2019

openshift-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Dec 2, 2019

openshift-ci-robot requested review from ingvagabund and soltysh December 2, 2019 01:39

sallyom force-pushed the bz1771810 branch from a8f892e to 6e2cb70 Compare December 2, 2019 17:00

sallyom force-pushed the bz1771810 branch 2 times, most recently from bf8effb to fe1b5e7 Compare December 3, 2019 01:15

sallyom mentioned this pull request Dec 3, 2019

add helper functions for running prometheus query openshift/library-go#602

Merged

sallyom force-pushed the bz1771810 branch from fe1b5e7 to 158e46f Compare December 10, 2019 20:56

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 19, 2019

sallyom force-pushed the bz1771810 branch from 158e46f to fbaad90 Compare December 24, 2019 13:34

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 24, 2019

sallyom force-pushed the bz1771810 branch 2 times, most recently from 2c95a69 to b7e0a6a Compare January 6, 2020 23:28

openshift-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 6, 2020

sallyom force-pushed the bz1771810 branch from b7e0a6a to 9ab1517 Compare January 7, 2020 15:23

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 7, 2020

sallyom force-pushed the bz1771810 branch from 9ab1517 to 1ea94db Compare January 7, 2020 15:32

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 7, 2020

sallyom force-pushed the bz1771810 branch 2 times, most recently from 089811d to aaa6d16 Compare January 7, 2020 16:51

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jan 7, 2020

sallyom force-pushed the bz1771810 branch 2 times, most recently from 9a76910 to 02f6c18 Compare January 13, 2020 19:52

soltysh approved these changes Jan 16, 2020

View reviewed changes

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 16, 2020

openshift-ci-robot assigned soltysh Jan 16, 2020

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 16, 2020

soltysh reviewed Jan 16, 2020

View reviewed changes

sallyom force-pushed the bz1771810 branch from 02f6c18 to 62d3875 Compare January 20, 2020 22:40

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jan 20, 2020

sallyom force-pushed the bz1771810 branch from 62d3875 to a0e2342 Compare January 22, 2020 02:25

sallyom added 2 commits January 28, 2020 14:22

add e2e to check that metrics is accessible

ef77c9b

bump(*)

5d98642

sallyom force-pushed the bz1771810 branch from a0e2342 to 5d98642 Compare January 28, 2020 19:26

soltysh approved these changes Jan 31, 2020

View reviewed changes

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jan 31, 2020

openshift-merge-robot merged commit 25e7a32 into openshift:master Jan 31, 2020

damemi mentioned this pull request Jan 13, 2021

Fix or disable TestMetricsAccessible e2e #322

Closed

Bug 1771810: Add e2e to verify kso metrics accessible #191

Bug 1771810: Add e2e to verify kso metrics accessible #191

Uh oh!

Conversation

sallyom commented Dec 2, 2019

Uh oh!

openshift-ci-robot commented Dec 2, 2019

Uh oh!

ingvagabund commented Dec 2, 2019

Uh oh!

sallyom commented Dec 2, 2019

Uh oh!

sallyom commented Dec 2, 2019

Uh oh!

ingvagabund commented Dec 3, 2019

Uh oh!

s-urbaniak commented Dec 3, 2019

Uh oh!

sallyom commented Dec 3, 2019

Uh oh!

s-urbaniak commented Dec 3, 2019

Uh oh!

sallyom commented Jan 7, 2020

Uh oh!

openshift-ci-robot commented Jan 7, 2020

Uh oh!

sallyom commented Jan 14, 2020

Uh oh!

soltysh left a comment

Choose a reason for hiding this comment

Uh oh!

soltysh Jan 16, 2020

Choose a reason for hiding this comment

Uh oh!

sallyom Jan 20, 2020

Choose a reason for hiding this comment

Uh oh!

soltysh Jan 16, 2020

Choose a reason for hiding this comment

Uh oh!

sallyom Jan 20, 2020

Choose a reason for hiding this comment

Uh oh!

sallyom commented Jan 29, 2020

Uh oh!

soltysh left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Jan 31, 2020

Uh oh!

openshift-ci-robot commented Jan 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants