Add wrapper which will allow running o/k tests as external binary in origin #1485

soltysh · 2023-02-22T17:43:38Z

This goes in pair with openshift/origin#27570

Write-up in openshift/enhancements#1291

openshift-ci-robot · 2023-02-22T17:43:48Z

@soltysh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

eb6684c|Create minimal wrapper needed to run k8s e2e tests: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

soltysh · 2023-02-22T17:44:06Z

/hold

openshift-ci-robot · 2023-02-23T17:14:47Z

@soltysh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

f8c6207|Create minimal wrapper needed to run k8s e2e tests: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

openshift-ci-robot · 2023-03-10T16:38:18Z

@soltysh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

4104a0c|Create minimal wrapper needed to run k8s e2e tests: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

openshift-ci-robot · 2023-03-10T16:57:19Z

@soltysh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

1fe6754|Update generated annotations: does not specify an upstream backport in the commit message
ee5ff54|Create minimal wrapper needed to run k8s e2e tests: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

openshift-ci-robot · 2023-03-21T16:34:50Z

@soltysh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

1d23c19|Update generated: does not specify an upstream backport in the commit message
7bd1168|Change annotation mechanics to allow injecting testMaps and filter out tests: does not specify an upstream backport in the commit message
d580c16|Create minimal wrapper needed to run k8s e2e tests: does not specify an upstream backport in the commit message
ee5aaf6|Move k8s-specific rules to our fork: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

openshift-ci-robot · 2023-03-31T15:47:57Z

@soltysh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

0988c0a|UPSTREAM: : Change annotation mechanics to allow injecting testMaps and filter out tests: does not specify an upstream backport in the commit message
0def31b|UPSTREAM: : Update generated: does not specify an upstream backport in the commit message
287f8e7|UPSTREAM: : Move k8s-specific rules to our fork: does not specify an upstream backport in the commit message
54f6da7|UPSTREAM: : Create minimal wrapper needed to run k8s e2e tests: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

soltysh · 2023-03-31T15:48:46Z

/hold cancel

soltysh · 2023-03-31T15:50:52Z

/remove-label backports/unvalidated-commits
/label backports/validated-commits

openshift-ci-robot · 2023-04-03T13:19:28Z

@soltysh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

54f6da7|UPSTREAM: : Create minimal wrapper needed to run k8s e2e tests: does not specify an upstream backport in the commit message
7301d93|UPSTREAM: : Update generated: does not specify an upstream backport in the commit message
d3fba9a|UPSTREAM: : Change annotation mechanics to allow injecting testMaps and filter out tests: does not specify an upstream backport in the commit message
d85b97f|UPSTREAM: : Move k8s-specific rules to our fork: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

soltysh · 2023-04-05T14:07:08Z

/retest

ingvagabund · 2023-04-06T09:25:18Z

openshift-hack/cmd/k8s-tests/provider.go

Which functions/commit are you referring to?

$ grep -rn "initializeTestFramework" cmd/openshift-tests/e2e.go:529: return initializeTestFramework(exutil.TestContext, opt.config, opt.DryRun) cmd/openshift-tests/upgrade.go:85: if err := initializeTestFramework(exutil.TestContext, config, testOpt.DryRun); err != nil { cmd/openshift-tests/openshift-tests.go:487: if err := initializeTestFramework(exutil.TestContext, config, testOpt.DryRun); err != nil { cmd/openshift-tests/provider.go:36:func initializeTestFramework(context *e2e.TestContextType, config *exutilcluster.ClusterConfiguration, dryRun bool) error { $ git log -1 commit bfcc30fde82a91fab1c881a836e4ad8e63c72fb5 (HEAD -> master, origin/release-4.15, origin/release-4.14, origin/master, origin/HEAD) Merge: e63252bcd5 69f130654a Author: OpenShift Merge Robot <[email protected]> Date: Thu Apr 6 03:59:44 2023 -0400 Merge pull request #27842 from ardaguclu/oc-debug-pod-name-generation not rely on deterministic pod names in oc debug tests

This was about these 2 files:

https://github.com/openshift/origin/blob/master/cmd/openshift-tests/provider.go)

https://github.com/openshift/origin/blob/master/test/extended/util/test.go

Fixed the comment.

ingvagabund · 2023-04-06T09:29:54Z

openshift-hack/cmd/k8s-tests/provider.go

What's the purpose of unmarshaling into &providerInfo? I.e. reference to providerInfo instead of just providerInfo?

providerInfo in origin is populated here: https://github.com/openshift/origin/blob/bfcc30fde82a91fab1c881a836e4ad8e63c72fb5/cmd/openshift-tests/provider.go#L78, so it's done on every test invocation.

tl;dr

When you invoke openshift-tests run openshift/conformance/parallel it creates a list of tests and runs one by one invoking openshift-tests run-test. The above decodeProvider is responsible to read what the cluster looks like during that run-test invocation. Since we don't want to rely on that mechanism in here, we need to inject that value through env variable, and then decode it and use it.

I'm updating openshift/enhancements#1291 I'll add more details how this works there.

ingvagabund · 2023-04-06T09:33:51Z

openshift-hack/cmd/k8s-tests/provider.go

What if initializeTestFramework gets invoked multiple times?. Invoking testfiles.AddFileSource does not implement any test for duplicates.

k8s-tests run-test is invoked once per tests, see my explanation above, k8s-tests run-test is identical to openshift-tests run-test it only run one test only.

I was thinking of a case when in the future initializeTestFramework gets accidentally invoked twice.

ingvagabund · 2023-04-06T09:53:53Z

openshift-hack/cmd/k8s-tests/runtest.go

What if there are multiple reports with report.NumAttempts > 0?

That's a copy of the mechanism as was in origin, but that's a good question, I'll make sure to look more into it and fix it.

ingvagabund · 2023-04-06T11:01:46Z

openshift-hack/e2e/annotate/annotate.go

Worth defining a function that will construct a name from the test name and the labels. In case the same naming constructions is (or will be) used some place else.

Probably, but origin will directly use the annotation mechanism, without modifying it.

ingvagabund · 2023-04-06T11:02:26Z

openshift-hack/e2e/annotate/annotate.go

The same here. Worth introducing a named function which will combine the test name and the labels into a string.

Same as above.

ingvagabund · 2023-04-06T12:05:44Z

openshift-hack/e2e/annotate/rules.go

These are new tests? Or, the tests are expected to be prefixed with [sig-...] from now on to be disabled? Or, these will appear after the 1.27 rebase lands?

Those tests and rules are copied from origin, see Move k8s-specific rules to our fork commit in origin PR, previously we had a mix of rules in here and in origin. This PR splits the rules so they live where the test is.

ingvagabund · 2023-04-06T12:10:54Z

/lgtm
The PR introduces a new command which is better to test up-front rather than provide fully reviewed code. Some of the code is copy-pasted from origin. Changes under openshift-hack/e2e/annotate looks ok.
/hold
Feel free to unhold. I dropped few comments. Mostly for me to better understand the flow.

openshift-ci · 2023-04-06T12:11:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ingvagabund, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~DOWNSTREAM_OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…tMaps and filter out tests

openshift-ci · 2023-04-06T12:47:48Z

New changes are detected. LGTM label has been removed.

openshift-ci-robot · 2023-04-06T12:47:57Z

@soltysh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

1399bb6|UPSTREAM: : Move k8s-specific rules to our fork: does not specify an upstream backport in the commit message
6040b96|UPSTREAM: : Change annotation mechanics to allow injecting testMaps and filter out tests: does not specify an upstream backport in the commit message
e3df01c|UPSTREAM: : Create minimal wrapper needed to run k8s e2e tests: does not specify an upstream backport in the commit message
f7b7abc|UPSTREAM: : Update generated: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

soltysh · 2023-04-06T12:48:49Z

/hold cancel

openshift-ci-robot · 2023-04-06T15:41:50Z

/retest-required

Remaining retests: 0 against base HEAD ff8ecbc and 2 for PR HEAD f7b7abc in total

soltysh · 2023-04-06T18:00:51Z

/test e2e-aws-ovn-crun

soltysh · 2023-04-07T13:40:34Z

/test e2e-aws-ovn-serial

soltysh · 2023-04-08T17:48:19Z

/test e2e-azure-ovn-upgrade

soltysh · 2023-04-10T10:15:56Z

/test e2e-azure-ovn-upgrade

openshift-ci · 2023-04-10T12:26:15Z

@soltysh: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agnostic-ovn-cmd	`f7b7abc`	link	false	`/test e2e-agnostic-ovn-cmd`
ci/prow/4.11-upgrade-from-stable-4.10-e2e-aws-ovn-upgrade	`f7b7abc`	link	false	`/test 4.11-upgrade-from-stable-4.10-e2e-aws-ovn-upgrade`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

soltysh · 2023-04-10T17:57:50Z

/test e2e-azure-ovn-upgrade

openshift-ci-robot added the backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. label Feb 22, 2023

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 22, 2023

soltysh closed this Feb 22, 2023

soltysh reopened this Feb 22, 2023

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 22, 2023

openshift-ci bot requested review from deads2k and sttts February 22, 2023 17:46

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 22, 2023

soltysh mentioned this pull request Feb 22, 2023

Running tests using external binary openshift/origin#27570

Merged

2 tasks

soltysh force-pushed the test_poc branch from eb6684c to f8c6207 Compare February 23, 2023 17:14

soltysh force-pushed the test_poc branch from f8c6207 to 4104a0c Compare March 10, 2023 16:38

soltysh force-pushed the test_poc branch from 4104a0c to 1fe6754 Compare March 10, 2023 16:57

soltysh force-pushed the test_poc branch from 1fe6754 to 1d23c19 Compare March 21, 2023 16:34

soltysh force-pushed the test_poc branch from 1d23c19 to 0def31b Compare March 31, 2023 15:47

soltysh changed the title ~~[WIP] POC for running external tests~~ Add wrapper which will allow running o/k tests as external binary in origin Mar 31, 2023

openshift-ci bot removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Mar 31, 2023

openshift-ci bot added backports/validated-commits Indicates that all commits come to merged upstream PRs. and removed backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. labels Mar 31, 2023

soltysh force-pushed the test_poc branch from 0def31b to 7301d93 Compare April 3, 2023 13:19

openshift-ci-robot added the backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. label Apr 3, 2023

ingvagabund reviewed Apr 6, 2023

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 6, 2023

openshift-ci bot assigned ingvagabund Apr 6, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 6, 2023

soltysh added 4 commits April 6, 2023 14:47

UPSTREAM: <carry>: Create minimal wrapper needed to run k8s e2e tests

e3df01c

UPSTREAM: <carry>: Change annotation mechanics to allow injecting tes…

6040b96

…tMaps and filter out tests

UPSTREAM: <carry>: Move k8s-specific rules to our fork

1399bb6

UPSTREAM: <drop>: Update generated

f7b7abc

soltysh force-pushed the test_poc branch from 7301d93 to f7b7abc Compare April 6, 2023 12:47

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Apr 6, 2023

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 6, 2023

soltysh added lgtm Indicates that a PR is ready to be merged. backports/validated-commits Indicates that all commits come to merged upstream PRs. and removed backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. labels Apr 6, 2023

openshift-merge-robot merged commit bda0cb5 into openshift:master Apr 10, 2023

soltysh deleted the test_poc branch April 11, 2023 08:18

Add wrapper which will allow running o/k tests as external binary in origin #1485

Add wrapper which will allow running o/k tests as external binary in origin #1485

Uh oh!

Conversation

soltysh commented Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Feb 22, 2023

Uh oh!

soltysh commented Feb 22, 2023

Uh oh!

openshift-ci-robot commented Feb 23, 2023

Uh oh!

openshift-ci-robot commented Mar 10, 2023

Uh oh!

openshift-ci-robot commented Mar 10, 2023

Uh oh!

openshift-ci-robot commented Mar 21, 2023

Uh oh!

openshift-ci-robot commented Mar 31, 2023

Uh oh!

soltysh commented Mar 31, 2023

Uh oh!

soltysh commented Mar 31, 2023

Uh oh!

openshift-ci-robot commented Apr 3, 2023

Uh oh!

soltysh commented Apr 5, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soltysh Apr 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ingvagabund commented Apr 6, 2023

Uh oh!

openshift-ci bot commented Apr 6, 2023

Uh oh!

openshift-ci bot commented Apr 6, 2023

Uh oh!

openshift-ci-robot commented Apr 6, 2023

Uh oh!

soltysh commented Apr 6, 2023

Uh oh!

openshift-ci-robot commented Apr 6, 2023

Uh oh!

soltysh commented Apr 6, 2023

Uh oh!

soltysh commented Apr 7, 2023

Uh oh!

soltysh commented Apr 8, 2023

soltysh commented Feb 22, 2023 •

edited

Loading

soltysh Apr 6, 2023 •

edited

Loading

openshift-ci bot commented Apr 10, 2023 •

edited

Loading