WIP: openshift-tests: run tests using ginkgo for common setup

[everettraven]

Steps to run dummy example:

export OPENSHIFT_TESTS_SKIP_STABLE_CLUSTER=true
export OPENSHIFT_SKIP_EXTERNAL_TESTS=true
export KUBECONFIG={pathToKubeConfig}

./openshift-tests run openshift/dummy-suite --monitor=watch-namespaces --run-multiple

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: everettraven
Once this PR has been reviewed and has the lgtm label, please assign neisw for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[stbenjam]

This is an interesting experiment; openshift-tests use of ginkgo has always been weird because of the per-process isolation. I would be curious if openshift/conformance behaves well. IIRC, a number of tests don't do proper clean up ballooning memory usage and even sometimes touch global state that affects other tests.

[stbenjam]

There are some alternatives to BeforeAll hooks called for in the OTE enhancement, specifically configs: https://github.com/openshift/enhancements/blob/master/enhancements/testing/openshift-tests-extension.md

I'm not sure how the run-multiple will behave with some planned features like intelligent scheduling with exclusions, memory/cpu requirements, etc.

[everettraven]

I'm not sure how the run-multiple will behave with some planned features like intelligent scheduling with exclusions, memory/cpu requirements, etc

I'd be curious to learn more about what all these planned features entail. I've not read through the OTE enhancement in its entirety yet, so if it is explained in more detail there I'll likely see that soon :).

This is an interesting experiment; openshift-tests use of ginkgo has always been weird because of the per-process isolation. I would be curious if openshift/conformance behaves well. IIRC, a number of tests don't do proper clean up ballooning memory usage and even sometimes touch global state that affects other tests.

The ballooning memory usage is something that seems a little more challenging to prevent, and makes sense as a reason for process isolation, but I wonder if that is more of a signal that we should be enforcing implementations of tests to clean up after themselves appropriately?

In theory, the approach I've taken here could still run batches of tests in multiple processes if we wanted to run more tests at a time in an individual process. Ginkgo also has parallelization logic (see https://onsi.github.io/ginkgo/#spec-parallelization)

For tests that don't play nice with other tests, Ginkgo has the notion of spec decorators like Serial (https://onsi.github.io/ginkgo/#serial-specs) that signals to Ginkgo that those particular specs should only be run in series and cannot run with any parallel tests.

I suspect running openshift/conformance as configured today against this change would probably fail, but with the right tweaks might work. I can certainly explore this a bit more in the future.

[openshift-merge-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[JoelSpeed]

To be removed

[JoelSpeed]

What's the benefit?

[everettraven]

Ginkgo already has native logic for handling parallel and serial tests. See https://onsi.github.io/ginkgo/#serial-specs

[openshift-ci]

@everettraven: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-csi	3ad15cc	link	true	/test e2e-gcp-csi
ci/prow/e2e-aws-csi	3ad15cc	link	true	/test e2e-aws-csi
ci/prow/go-verify-deps	3ad15cc	link	true	/test go-verify-deps
ci/prow/e2e-aws-ovn-microshift	3ad15cc	link	true	/test e2e-aws-ovn-microshift
ci/prow/e2e-aws-ovn-microshift-serial	3ad15cc	link	true	/test e2e-aws-ovn-microshift-serial
ci/prow/e2e-metal-ipi-ovn-ipv6	3ad15cc	link	true	/test e2e-metal-ipi-ovn-ipv6

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.