Skip to content

About

A Kubernetes Operator to facilitate the setup and management of Cryostat.

Topics

Resources

Stars

37 stars

Watchers

6 watching

Forks

Repository files navigation

cryostat-operator

CI build Test CI push Google Group : Cryostat Development

A Kubernetes Operator to automate deployment of Cryostat.

SEE ALSO

  • cryostat-core for the core library providing a convenience wrapper and headless stubs for use of JFR using JDK Mission Control internals.

  • cryostat for the main API backend to detect JVMs and manage JFR.

  • cryostat-web for the React graphical frontend included as a submodule in Cryostat and built into Cryostat's OCI images.

  • jfr-datasource for the JFR datasource for Grafana.

  • cryostat-grafana-dashboard for the Grafana dashboard.

USING

Requirements

Instructions

Once deployed, the cryostat instance can be accessed via web browser at the URL provided by:

kubectl get cryostat -o jsonpath='{$.items[0].status.applicationUrl}'

To use Cryostat to monitor or profile Cryostat itself - since it is also an available JVM target - you may use the Cryostat web UI to define a Custom Target with the connection URL localhost:0. This is a special value which tells Cryostat's JVM that it should connect to itself directly, without the need to expose a JMX port over the network.

INSTALLATION

OperatorHub

The operator's primary installation method is via OperatorHub. A more detailed installation guide is available here. Installation of the Operator without OLM/OperatorHub is intended for development purposes and is not a supported release configuration.

Bundle Deployment

The operator can be deployed using OLM using make deploy_bundle. This will deploy quay.io/cryostat/cryostat-operator-bundle:$IMAGE_VERSION to your configured cluster using oc or kubectl (kubeconfig). You can set the variables IMAGE_NAMESPACE or IMAGE_VERSION to deploy different builds of the bundle. Once this is complete, the Cryostat Operator will be deployed and running in your cluster.

Manual Deployment

make install will create CustomResourceDefinitions and do other setup required to prepare the cluster for deploying the operator, using oc or kubectl on whichever OpenShift/Kubernetes cluster is configured with the local client. make uninstall destroys the CRDs and undoes the setup.

make run can be used to run the operator controller manager as a process on your local development machine and observing/interacting with your cluster. This may be useful in some development scenarios, however in this case the operator process will not have access to certain in-cluster resources such as environment variables or service account token files.

make deploy will deploy the operator using static manifests to an arbitrary namespace (default to cryostat-operator-system).

  • make DEPLOY_NAMESPACE=foo-namespace deploy can be used to deploy to a namespace named foo-namespace. For a convenient shorthand, use make DEPLOY_NAMESPACE=$(kubectl config view --minify -o 'jsonpath={.contexts[0].context.namespace}') deploy to deploy to the currently active OpenShift/Kubernetes namespace.
  • make undeploy will likewise remove the operator, and also uses the DEPLOY_NAMESPACE variable. This also respects the IMAGE_TAG environment variable, so that different versions of the operator can be easily deployed.
  • To obtain such static manifests remotely without further customizations, use:
    # Replace ref with any version tag or branch
    kubectl kustomize "https://github.com/cryostatio/cryostat-operator.git/config/default?ref=v2.4.0"

Configuration

Once deployed, the operator deployment will be active in the cluster, but no Cryostat instance will be created. To trigger its creation, add a Cryostat CR using the UI for operator "provided APIs". Full details on the configuration options in the Cryostat CRD can be found in Configuring Cryostat. When running on Kubernetes, see Network Options for additional mandatory configuration in order to access Cryostat outside of the cluster.

For convenience, a full deployment can be created using kubectl create -f config/samples/operator_v1beta2_cryostat.yaml, or more simply, make create_cryostat_cr.

The container images used by the operator for the core application, jfr-datasource, and the Grafana dashboard can be overridden by setting the RELATED_IMAGE_CORE, RELATED_IMAGE_DATASOURCE, and RELATED_IMAGE_GRAFANA environment variables, respectively, in the operator deployment.

SECURITY

By default, the operator expects cert-manager to be available in the cluster. This allows the operator to deploy Cryostat with all communication between its internal services done over HTTPS. If you wish to disable this feature and not use cert-manager, you can set the environment variable DISABLE_SERVICE_TLS=true when you deploy the operator. We provide make cert_manager and make remove_cert_manager targets to easily install/remove cert-manager from your cluster.

User Authentication

Users can use oc whoami --show-token to retrieve their OpenShift OAuth token for the currently logged in user account. This token can be used when directly interacting with the deployed Cryostat instance(s).

When using the web-client, users can login with their username and password associated with their OpenShift account. User credentials will be remembered for the duration of the session.

If the current user account does not have sufficient permissions to list routes, list endpointslices, or perform other actions that Cryostat requires, then the user may also try to authenticate using the Operator's service account. This, of course, assumes that the user has permission to view this service account's secrets.

oc get secrets | grep cryostat-operator-service-account-token will provide at least one such operator service account token secret name which can be used - for example, cryostat-operator-service-account-token-7tt7l. The token can then be retrieved for use in authenticating as the operator service account:

$ oc describe secret cryostat-operator-service-account-token-7tt7l
Name:         cryostat-operator-service-account-token-7tt7l
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/created-by: openshift.io/create-dockercfg-secrets
              kubernetes.io/service-account.name: cryostat-operator-service-account
              kubernetes.io/service-account.uid: 692aa8c7-081e-4a51-9355-be3eaa8f9fa6

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:          7209 bytes
namespace:       7 bytes
service-ca.crt:  8422 bytes
token:           eyJhbGciOiJSUzI1NiIsImtpZCI6IkhYZC13eDdGVGwyQzdGNVpZVndScEZ2VmRxWTlzbnBUUG9HRkJpejJkV3cifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNyeW9zdGF0LW9wZXJhdG9yLXNlcnZpY2UtYWNjb3VudC10b2tlbi03dHQ3bCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjcnlvc3RhdC1vcGVyYXRvci1zZXJ2aWNlLWFjY291bnQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2OTJhYThjNy0wODFlLTRhNTEtOTM1NS1iZTNlYWE4ZjlmYTYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpjcnlvc3RhdC1vcGVyYXRvci1zZXJ2aWNlLWFjY291bnQifQ.M7C1V0bN3aILBflO7TTOTikw7wLGRJ79-OkCDQIZbu71QLdX05jyCxxtlH32lr8jz6HwxfXXweh3ifG_2lbe7_TbM8jxmBoMdLuc4Q_akpmA-GQuDPrRxfHGJApYGQ6CVug3KHSrQwj2M4QrSUz7FoeQGaOH9BnWj1TrHGmOZUPJ6u7JSu2OwoLBda6rF-M4Bl72DmkyMAzikreRgPEk4D7gTCY0yNvsQDuUAwpFwmEukRC2WyTAVTpKPgThZUk-UJ-dXufbhAcqIRt6jeCQ19_Bo0zXc_ELgQydxuTack1ndT3HwRmwwNuZDFv-G3Y0YdjfRh00DqEvSn9ynZzwueDCJUxlHdznytfUWk9PA712JENpFC7b-zSHnjymIcFeUd8s_Zq_-JKrDIPnH0oZDRO_MUpKEC7Jz_8SeFJHLLGfBZt_aP4VwQHEUThiFQPwrfbd8tppUG2TKcekPScKcauy-BCI52odBzapP6meilMQVrmRtu7i30L05vgQiST_OsmSP8CuKW13a-leCCtN_aNQGqlWvLhP81H95ui-PvMzwMIDlfDZ03ycuYg4R4eUG3nUq7-42wrSdFLo8gm9wsl7y1ZRMQwHR1DCVBbHYS0iFOcmwto2Ejlrgvn3Cs0pDS7pDVoFkH2FsTopEw3jXtnkMs15mSmBnHz-UjF-l08

or more briefly:

$ oc get -o jsonpath='{.data.token}' secret cryostat-operator-service-account-token-7tt7l | base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6IkhYZC13eDdGVGwyQzdGNVpZVndScEZ2VmRxWTlzbnBUUG9HRkJpejJkV3cifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNyeW9zdGF0LW9wZXJhdG9yLXNlcnZpY2UtYWNjb3VudC10b2tlbi03dHQ3bCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjcnlvc3RhdC1vcGVyYXRvci1zZXJ2aWNlLWFjY291bnQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2OTJhYThjNy0wODFlLTRhNTEtOTM1NS1iZTNlYWE4ZjlmYTYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpjcnlvc3RhdC1vcGVyYXRvci1zZXJ2aWNlLWFjY291bnQifQ.M7C1V0bN3aILBflO7TTOTikw7wLGRJ79-OkCDQIZbu71QLdX05jyCxxtlH32lr8jz6HwxfXXweh3ifG_2lbe7_TbM8jxmBoMdLuc4Q_akpmA-GQuDPrRxfHGJApYGQ6CVug3KHSrQwj2M4QrSUz7FoeQGaOH9BnWj1TrHGmOZUPJ6u7JSu2OwoLBda6rF-M4Bl72DmkyMAzikreRgPEk4D7gTCY0yNvsQDuUAwpFwmEukRC2WyTAVTpKPgThZUk-UJ-dXufbhAcqIRt6jeCQ19_Bo0zXc_ELgQydxuTack1ndT3HwRmwwNuZDFv-G3Y0YdjfRh00DqEvSn9ynZzwueDCJUxlHdznytfUWk9PA712JENpFC7b-zSHnjymIcFeUd8s_Zq_-JKrDIPnH0oZDRO_MUpKEC7Jz_8SeFJHLLGfBZt_aP4VwQHEUThiFQPwrfbd8tppUG2TKcekPScKcauy-BCI52odBzapP6meilMQVrmRtu7i30L05vgQiST_OsmSP8CuKW13a-leCCtN_aNQGqlWvLhP81H95ui-PvMzwMIDlfDZ03ycuYg4R4eUG3nUq7-42wrSdFLo8gm9wsl7y1ZRMQwHR1DCVBbHYS0iFOcmwto2Ejlrgvn3Cs0pDS7pDVoFkH2FsTopEw3jXtnkMs15mSmBnHz-UjF-l08

BUILDING

Requirements

  • go (see go.mod version)
  • operator-sdk v1.42.0
  • podman or docker
  • jq v1.6+
  • yq v4.35+
  • ginkgo (Optional)

Instructions

make generate manifests manager will trigger code/YAML generation and compile the operator controller manager, along with running some code quality checks.

make oci-build will build an OCI image from the generated YAML and compiled binary to the local registry, tagged as quay.io/crystatio/cryostat-operator. This tag can be overridden by setting the environment variables IMAGE_NAMESPACE and OPERATOR_NAME. IMAGE_VERSION can also be set to override the tagged version.

make bundle will create an OLM bundle. This will generate a CSV, CRDs and other manifests, and other required configurations for an OLM bundle versioned with version $IMAGE_VERSION in the bundle/ directory. make bundle-build will create an OCI image of this bundle, which can then be pushed to an image repository such as quay.io.

make catalog-build will build an OCI image of the operator catalog (i.e. index) with version $IMAGE_VERSION that includes the bundle image of the same version.

DEVELOPMENT

An invocation like export IMAGE_NAMESPACE=quay.io/some-user export IMAGE_VERSION=test-version make generate manifests manager oci-build bundle bundle-build podman image prune -f && podman push $IMAGE_NAMESPACE/cryostat-operator:$IMAGE_VERSION && podman push $IMAGE_NAMESPACE/cryostat-operator-bundle:$IMAGE_VERSION make deploy_bundle is handy for local development testing using ex. CodeReady Containers. This exercises a similar build and deployment path as what end users using OLM and OperatorHub will eventually receive.

TESTING

Requirements

  • (optional) oc
  • (optional) crc
  • (for test-scorecard-local) docker or podman
  • (optional, for test-scorecard-local-kind) kind and docker or podman

Instructions

make test-envtest will run controller tests using ginkgo if installed, or go test if not, requiring no cluster connection.

make test-scorecard will run the Operator SDK's scorecard test suite. This requires a Kubernetes or OpenShift cluster to be available and logged in with your kubectl or oc client. The recommended setup for development testing is CodeReady Containers (crc).

make test-scorecard-local runs the custom scorecard tests from the local source against BUNDLE_IMG. It uses IMAGE_BUILDER (podman by default, or docker) to pull the bundle and copy its manifests and metadata into a temporary directory. The container and temporary directory are removed when the tests finish. There is no need to regenerate ./bundle when switching bundle images. For example:

BUNDLE_IMG=quay.io/cryostat/cryostat-operator-bundle:latest-ocp \
SCORECARD_TEST_SELECTION=console-plugin make test-scorecard-local

With SCORECARD_TEST_ONLY=true, set BUNDLE_IMG to the bundle already installed in SCORECARD_NAMESPACE.

Before the scorecard tests are run, all cryostat and cryostat-operator resources will be deleted to ensure a clean slate.

make test will run all tests.

Agent Scorecard tests

cryostat-agent deploys the sample Java application with a bundled Cryostat Agent, configures mutual TLS using the Cryostat CA, waits for Agent registration, and runs the recording lifecycle used by cryostat-recording. cryostat-agent-autoconfig deploys an image without the Agent and labels only the Deployment. It verifies that the Deployment webhook propagates the labels and the Pod webhook injects the Agent while preserving the application's JVM options, then checks registration and runs the same recording lifecycle. Both tests call Cryostat through its authenticated in-cluster Service. Neither application exposes remote JMX. Application, init-container, and Cryostat logs are included in the test output. Sample resources are owned by the test's Cryostat CR and removed with it.

The sample source and image builds live in cryostatio/test-applications. Its existing workflows build both samples and publish them under quay.io/redhat-java-monitoring:

  • cryostat-operator-scorecard-app: application without an Agent.
  • cryostat-operator-scorecard-agent-app: application with the Agent JAR at /opt/cryostat/cryostat-agent-shaded.jar.

make scorecard-build builds only the test runner, which stays under quay.io/cryostat. The runner uses the published sample images by default. SCORECARD_SAMPLE_VERSION selects their tag (default latest); override SCORECARD_SAMPLE_IMG and SCORECARD_AGENT_IMG to embed specific references, including digests, in the runner. The sample tag is independent of the runner's timestamped tag. For tests run directly from source, use CRYOSTAT_SCORECARD_SAMPLE_IMAGE and CRYOSTAT_SCORECARD_AGENT_IMAGE.

To test unpublished sample changes, supply a local test-applications checkout. The Kind flow then builds both samples using that checkout's scripts and serves them from its local registry:

SCORECARD_APPLICATIONS_DIR=/path/to/test-applications \
OPERATOR_IMG=quay.io/cryostat/cryostat-operator:latest \
CORE_IMG=quay.io/cryostat/cryostat:latest \
SCORECARD_TEST_SELECTION="cryostat-agent,cryostat-agent-autoconfig" \
make test-scorecard-local-kind

The sample scripts require Podman. They copy the bundled JAR from AGENT_INIT_IMG, matching the Operator's injection artifact. To build samples separately for an existing cluster:

MANIFEST_PUSH=false make scorecard-app-build \
  SCORECARD_APPLICATIONS_DIR=/path/to/test-applications \
  PLATFORMS=linux/amd64 \
  SCORECARD_SAMPLE_IMG=quay.io/me/scorecard-app:test \
  SCORECARD_AGENT_IMG=quay.io/me/scorecard-agent-app:test
# Make these images available to the cluster before running the tests.
CRYOSTAT_SCORECARD_SAMPLE_IMAGE=quay.io/me/scorecard-app:test \
CRYOSTAT_SCORECARD_AGENT_IMAGE=quay.io/me/scorecard-agent-app:test \
SCORECARD_TEST_SELECTION="cryostat-agent,cryostat-agent-autoconfig" \
make test-scorecard-local

Publishing the Agent Scorecard images

Use the Go version in go.mod and the Operator SDK version in Makefile (bin/operator-sdk takes precedence over a globally installed SDK). From the Operator repository root, build the runner with the published sample references:

export CUSTOM_SCORECARD_VERSION="5.0.0-$(date -u +%Y%m%d%H%M%S)"
export CUSTOM_SCORECARD_IMG="quay.io/cryostat/cryostat-operator-scorecard:${CUSTOM_SCORECARD_VERSION}"
# Prefer a published version/date tag or digest when preparing a runner release.
export SCORECARD_SAMPLE_IMG="quay.io/redhat-java-monitoring/cryostat-operator-scorecard-app:<published-tag>"
export SCORECARD_AGENT_IMG="quay.io/redhat-java-monitoring/cryostat-operator-scorecard-agent-app:<published-tag>"

make scorecard-build IMAGE_BUILDER=podman \
  PLATFORMS=linux/amd64,linux/arm64 MANIFEST_PUSH=false
make bundle

The build embeds the sample image references in the runner. make bundle regenerates bundle/tests/scorecard/config.yaml from the updated Scorecard configuration; it does not publish an image. Review the generated diff, including that file and config/scorecard/patches/custom.config.yaml, before including the updated Scorecard configuration in the PR. For an AMD64-only test cluster, PLATFORMS=linux/amd64 also works.

To make the runner available from Quay before merge, optionally publish it after the build and bundle regeneration succeed:

podman login quay.io
podman manifest push --all "$CUSTOM_SCORECARD_IMG" "docker://$CUSTOM_SCORECARD_IMG"

This command uploads a container image. Git commits and pushes are separate. Once the Operator change is merged upstream, its publishing workflow builds and publishes the checked-in runner tag under quay.io/cryostat if it does not already exist. Before merge, use the local Kind flow to validate the runner and replace the build-number placeholders in both Scorecard configs with a fresh timestamped tag.

Scorecard testing on a local kind cluster

make test-scorecard-local-kind runs the scorecard suite end-to-end against a dedicated kind cluster, with no external image registry required. It is a turnkey wrapper that:

  1. creates (or reuses) a kind cluster named cryostat-scorecard, configured with a containerd registry mirror;
  2. starts a local container registry, published to the host as localhost:5000 and mirrored to the same reference in each node's containerd, so one image reference resolves both on the host and in the cluster;
  3. installs the cluster prerequisites: OLM, cert-manager, and ingress-nginx (with an in-cluster CoreDNS rewrite so the tests can reach Cryostat via its ingress);
  4. builds the scorecard test image and the operator bundle locally, pushes them to the local registry, and runs the scorecard suite against the on-disk ./bundle.

Because the scorecard test image and bundle are served from the local registry, you can validate that a set of operand images passes the scorecard suite without publishing any scorecard-specific images. Supply the operand images under test via OPERATOR_IMG and CORE_IMG (and any other RELATED_IMAGE_* overrides), for example:

CORE_IMG=quay.io/my-user/cryostat:my-feature-pr \
OPERATOR_IMG=quay.io/my-user/cryostat-operator:my-feature-pr \
make test-scorecard-local-kind

Run a subset of tests with SCORECARD_TEST_SELECTION (e.g. SCORECARD_TEST_SELECTION=cryostat-cr), and tear everything down afterwards with make clean-scorecard-local-kind.

The container runtime backing kind is auto-detected from IMAGE_BUILDER (podman by default); set KIND_RUNTIME=docker if kind uses the docker provider. Under the rootless podman provider, run this as the only kind cluster on the host — creating it alongside another running kind cluster can disrupt cluster networking. Nothing here needs root.

The scorecard-kind-prereqs and scorecard-kind-push targets are shared with the CI scorecard job, and all of these can be invoked individually against an existing cluster: scorecard-kind-cluster, scorecard-kind-registry, scorecard-kind-prereqs, scorecard-kind-push (see make help). CI does not use scorecard-kind-registry: helm/kind-action's built-in registry support provides the equivalent registry there, addressed as kind-registry:5000 via the hosts entry the action adds on the runner.

About

A Kubernetes Operator to facilitate the setup and management of Cryostat.

Topics

Resources

Stars

37 stars

Watchers

6 watching

Forks

Releases

Packages

Used by

Contributors

Languages