enhancements/x509-trust: Propose a new enhancement

[wking]

Previous discussion in rhbz#1771564, openshift/installer#2658, etc., etc. ;). CC @abhinavdahiya, @bparees, @deads2k @dhansen, @dustymabe, @iamemilio, @kalexand-rh, @sdodson

[bparees]

running update-ca-trust requires root. not all containers can run as root.

[wking]

not all containers can run as root.

What's the alternative? Holding your trust under ~? Or do you have to union keys in Kube-space and use the single-bundle /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem workaround?

[bparees]

the latter is more or less what we did for proxy.

or you need to load your CAs from additional locations, if you have that flexibility in your transport libs.

[cgwalters]

What we really want of course is user namespaces so that "as root" is really only in the container. xref https://lwn.net/Articles/796700/

[bparees]

There are really two places to make this decision:

1. does the admin want to supply the total set of cluster CAs, or do they want to supply additionalCAs that should be appended to the cluster's system trust (as determined by the ubi image, basically)
2. does the each component accept the provided cluster CA bundle as the total set of CAs it trusts, or does it append the provided set to its own system trust bundle (it will always append its own component-level CAs, if any)

We'd really only talked about (1) so far, but you seem to be talking about (2).

[wking]

We'd really only talked about (1) so far, but you seem to be talking about (2).

This is basically "union with UBI defaults". I don't really care if that happens Kube-side (1) or container-side (2). I just assumed (2), because that's what we do now. Do you have a reason to pivot to (1)?

[bparees]

actually we do it somewhat kube side today and component side.

1. the proxy controller takes the additional CAs and appends the UBI system bundle to it
2. the proxy controller injects that new bundle into the labeled CMs
3. components consume that CM in whatever manner they choose. They may use it as the only set of CAs, they might union it with their own system trusts, they might union it with their own other sources of CAs, etc.

[wking]

actually we do it somewhat kube side today and component side.

Do you have a reason to preserve this, instead of pivoting to explicit append or override semantics?

1. the proxy controller takes the additional CAs and appends the UBI system bundle to it

Is this documented anywhere? I don't see anything about it here or here.

[bparees]

Do you have a reason to preserve this, instead of pivoting to explicit append or override semantics?

we aren't going to (successfully) dictate what components choose to do with the content... so i think on a component consumption perspective you will always have:

1. some that use what is provided as the only CAs
2. some that combine it w/ their own image system CAs
3. some that combine it with their own component-specific CA configuration

Plus (3) will probably always make sense for some components and not for others. meaning at least (1) and (3) are likely to always need to be allowed/implemented. (1) vs (2) we can potentially declare a best practice rule about..I think we'd choose (1) there.

Is this documented anywhere? I don't see anything about it here or here.

Today the additional CA bundle is a combination of user provided CAs plus the system CAs from the network operator image. In the future adding the system trusts to the bundle should be a configurable optional so customers who want to explicitly control the trusted CAs can do so.

[wking]

we aren't going to (successfully) dictate what components choose to do with the content...

We aren't? If I gave the CVO a ConfigMap with my company CA and override, and someone MitMed my upstream with a cert signed by a sloppy/compromised big CA, and the CVO accepted the connection to the attacker, I would be really mad at the CVO and file a strongly worded bug ;).

[bparees]

it's not just CVO, OLM operators will also consume this. which will include ISV operators, community operators, etc.

[smarterclayton]

I expected a new trust bundle system to expand the existing additionalTrustedCA. It’s not clear to me why we would have a second system?

https://github.com/openshift/api/blob/master/config/v1/types_proxy.go#L65

I expect to cover everything described here. Why would we have a second one?

Or, if it’s not sufficient, how does something new interact? How do they conflict?

[wking]

It’s not clear to me why we would have a second system?

Because you might have X.509 trust consumers that do not expect to be speaking to a proxy. This enhancement spun off from openshift/installer#2658 where I proposed setting TrustedCA on the proxy config in the absence of other proxy settings to support the network-routed-proxy case, and the pushback was around the proxy config only being used for the ask-apps-explicitly-to-use-a-proxy case. And at least the CVO currently only uses the proxy-config's TrustedCA when HTTPSProxy is set. This proposal works around that pushback by defining a higher-level default trust config (the default-ca-bundle ConfigMap) that components should be clearly falling back to even if they feel like the proxy config doesn't apply to them. So the CVO logic would be:

1. If HTTPSProxy is set (potentially also if HTTPProxy is set to an https:// URI) and TrustedCA is set, use TrustedCA when validating TLS connections. Otherwise...
2. If default-ca-bundle exists, use it when validating TLS connections. Otherwise...
3. Use the container's default (UBI) trust store when validating TLS connections.

That would support use cases like:

I'm not using a proxy, but I don't want my nodes trusting all the UBI CAs. The CVO needs to talk to api.openshift.com (for Cincy, signed by Let's Encrypt) and mirror.openshift.com or storage.googleapis.com (for signatures, signed by DigiCert and Google Trust Services respectively). Limit the CVO trust to those three CAs.

[danehans]

potentially also if HTTPProxy is set to an https:// URI

Note that if HTTPSProxy is unset, HTTPProxy is used to proxy https requests regardless of the url scheme.

[bparees]

@wking how do you see this interacting w/ the existing "proxy" CA mechanism?

Today the "proxy" mechanism is proxy in one specific way only: the labeled configmaps get content injected based on the CM linked from the ProxyConfig resource.

Nothing else about the flow mentions proxy:

1. the installer collects the CA bundle as "additionalTrustCAs"
2. the configmap name in openshift-config can be anything and defaults to "user-ca-bundle"
3. the proxyconfig field itself that references it is named "trustedCA"
4. the label that causes the CAs to be injected into other CMs is named "config.openshift.io/inject-trusted-cabundle"

So if we're going to introduce a second mechanism for distributing CA bundles, what is the expectation for existing consumers of the "proxy" bundle?

And if we end up with disjoint sets (one bundle for the "proxy' and another bundle for "other CAs") how do we expect components to consume both, if they need to? Up to them to munge things together?

I think it might be better if we can find a way to evolve the existing mechanism (e.g. break the coupling to ProxyConfig) as a first step.

Things the current "proxy" mechanism is lacking:

1. it's tied to specifying values in a ProxyConfig resource even if you don't really have a proxy
2. it doesn't offer the option of "append" vs "replace" (you will always get the UBI system trusts appended to whatever trusts you supply injected into the CM. Components themselves may also add their system trusts or they may replace their trusts with what is injected into their CM).

Can we do something simpler and just address those? Do we even need proxy specific CAs, or can we live with a global cluster CA bundle for now that serves as the proxy bundle as well, for things that want to talk to a proxy? (effectively just rename what we did in 4.2 as the global cluster CA distributor instead of the proxy CA distributor)

[wking]

So if we're going to introduce a second mechanism for distributing CA bundles, what is the expectation for existing consumers of the "proxy" bundle?

I'll fill in this today.

[wking]

I've pushed 8698b5b -> 3d92920, renaming the proxy-only injection label to config.openshift.io/inject-proxy-cabundle and re-purposing config.openshift.io/inject-trusted-cabundle to be the cluster-scoped default trust. I expect most of the existing proxy consumers can pivot to inject-proxy-cabundle (unless they have non-egress-TLS X.509 logic I'm not aware of, which is possible). Does that address your concerns, @bparees?

[bparees]

@wking there's a bit of a migration headache here. In the end state you're saying:

1. CM's labeled w/ inject-trusted-ca-bundle get the content of the "default-ca-bundle" CM in openshift-config
2. CM's labeled w/ inject-proxy-cabundle get the content of the CM referenced by the proxyconfig.

In the current state, CM's labeled with inject-trusted-ca-bundle get the content of the CM referenced by the proxyconfig.

To do this safely it seems like we need some period of time in which the "proxy" CMs are all labeled w/ both labels. Then switch the proxy injection logic over to look for injected-proxy-cabundle. Then remove the inject-trusted-ca-bundle label. Then we can introduce the new behavior that respects the inject-trusted-ca-bundle label.

Otherwise you risk the wrong/unexpected thing being injected into a CM.

You might be able to simplify the migration by not reusing the inject-trusted-ca-bundle label for the new behavior.

[wking]

You might be able to simplify the migration by not reusing the inject-trusted-ca-bundle label for the new behavior.

Good idea. I've tried to write that up in 00ead3d -> e5d3ceb; how does the new Removing a deprecated feature section look to you?

[iamemilio]

Could we automate the creation of environment variables containing the cert bundles, or create a volume using the same mechanism as proxy. It would streamline how we use the injection process.

[wking]

Could we automate the creation of environment variables containing the cert bundles...

It's not clear to me how this would work. Can you give example Deployment YAML and talk us through this?

[iamemilio]

Sure, I was just extrapolating from this excerpt from the global cluster egress policy enhancement.

deployments with the config.openshift.io/inject-proxy: will get the current proxy environment variables injected (HTTP_PROXY, HTTPS_PROXY, NO_PROXY) by the cluster version operator.

So in our case we could add config.openshift.io/add-trusted-ca-bundle: <container name> to our deployments and the cluster-version-operator would add a volume that mounts the trusted-ca-bundle configmap, and an inline command to add it to the container's trusted CAs.

[danehans]

@iamemilio I'm happy that I came across your comment. I'm working on adding proxy support to the ingress operator and was considering a similar approach. I feel like we need a better solution to mounting the trust bundle than what currently exists (see image registry operator).

@wking let's use ingress operator as an example. The deployment would be updated with:

  annotations:
    config.openshift.io/add-trusted-ca-bundle: "ingress-operator" # container name to inject bundle

CVO watches deployments with the above annotation and creates a volume for the operator's trust bundle configmap and mounts the volume in the container(s) specified by the annotation. The ingress operator's deployment would look like the following after CVO finishes reconciliation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-operator
  namespace: openshift-ingress-operator
  annotations:
    config.openshift.io/add-trusted-ca-bundle: ingress-operator
spec:
<SNIP>
      containers:
        - name: ingress-operator
<SNIP>
          volumeMounts:
            - name: trusted-ca
              mountPath: /var/run/configmaps/trusted-ca/
<SNIP>
        configMap:
          name: trusted-ca
          optional: true
          items:
            - key: ca-bundle.crt
              path: tls-ca-bundle.pem

[bparees]

i think you will have a hard time making the case that the CVO should do this.

the only reason the CVO took ownership of setting env vars on the operator deployments is because it is impossible for the operator to set env vars on its own deployment(the cvo would stomp away the values). The same restriction doesn't apply to the configmap volume mounting, since the operator can define its deployment with a CM volume mount, and define the CM to be mounted.

[danehans]

@bparees thanks for the feedback. I'll look into having the ingress operator programmatically manage it's own trust bundle volume/volumeMount. However, having a single controller that manages this would provide consistency across operators.

[iamemilio]

@bparees what would you suggest as a solution then? Should an operator be given the responsibility for injecting this into container deployments, or do you think we should drop it an just add it manually to all the services.

[bparees]

@iamemilio today operators that need proxy CAs in their operator deployment, configure themselves accordingly:

1. they define a configmap that will have the CAs injected into it:
   https://github.com/openshift/cluster-image-registry-operator/blob/a0efd48f8bc2c8d18d53fc5a9e5b8515b94bc2a4/manifests/04-ca-trusted.yaml#L8

2. they mount that configmap in their deployment:
   https://github.com/openshift/cluster-image-registry-operator/blob/a0efd48f8bc2c8d18d53fc5a9e5b8515b94bc2a4/manifests/07-operator.yaml#L93

3. they watch the mounted content for changes and restart their process as needed:
   https://github.com/openshift/cluster-image-registry-operator/blob/a0efd48f8bc2c8d18d53fc5a9e5b8515b94bc2a4/manifests/07-operator.yaml#L62-L73

That is what i would expect here

[bparees]

how does the new Removing a deprecated feature section look to you?

i like it.

[ironcladlou]

@Miciah PTAL, would like to understand if this solves the ingresscontroller trust issues.

[smarterclayton]

The existing proxy value has to be unified with this (or we just use that).

[smarterclayton]

Why?

[wking]

Why?

Because you don't need to have a global trust store for the whole cluster. You get better isolation against compromised CAs if each component only trusts the CAs it needs to trust (e.g. the CVO scoping I sketched out here). But while that scoping is nice for the paranoid, it's also a pain, which is why I think we also want to make it easy for folks to mix and match cluster-scoped defaults and per-component overrides to match their security stance.

[smarterclayton]

I expected a new trust bundle system to expand the existing additionalTrustedCA. It’s not clear to me why we would have a second system?

https://github.com/openshift/api/blob/master/config/v1/types_proxy.go#L65

I expect to cover everything described here. Why would we have a second one?

Or, if it’s not sufficient, how does something new interact? How do they conflict?

[smarterclayton]

We do not deprecate features like this. It is forever.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wking
To complete the pull request process, please assign mfojtik
You can assign the PR to them by writing /assign @mfojtik in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[danehans]

Each component will define its own well-known ConfigMap(s) in its own namespace, with the same ca-bundle.crt and composition keys as the cluster-scoped ConfigMap.

Operators currently define their own configmap statically. See the registry operator as an example. If I understand correctly, operators will need to get the value of composition from the cluster-scoped configmap and create the operator-scoped configmap programatically. Is my understanding correct?

[wking]

If I understand correctly, operators will need to get the value of composition from the cluster-scoped configmap and create the operator-scoped configmap programatically

Nope, just make a create-only ConfigMap, as described here. Setting composition locally will get clobbered by the network operator using this logic if you have any of the network-operator labels set, but admins could also unset the network-operator labels and set composition manually on the per-component ConfigMap as they see fit.

[smarterclayton]

I expected the following:

1. There is a general purpose, global for the cluster, "additionalTrustedCAs" that every component uses unless otherwise told not to, in some global configuration like "trust" or whatever (we could use proxy as is, or add a new global config)
2. Downstream components may declare other additionalTrustedCAs that are appended to this list (proxy CA trust), and if people really didn't want to trust the global CAs, they could support trustedCAs that ignores the upstream in their components

[iamemilio]

There is a general purpose, global for the cluster, "additionalTrustedCAs" that every component uses unless otherwise told not to, in some global configuration like "trust" or whatever (we could use proxy as is, or add a new global config)

@smarterclayton I believe the consensus here was that a new config should be created so that we don't muddy the proxy work-flow. Trevor nicely summarized this here. We are proposing that the global configmap that cluster resources use by default be the newly created default-ca-bundle configmap.

Downstream components may declare other additionalTrustedCAs that are appended to this list (proxy CA trust), and if people really didn't want to trust the global CAs, they could support trustedCAs that ignores the upstream in their components

I believe this is also a core goal of the original proposal that is outlined in the Per-component Trust section.

[deads2k]

I expected the following:

1. There is a general purpose, global for the cluster, "additionalTrustedCAs" that every component uses unless otherwise told not to, in some global configuration like "trust" or whatever (we could use proxy as is, or add a new global config)
2. Downstream components may declare other additionalTrustedCAs that are appended to this list (proxy CA trust), and if people really didn't want to trust the global CAs, they could support trustedCAs that ignores the upstream in their components

I don't think this is a secure plan for customers. Trust is subdivided inside of our clusters to ensure that we're two bugs away from compromise. If trust is indiscriminately scoped and CAs which are not valid signers for certain certificates are trusted, then you can end up in states where instead of two compromises (the signer and DNS or network), you will only be one compromise (DNS or network) because the wrong trust domain is used.

@wking identified the same problem here: #115 (comment) . There are legitimate cases for subdivided trust. Internally signed trust relationships are another.

The idea of default trust for external (outside the cluster) signers makes sense. The idea of unioning that trust doesn't make sense as the only option because if you know to expect certain signers you want to use those (again @wking has explained this above too), but it's ok as an option. Neither of these trust models are the same as proxy trust.

Given the usages we have already seen where we need to have different trust bundles for different registries, it is not obvious that a big-ball-o- trust is what we really desire. For image-registries we subdivided based on names, which may be more appropriate overall and will lead more correct injection by the consuming components. It doesn't preclude a component from making a big-ball-of-trust, but it enables components that can do better to be able to .

[bparees]

@wking I just had a discussion with @smarterclayton and @deads2k about how to move this forward and our conclusion was that the primary use case this was going to fulfill (components that need to talk to cloudproviders that are using certs that can be validated w/ the system trust store) is better solved by proper consumption of the existing cloudprovider config on the cluster which includes a CA reference.

(and in fact the existing consuming components we're aware of that had issues talking to, for example openstack, have been addressed, for example: https://github.com/openshift/cluster-image-registry-operator/blob/master/pkg/storage/swift/swift.go#L148-L154, which was added via openshift/cluster-image-registry-operator#445)

So barring a new significant use case of "i need to provide CAs for some external service to many cluster components", we concluded that there is no present need for a "global CA distribution mechanism" at this point.

Unless you disagree, I think this can be closed out for now.

[wking]

So barring a new significant use case of "i need to provide CAs for some external service to many cluster components"...

Isn't the "I'm pushing traffic through a proxy by configuring network routing, not via a Proxy config object" a use case for "I need to provide CAs for some external service to several cluster components"? That's just the components that are already configured to get proxy CAs injected. Or maybe the narrower additionalTrustBundle scoping you're recommending means that we actually do want the code change from openshift/installer#2658?

[bparees]

Isn't the "I'm pushing traffic through a proxy by configuring network routing, not via a Proxy config object" a use case for "I need to provide CAs for some external service to several cluster components"?

Isn't that solved by the existing proxy CA mechanism? (worst case, you configure a proxyconfig with no actual proxy values, but with a dummy noproxy value). It's still a scenario of "all components making external requests must trust this CA for all their connections" (solved by proxy CA controller) which is distinct from "connection to this specific endpoint must use this additional CA" (not solved by proxy CA controller).

That's just the components that are already configured to get proxy CAs injected.

any component that makes external requests is supposed to respect the proxy CA injection/configuration. The only iffy case was "my component makes calls to the hosting cloud provider and we assumed you'd never go through a proxy to talk to the hosting cloud provider". The answer to that case is what i linked to the in the registry-operator: respect the cloudconfig which provides the CA to use.

[bparees]

with respect to: openshift/installer#2658

i think we can simply document the solution, which i believe is:
if you have a transparent network proxy, provide the installer with a proxy config that consists of a noproxy value, but no actual proxy hosts.

i put a comment in https://bugzilla.redhat.com/show_bug.cgi?id=1771564 about needing to improve the doc, but i didn't mention documenting that particular aspect of it.

[wking]

... which is distinct from "connection to this specific endpoint must use this additional CA" (not solved by proxy CA controller).

Yeah. I'm not aware of any approach that addresses this, short of passing around separate bundles for each possible endpoint. Would be nice if there was an OpenSSL approach that be configured to allow host-specific CA selection, but I don't think that exists today.

if you have a transparent network proxy, provide the installer with a proxy config that consists of a noproxy value, but no actual proxy hosts.

Folks who fiddled with this couldn't get it to work, but I'm not sure they tried that specific approach. I can try to hunt down the transparent network proxy bugs and see if they've tried it and if not, whether it would work for them going forward.

And pulling in your bug comment:

perhaps it's for things like talking to a self-hosted disconnected mirror registry?

Still not clear to me how you're supposed to address this at install time. Do we want a new enhancement around that question more narrowly?

[wking]

ok.

• openstack-self-signed-certificates #191 is an example of the "certs from your provider config" angle.
• additionalTrustBundle is growing better docs in Bug 1771564: docs/user/customization: Document additionalTrustBundle vs. proxy installer#3039, and now more clearly covers the node and sometimes proxy consumers. Name doesn't make that obvious, but too late to fix now, so docs are what we get.
• trustedCA grew better docs around system node injection in Bug 1797107: config/v1/types_proxy: Clarify trustedCA semantics api#582, and explicitly allows consumers to use trusted-ca-bundle for non-proxy connections or not as they see fit.
• The CVO is moving to use trusted-ca-bundle in Bug 1797123: pkg/cvo: Fetch proxy CA certs from openshift-config-managed/trusted-ca-bundle cluster-version-operator#311.
• The MCO might move to use trusted-ca-bundle with rhbz#1784201.

I'm now on the same page as @bparees earlier with no known, real-world use-cases that aren't addressed by the existing logic, so.

/close

[openshift-ci-robot]

@wking: Closed this PR.

Details

In response to this:

ok.

• openstack-self-signed-certificates #191 is an example of the "certs from your provider config" angle.
• additionalTrustBundle is growing better docs in Bug 1771564: docs/user/customization: Document additionalTrustBundle vs. proxy installer#3039, and now more clearly covers the node and sometimes proxy consumers. Name doesn't make that obvious, but too late to fix now, so docs are what we get.
• trustedCA grew better docs around system node injection in Bug 1797107: config/v1/types_proxy: Clarify trustedCA semantics api#582, and explicitly allows consumers to use trusted-ca-bundle for non-proxy connections or not as they see fit.
• The CVO is moving to use trusted-ca-bundle in Bug 1797123: pkg/cvo: Fetch proxy CA certs from openshift-config-managed/trusted-ca-bundle cluster-version-operator#311.
• The MCO might move to use trusted-ca-bundle with rhbz#1784201.

I'm now on the same page as @bparees earlier with no known, real-world use-cases that aren't addressed by the existing logic, so.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.