Add ability to specify/target downstream Rancher "Projects" with Bundles

[nickgerace]

Since Fleet is a required component of Rancher as of Rancher v2.6, Fleet should take advantage of the integration point and enable the user to specify/target downstream clusters by Rancher "Project".

Workaround: split Rancher project by FleetWorkspace.

[skaven81]

I really do want this feature implemented, but I'm wondering what the best way would be to implement it, such that it allows Fleet to continue existing as an independent open source project that doesn't require Rancher to function.

Fundamentally, assigning a new namespace (that fleet-agent creates in the service of deploying a Bundle) to a Project, is a simple matter of ensuring that a specific label is present on the namespace when it is created. For Fleet users that have small or relatively static environments, the projectID labels can be included in the Bundle in a Helm chart's values.yaml or updated via a Fleet overlay.

It's for larger environments (like mine) where this feature is really needed. And really what it boils down to is how to create a Bundle that can dynamically generate the projectID label(s), based on some kind of query against information readily available in either the master cluster (where the fleet-controller is running) or the downstream cluster (where the Bundle is being deployed).

Before getting into the technical details of a possible implementation, we should consider what this should look like from a Fleet user perspective. There are two approaches that I see.

One is that assigning namespace(s) to Projects happens in the context of the Bundle's fleet.yaml. Some sort of specification/DSL is provided that allows a Fleet user to issue a K8s query of some kind (or maybe invoke a fleet-controller plugin) that returns a string that is then used as a label to be added to any namespaces created for the Bundle. This viewpoint could be useful as it tends toward a solution where the fleet-controller (or perhaps fleet-agent in the downstream cluster) is empowered to run some sorts of kube-apiserver queries on behalf of the Bundle author and use that data to generate dynamic content for the deployed Bundle. This starts to turn into a generic and possibly very powerful feature that would complement the "overlays" functionality and would thus be independent of Rancher (though being able to assign namespaces to a Project would be one of many uses).

Another perspective would be to look at this request narrowly and reason that since the goal is simply to ensure namespaces are created in the right Project, and that since Projects are, by definition, Rancher-specific, that the implementation should be done entirely (or at least mostly) in Rancher itself, not in Fleet. So in this approach, Fleet itself has no concept of a Project or the ability to dynamically add labels to a namespace. Rather, it's Rancher itself that contains the directives to say which Fleet-generated namespaces are to be assigned to which namespace, and then something along the lines of a mutating admission webhook ensures that any Fleet-derived namespaces that get created, are automatically assigned to the right Project. This approach has the advantage of being narrowly focused and thus is probably easiest to implement. However, it has the disadvantage that a crucial aspect of the Bundle's configuration -- what Project it is meant to be deployed to -- is not included anywhere in the Bundle's configuration in git. It breaks the "GitOps" promise of Fleet.

My personal preference would be the first approach: give Fleet additional non-Rancher-specific features that enable Bundle authors to get the desired behavior when deploying to Rancher-managed clusters. This is mostly driven by the fact that it preserves the "GitOps promise" because it retains all of the Bundle deployment metadata in the Bundle itself (no "spooky action at a distance"). It's also because this approach would result in generic functionality that is likely to be useful in the future outside this one narrow use-case.

As I think through this, my suggestion for a technical implementation was to add the ability to call up "overlay plugins" as part of the Bundle rendering process (e.g. where "overlays" are handled). The "overlay plugins" could be implemented as a CRD that ships with Fleet, and behave essentially the same as a mutating admission webhook. The input would be the BundleDeployment, perhaps before or after overlays (maybe allow for both!). The output would be an updated BundleDeployment with the plugin's modifications made. Rancher could provide a "project assignment" plugin that does this.

Then the more I think about it, it sounds like this would just be a re-implementation of the existing upstream mutating admission webhooks feature, and now I consider that perhaps that's actually the right answer here. Rancher installs and manages a mutating admission webhook that watches for BundleDeployments. It then looks for a special annotation (or perhaps, special macros in the labels) and does the work of modifying the resulting resource with the correct configuration.

For the webhook approach to work, Fleet would need to be OK with mutating admission webhooks modifying BundleDeployments that it creates, without it triggering an endless loop of "modified" events. But other than that, I think this approach would be relatively easy to implement, sticks to upstream Kubernetes standards, keeps the intended configuration within the git repository with the Bundle, and allows further extensions beyond just Rancher.

[skaven81]

If we assume the webhook approach is used as described above, then the YAML for a namespace in the Fleet Bundle could look like this:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    field.cattle.io/projectId: {{rancher-projectid-lookup-by-name: System}}
  name: my-namespace

Other macro types could allow for different methods of lookup, for example, to be added to the same Project as another, existing namespace:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    field.cattle.io/projectId: {{rancher-projectid-lookup-by-namespace: kube-system}}
  name: my-namespace

The mutating admission webhook would simply look for these {{rancher-projectid-lookup-* labels when examining new or modified BundleDeployments, and upon seeing one, is able to use its privileged context to see a) which cluster the BundleDeployment is assigned to, and therefore b) to enumerate the Projects in that cluster's namespace, which allows c) obtaining the desired projectID.

The second approach (cloning an existing namespace) would require the webhook to talk to the Rancher API rather than the Kubernetes API, but the data is still there. And Rancher still gets to own the whole "how to assign namespaces to projects" mechanism (it owns and maintains the webhook) but Fleet requires only a minor modification to make it tolerant of modifications made by a MutatingAdmissionWebhook.

[all4innov]

any news on this issue ? as life changing feature