Dynamic RBAC for "disabled" & "readonly" NS Modes

With this change, those using "disabled" and "readonly" namespace modes
can now have the Operator create the required RBAC in a target
namespace.  Specifically, if during initialization the Operator detects
that it has the privileges required to create the RBAC it requires
(ServiceAccounts, Roles and RoleBindings) within a target namespace
to allow it to work properly with that namespace, it will go ahead and
reconcile that RBAC.  In other words, it will create the required RBAC
if it doesn't exist, or it will recreate it if it exists but is
invalid.  And now when a new ControllerGroup is created and started for
for a specific namespace, e.g. when a new target namespace is added
post-installation, the Operator will also reconcile any required
RBAC within that namespace (assuming it has the privileges to do so).

Additionally, on each refresh interval for the namespace controller,
the required RBAC in all targeted namespaces will now also be
reconciled.  This allows the Operator to self-heal it's required RBAC
when it has the ability to do so, therefore ensuring it has the RBAC
required to function properly.

When using the "disabled" namespace mode, a namespace controller is
now also created in support of RBAC reconciliation, providing the
ability to reconcile RBAC on each refresh interval even when the
Operator does not have namespace privileges. This is done by
creating a fake clientset, populated with only those namespaces
specified via the NAMESPACE environment variable during Operator
installation.  Therefore, while the namespace controller does not
truly have access to namespace resources within the Kubernetes cluster
while in this mode because it lacks the proper privileges (and
therefore cannot react to namespace creations, updates and deletions),
this allows it to leverage the same reconciliation logic as when using
"dynamic" or "readonly" modes, and therefore reconcile RBAC while in a
"disabled" mode (assuming it has the privileges to do so).  This also
allows the  Operator to properly respond to the loss of any RBAC
required to run various informers used by the Operator's various
controllers, even when in a "disabled" mode (this capability is
discussed further below).

To leverage the ability to reconcile RBAC when using the "disabled" or
"readonly" operating modes, the Operator ServiceAccount must be
assigned the privileges within any targeted namespace required to create
it's  required RBAC resources within that namespace.  A new
'add-targeted-namespace-dynamic-rbac.sh' script has been provided as a
means to create a new namespace that has these privileges.  However,
unlike the current 'add-targeted-namespace.sh' script, which creates the
actual ServiceAccounts, Roles and RoleBindings within a namespace so
that the Operator can utilize that namespace to create PostgreSQL
clusters, this script creates a Role and RoleBinding within the
namespace that allows the Operator itself to create and reconcile the
ServiceAccounts, Roles and RoleBindings it requires.

Both the Bash and Ansible installers have also been updated in support
of this feature.  When using Bash, if 'PGO_DYNAMIC_RBAC=true', then
the 'add-targeted-namespace-dynamic-rbac.sh' script is run to create
targeted namespaces (instead of 'add-targeted-namespace.sh'), ensuring
the Operator then has permissions to create it's required RBAC within
those namespaces per the details discussed above.  This same principle
applies to the Ansible installer to, specifically producing the same
results if "pgo_dynamic_rbac='true'" in the Ansible inventory file.

Finally, with this commit error handling for the various informers
utilized  across the Operator's various controllers has also been
updated. Specifically, now when starting a new controller group
containing the various controllers required for the Operator to
create PG clusters within a target namespace, a check is first done
to ensure the Operator has the proper privileges as required to run
the various informers in those controllers (specifically the required
"list" privileges).  If this check fails, starting of the controller
group is aborted and then retried on the next refresh interval for
the namespace controller.

Additionally, now on each refresh interval if the Operator detects
that it no longer has the privileges to properly run all controllers,
the controller group for that namespace will be stopped and removed,
after which an attempt will be made to start it again on the next
namespace refresh interval (where it will again fail, and the
controller group will not be started, if it still does not have the
proper privileges). The changes prevent the informers from
overwhelming the Operator logs with a huge volume of privilege error
messages in the event any required privileges are lost.

Is should also be noted that as of this commit the  Ansible and Bash
installers no longer create namespaces when using the "dynamic"
namespace Operating mode.  Instead, the Operator itself will handle
namespace creation and initialization when in this mode.  At this point
users on Kube 1.12 or less (e.g. OpenShift 3.11 users) should utilize
either the "readonly" or "disabled" namespace  Operating modes, since
the "dynamic" is not fully/properly supported with these Kube versions.

And finally, this commit also includes the following:
- Worker queues have been added to the Namespace controller, and both
the Operator and Scheduler have been updated accordingly
- Requeuing has been removed from both the ConfigMap controller to
prevent a large volume of error under certain circumstances, i.e. if
required RBAC is missing.  Instead the action will simply be retried
on the next refresh interval.

Author: andrewlecuyer

Gopkg.lock

@@ -98,7 +98,7 @@ func CreateNamespace(clientset *kubernetes.Clientset, createdBy string, request
 	//iterate thru all the args (namespace names)
 	for _, namespace := range request.Args {
 
-		if err := ns.CreateNamespaceAndRBAC(clientset, apiserver.InstallationName,
+		if err := ns.CreateNamespace(clientset, apiserver.InstallationName,
 			apiserver.PgoNamespace, createdBy, namespace); err != nil {
 			resp.Status.Code = msgs.Error
 			resp.Status.Msg = err.Error()
@@ -148,14 +148,21 @@ func UpdateNamespace(clientset *kubernetes.Clientset, updatedBy string, request
 	//iterate thru all the args (namespace names)
 	for _, namespace := range request.Args {
 
-		if err := ns.UpdateNamespaceAndRBAC(clientset, apiserver.InstallationName,
+		if err := ns.UpdateNamespace(clientset, apiserver.InstallationName,
 			apiserver.PgoNamespace, updatedBy, namespace); err != nil {
 			resp.Status.Code = msgs.Error
 			resp.Status.Msg = err.Error()
 			return resp
 		}
-		resp.Results = append(resp.Results, "updated namespace "+namespace)
+		//apply targeted rbac rules here
+		if err := ns.ReconcileTargetRBAC(clientset, apiserver.PgoNamespace,
+			namespace); err != nil {
+			resp.Status.Code = msgs.Error
+			resp.Status.Msg = err.Error()
+			return resp
+		}
 
+		resp.Results = append(resp.Results, "updated namespace "+namespace)
 	}
 
 	return resp

apiserver/namespaceservice/namespaceimpl.go

@@ -58,6 +58,9 @@ const (
 	// DefaultConfigMapWorkerCount defines the default number or workers for the worker queue
 	// in the ConfigMap controller
 	DefaultConfigMapWorkerCount = 2
+	// DefaultNamespaceWorkerCount defines the default number or workers for the worker queue
+	// in the Namespace controller
+	DefaultNamespaceWorkerCount = 3
 	// DefaultPGClusterWorkerCount defines the default number or workers for the worker queue
 	// in the PGCluster controller
 	DefaultPGClusterWorkerCount = 1

config/defaults.go

@@ -240,6 +240,7 @@ type PgoStruct struct {
 	ConfigMapWorkerCount           *int
 	ControllerGroupRefreshInterval *int
 	NamespaceRefreshInterval       *int
+	NamespaceWorkerCount           *int
 	PGClusterWorkerCount           *int
 	PGOImagePrefix                 string
 	PGOImageTag                    string

config/pgoconfig.go

@@ -36,7 +36,8 @@ func (c *Controller) handleConfigMapSync(key string) error {
 
 	namespace, configMapName, err := cache.SplitMetaNamespaceKey(key)
 	if err != nil {
-		return err
+		log.Error(err)
+		return nil
 	}
 
 	configMap, err := c.cmLister.ConfigMaps(namespace).Get(configMapName)