Leader election events are recorded with the wrong HTTP client

[rubofelipe]

The version v0.15.0 introduced an issue for controllers that use a separated kubeconfig for leader election:

mgr, err := ctrl.NewManager(configA, manager.Options{
    ...
    LeaderElectionConfig: configB,
})

The lease object is correctly created in the cluster pointed by the configB and the controller is able to acquire the leadership. However, an error is returned on creation of the Event object associated to the coordination object (e.g Lease):

E0821 20:38:02.494486       1 event.go:289] Unable to write event: '&v1.Event{*****}': 'Post "https://api.****:443/api/v1/namespaces/my-namespace/events": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ca")'(may retry after sleeping)

The kubernetes API server is not trusting the certificate authority presented by the HTTP client.

The origin of this problem is the commit 4fd4f6e

A new an unique HTTP client was created into Options struct:

controller-runtime/pkg/cluster/cluster.go

Lines 264 to 272 in c20ea14

	// setOptionsDefaults set default values for Options fields.
	func setOptionsDefaults(options Options, config *rest.Config) (Options, error) {
	if options.HTTPClient == nil {
	var err error
	options.HTTPClient, err = rest.HTTPClientFor(config)
	if err != nil {
	return options, err
	}
	}

This HTTP client is being copied to the Cluster struct and being passed as argument to the function that creates the leaderRecorderProvider:

controller-runtime/pkg/manager/manager.go

Lines 351 to 360 in c20ea14

	if options.LeaderElectionConfig == nil {
	leaderConfig = rest.CopyConfig(config)
	leaderRecorderProvider = recorderProvider
	} else {
	leaderConfig = rest.CopyConfig(options.LeaderElectionConfig)
	leaderRecorderProvider, err = options.newRecorderProvider(leaderConfig, cluster.GetHTTPClient(), cluster.GetScheme(), options.Logger.WithName("events"), options.makeBroadcaster)
	if err != nil {
	return nil, err
	}
	}

So the options.newRecorderProvider is receiving the LeaderElectionConfig (i.e. configB) and an HTTPClient based on configA. The options.newRecorderProvider is a function pointer that is pointing to NewProvider function:

controller-runtime/pkg/internal/recorder/recorder.go

Lines 114 to 126 in c20ea14

	func NewProvider(config *rest.Config, httpClient *http.Client, scheme *runtime.Scheme, logger logr.Logger, makeBroadcaster EventBroadcasterProducer) (*Provider, error) {
	if httpClient == nil {
	panic("httpClient must not be nil")
	}
	corev1Client, err := corev1client.NewForConfigAndClient(config, httpClient)
	if err != nil {
	return nil, fmt.Errorf("failed to init client: %w", err)
	}
	p := &Provider{scheme: scheme, logger: logger, makeBroadcaster: makeBroadcaster, evtClient: corev1Client.Events("")}
	return p, nil
	}

The documentation of NewForConfigAndClient function gives us an important info:

// NewForConfigAndClient creates a new CoreV1Client for the given config and http client.
// Note the http client provided takes precedence over the configured transport values.
func NewForConfigAndClient(c *rest.Config, h *http.Client) (*CoreV1Client, error) {

The HTTP Client transport configurations have precedence in relation to the rest.Config argument. Because this HTTP Client was created with configA (which has different TLS certificate chain of trust), that means that is impossible to create Events on cluster informed on LeaderElectionConfig option, since the HTTP client won't use CA informed on LeaderElectionConfig.

[alvaroaleman]

/kind bug