Support for separating deployment from management cluster

[avorima]

Most (if not all) Cluster API providers can be deployed in a Kubernetes cluster that's separated from the environment where they provision resources, e.g. the AWS provider could be in a different VPC than the instances it deploys.
This is currently not possible with Kamaji, because it deploys control planes in the same cluster that it watches custom resources in.

I have the following setup in mind:

cluster A: CAPI controlllers
cluster B: control planes
cluster C: same as B but different location

This would require breaking the ownership chain at one point, e.g. between KCP and TCP, so resources would need to be cleaned up by the controller directly. The benefit is that is allows each cluster to implement different resource and access management policies.

[prometherion]

It's something we always had in mind, and waiting for community's rising interest!

We considered providing a dedicated kubeconfig to the Kamaji CAPI Control Plane provider to select which Kubernetes cluster should host the Tenant Control Plane resources. If I understood correctly, you'd like to have more potential clusters, e.g.:

site a: hosting control planes for tenant `alpha`
site b: hosting control planes for tenant `bravo`
site {x...y}: hosting worker nodes of tenants

Such an approach would require a new field in the KamajiControlPlane resource, I guess. We could reference a secret containing the kubeconfig, or maybe a Cluster API cluster name?

In this latter case, should we support cross-namespace Cluster references? Or should it be a local reference as we're all ready expecting with Secret or ConfigMap references?

Happy an excited to discuss more!

[avorima]

While I agree that this makes misuse/misconfigurations easier, cross-namespace references don't necessarily require users to also have access in the namespaces they reference.
Provider implementations already tend to be allowed to reference cross-namespace because of ClusterRole + ClusterRoleBinding, but generally don't do it out of architectural benefits like ownership.

Nonetheless, I'm fine with only allowing local references. Keeps things simple and as you said it's the preferred way of doing things in CAPI.

So in the case of a kubeconfig reference this could look like:

type ExternalClusterReference struct {
	KubeconfigSecretName string `json:"kubeconfigSecretName"`
	TargetNamespace      string `json:"targetNamespace,omitempty"`
}

type KamajiControlPlaneSpec struct {
	ExternalClusterRef *ExternalClusterReference `json:"externalClusterRef,omitempty"`
}

I think it would be best to put everything into a struct that groups all external cluster reference configurations together to allow iterating on the fields without affecting the rest of KCP spec. It would be disabled and hidden by default. I could also imagine a feature flag, e.g. USE_EXTERNAL_CLUSTER_REF=true, that allows admins to decide whether they want to expose this feature to their users.
The KubeconfigSecretName points to a secret in the same namespace that is expected to contain a kubeconfig. This kubeconfig will be used when provisioning TCPs and copying secrets that kamaji creates for it.
I added the TargetNamespace to allow managing resources in a different namespace than the namespace the KCP is currently running in. It would be used when the external cluster reference is not nil and default to the KCP namespace. Could be enforced by defaulting webhook.

What do you think?

[prometherion]

What do you think?

That I love designing features directly with the community: that's what I'm looking for, feeling flattered.

I think it would be best to put everything into a struct that groups all external cluster reference configurations together to allow iterating on the fields without affecting the rest of KCP spec. It would be disabled and hidden by default. I could also imagine a feature flag, e.g. USE_EXTERNAL_CLUSTER_REF=true, that allows admins to decide whether they want to expose this feature to their users.

I like this approach of features flag, this would be very helpful since it would be bring more control on the feature that could arise some security concerns.

So in the case of a kubeconfig reference this could look like:

The YAML LGTM, even tho it could change according to the eventual implementation.

[avorima]

Amazing. We'll start working on a PoC soon so we'll see if this design holds up.

[prometherion]

A first implementation is available with #108, internals and design discussion are in the PR, I just had to add a field we didn't talk about (the Namespace of the remote cluster where the Tenant Control Plane must be deployed) as well as the name of the key containing the kubeconfig file to use for connecting to the remote cluster.

[avorima]

Thank you! I'll check it out

[pacharya-pf9]

Definitely a useful feature.
+1

[prometherion]

#108 has been merged, marking this feature in ALPHA state using feature flags.

It would be great to have some tests on this trying to spot bugs, the discussion can be closed since bugs must be filled in the Issue section.