You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
50
55
51
56
## Summary
52
-
We need to run multiple CAPI instances in one cluster and divide the namespaces to be watched by given instances.
57
+
As a Service Provider/Consumer, a management cluster is used to provision and manage the lifecycle of Kubernetes clusters using the Kubernetes Cluster API (CAPI).
58
+
Two distinct paradigms coexist to address different operational and security requirements.
53
59
54
-
We want and consider:
55
-
- each CAPI instance:
56
-
- is running in separate namespace and is using its own service account
57
-
- can select by command the line arguments the list of namespaces:
58
-
- to watch - e.g.: `--namespace <ns1> --namespace <ns2>`
59
-
- to exclude from watching - e.g.: `--excluded-namespace <ns1> --excluded-namespace <ns2>`
60
-
- we are not supporting multiple versions of CAPI
61
-
- all running CAPI-instances:
62
-
- are using the same container image (same version of CAPI)
- NOTE: the web-hooks are pointing from the CRDs into the first instance only
71
-
- the `ClusterRole/capi-aggregated-manager-role`
72
-
- the `ClusterRoleBinding/capi-manager-rolebinding` to bind all service accounts for CAPI instances (e.g. `capi1-system:capi-manager`, ..., `capiN-system:capi-manager`) to the `ClusterRole`
60
+
### Paradigm 1: Isolated Cluster Management
61
+
Each Kubernetes cluster operates its own suite of CAPI controllers, targeting specific namespaces as a hidden implementation engine.
62
+
This paradigm avoids using webhooks and prioritizes isolation and granularity.
63
+
64
+
**Key Features**:
65
+
66
+
-**Granular Lifecycle Management**: Independent versioning and upgrades for each cluster's CAPI components.
67
+
-**Logging and Metrics**: Per-cluster logging, forwarding, and metric collection.
68
+
-**Resource Isolation**: Defined resource budgets for CPU, memory, and storage on a per-cluster basis.
69
+
-**Security Requirements**:
70
+
-**Network Policies**: Per-cluster isolation using tailored policies.
71
+
-**Cloud Provider Credentials**: Each cluster uses its own set of isolated credentials.
72
+
-**Kubeconfig Access**: Dedicated access controls for kubeconfig per cluster.
73
+
74
+
In the current state there is the option `--namespace=<ns>` and CAPI can watch only one specified namespace while using this option or all namespaces without this option.
75
+
The extension to enable the existing command-line option `--namespace=<ns1, …>` define multiple namespaces is proposed in this PR [#11397](https://github.com/kubernetes-sigs/cluster-api/pull/11397).
76
+
77
+
### Paradigm 2: Centralized Cluster Management
78
+
This paradigm manages multiple Kubernetes clusters using a shared, centralized suite of CAPI controllers. It is designed for scenarios with less stringent isolation requirements.
- Operates under simplified constraints compared to [Paradigm 1](#paradigm-1-isolated-cluster-management).
83
+
- Reduces management overhead through centralization.
84
+
- Prioritizes ease of use and scalability over strict isolation.
76
85
77
-
The proposed PRs implementing such a namespace separation:
78
-
*https://github.com/kubernetes-sigs/cluster-api/pull/11397 extend the commandline option `--namespace=<ns1, …>`
79
-
*https://github.com/kubernetes-sigs/cluster-api/pull/11370 add the new commandline option `--excluded-namespace=<ns1, …>`
86
+
The addition of the new command-line option `--excluded-namespace=<ns1, …>` is proposed in this PR [#11370](https://github.com/kubernetes-sigs/cluster-api/pull/11370).
87
+
88
+
### Challenge: Coexistence of Both Paradigms
89
+
To enable [Paradigm 1](#paradigm-1-isolated-cluster-management) and [Paradigm 2](#paradigm-2-centralized-cluster-management) to coexist within the same management cluster, the following is required:
90
+
91
+
-**Scope Restriction**: [Paradigm 2](#paradigm-2-centralized-cluster-management) must have the ability to restrict its scope to avoid interference with resources owned by [Paradigm 1](#paradigm-1-isolated-cluster-management).
92
+
-**Resource Segregation**: [Paradigm 2](#paradigm-2-centralized-cluster-management) must be unaware of CAPI resources managed by [Paradigm 1](#paradigm-1-isolated-cluster-management) to prevent cross-contamination and conflicts.
93
+
94
+
This coexistence strategy ensures both paradigms can fulfill their respective use cases without compromising operational integrity.
80
95
81
96
## Motivation
82
-
Our motivation is to have a provisioning cluster which is provisioned cluster at the same time while using hierarchical structure of clusters.
83
-
Two namespaces are used by management cluster and the rest of namespaces are watched by CAPI manager to manage other managed clusters.
97
+
For multi-tenant environment a cluster is used as provision-er using different CAPI providers using CAPI requires careful consideration of namespace isolation
98
+
to maintain security and operational boundaries between tenants. In such setups, it is essential to configure the CAPI controller instances
99
+
to either watch or exclude specific groups of namespaces based on the isolation requirements.
100
+
This can be achieved by setting up namespace-scoped controllers or applying filters, such as label selectors, to define the namespaces each instance should monitor.
101
+
By doing so, administrators can ensure that the activities of one tenant do not interfere with others, while also reducing the resource overhead by limiting the scope of CAPI operations.
102
+
This approach enhances scalability, security, and manageability, making it well-suited for environments with strict multi-tenancy requirements.
103
+
Our motivation is to have a provisioning cluster that also serves as a provisioned cluster, leveraging a hierarchical structure of clusters.
104
+
Two namespaces are used by the management cluster, while the remaining namespaces are monitored by the CAPI manager to oversee other managed clusters.
105
+
106
+
References (related issues):
84
107
85
-
Our enhancement is also widely required many times from the CAPI community:
We need to extend the existing feature to limit watching on specified namespace.
91
-
We need to run multiple CAPI controller instances:
92
-
- each watching only specified namespaces: `capi1-system`, …, `capi$(N-1)-system`
93
-
- and the last resort instance to watch the rest of namespaces excluding the namespaces already watched by previously mentioned instances
94
-
95
-
This change is only a small and strait forward update of the existing feature to limit watching on specified namespace by commandline `--namespace <ns>`
115
+
There are some restrictions while using multiple providers, see: https://cluster-api.sigs.k8s.io/developer/core/support-multiple-instances
116
+
But we need to:
96
117
118
+
1. extend the existing "cache configuration" feature `--namespace=<ns>` (limit watching a single namespace) to limit watching on multiple namespaces.
119
+
2. add new feature to watch on all namespaces except selected ones.
120
+
3. then we run multiple CAPI controller instances:
121
+
- each watching only specified namespaces: `capi1-system`, …, `capi$(N-1)-system`
122
+
- and the last resort instance to watch the rest of namespaces excluding the namespaces already watched by previously mentioned instances
97
123
98
124
### Non-Goals/Future Work
99
125
Non-goals:
100
-
* it's not necessary to work with the different versions of CRDs, we consider to:
101
-
* use same version of CAPi (the same container image):
126
+
127
+
* it's not necessary to work with the different versions of CRDs, we consider:
128
+
* use the same version of CAPi (the same container image):
102
129
* share the same CRDs
103
130
* the contract and RBAC need to be solved on specific provider (AWS, AZURE, ...)
104
131
105
132
106
133
## Proposal
107
134
We are proposing to:
135
+
108
136
* enable to select multiple namespaces: add `--namespace=<ns1, …>` to extend `--namespace=<ns>` to watch on selected namespaces
109
-
* the code change is only extending an existing hash with one item to multiple items
110
-
* the maintenance complexity shouldn't be extended here
137
+
* the code change involves extending an existing hash to accommodate multiple items.
138
+
*This change is only a small and straightforward update of the existing feature to limit watching on specified namespace. The maintenance complexity shouldn't be extended here
111
139
* add the new commandline option `--excluded-namespace=<ens1, …>` to define list of excluded namespaces
112
140
* the code change is only setting an option `Cache.Options.DefaultFieldSelector` to disable matching with any of specified namespace's names
113
141
* the maintenance complexity shouldn't be extended a lot here
114
142
143
+
Our objectives include:
144
+
145
+
- Each CAPI instance runs in a separate namespace and uses its own service account.
146
+
- can specify namespaces through command-line arguments:
147
+
- to watch - e.g.: `--namespace <ns1> --namespace <ns2>`
148
+
- to exclude from watching - e.g.: `--excluded-namespace <ns1> --excluded-namespace <ns2>`
149
+
- we do not support multiple versions of CAPI
150
+
- all running CAPI-instances:
151
+
- are using the same container image (same version of CAPI)
- NOTE: the web-hooks are pointing from the CRDs into the first instance only
160
+
- cluster roles and managing access:
161
+
- default CAPI deployment define global cluster role:
162
+
- the `ClusterRole/capi-aggregated-manager-role`
163
+
- the `ClusterRoleBinding/capi-manager-rolebinding` to bind the service account `<instance-namespace>:capi-manager` for CAPI instance (e.g. ) to the `ClusterRole`
164
+
- in case of [Paradigm 1](#paradigm-1-isolated-cluster-management) we can define a cluster role per instance and grant access only to namespaces whose will be watched from the given instance
165
+
- in case of [Paradigm 2](#paradigm-2-centralized-cluster-management) we need to have access to all namespaces as defined in default CAPI deployment cluster role.
166
+
167
+
115
168
### A deployment example
116
-
Let's consider an example how to deploy multiple instances:
169
+
Let's consider an example how to deploy multiple instances for the [Paradigm 1+2](#challenge-coexistence-of-both-paradigms)
117
170
118
171
#### Global resources:
119
-
* CRDs (*.cluster.x-k8s.io) - webhooks will point into first instance, e.g.:
172
+
173
+
* CRDs (*.cluster.x-k8s.io) - webhooks will point into the first instance, e.g.:
120
174
```yaml
121
175
spec:
122
176
conversion:
@@ -167,15 +221,24 @@ Let's consider an example how to deploy multiple instances:
167
221
```
168
222
169
223
### User Stories
170
-
We need to deploy two CAPI instances in the same cluster and divide the list of namespaces to assign some well known namespaces to be watched from the first instance and rest of them to assign to the second instace.
224
+
We need to deploy multiple CAPI instances in the same cluster and divide the list of namespaces to assign certain well-known namespaces to be watched from the given instances and define an instance to watch on the rest of them.
225
+
E.g.:
171
226
172
-
#### Story 1 - RedHat Hierarchical deployment using CAPI
227
+
* instance1 (deployed in `capi1-system`) is watching `ns1.1`, `ns1.2`, ... `ns1.n1`
228
+
* instance2 (deployed in `capi2-system`) is watching `ns2.1`, `ns2.2`, ... `ns2.n2`
229
+
* ...
230
+
* last-resort instance (deployed in `capiR-system`) is watching the rest of namespaces
231
+
232
+
#### Story 1 - Isolated Cluster Management
233
+
We need to limit the list of namespaces to watch. It's possible to do this now, but only on one namespace and we need to watch on multiple namespaces by one instance.
234
+
235
+
#### Story 2 - Centralized Cluster Management
236
+
We need to exclude the list of namespaces from watch to reduces management overhead through centralization.
237
+
238
+
#### Story 3 - Hierarchical deployment using CAPI
173
239
Provisioning cluster which is also provisioned cluster at the same time while using hierarchical structure of clusters.
174
-
Two namespaces are used by management cluster and the rest of namespaces are watched by CAPI manager to manage other managed clusters.
240
+
Two namespaces are used by the management cluster, while the remaining namespaces are watched by the CAPI manager to oversee other managed clusters.
0 commit comments