Skip to content

Commit e6a8478

Browse files
marek-vebermarek-veber
committed
first namespace-separation-proposal version
1 parent 2016b0d commit e6a8478

File tree

1 file changed

+297
-0
lines changed

1 file changed

+297
-0
lines changed

‎docs/proposals/20241127-namespace-separation.md

+297
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,297 @@
1+
---
2+
title: Multiple Instances of The Same Provider
3+
authors:
4+
- "@marek-veber"
5+
reviewers:
6+
- "@marek-veber"
7+
creation-date: 2024-11-27
8+
last-updated: 2025-01-15
9+
status: provisional
10+
see-also:
11+
replaces:
12+
superseded-by:
13+
---
14+
15+
# Support running multiple instances of the same provider, each one watching different namespaces
16+
17+
## Table of Contents
18+
19+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
20+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
21+
22+
- [Glossary](#glossary)
23+
- [Summary](#summary)
24+
- [Motivation](#motivation)
25+
- [Goals](#goals)
26+
- [Non-Goals/Future Work](#non-goalsfuture-work)
27+
- [Proposal](#proposal)
28+
- [User Stories](#user-stories)
29+
- [Story 1](#story-1)
30+
- [Story 2](#story-2)
31+
- [Requirements (Optional)](#requirements-optional)
32+
- [Functional Requirements](#functional-requirements)
33+
- [FR1](#fr1)
34+
- [FR2](#fr2)
35+
- [Non-Functional Requirements](#non-functional-requirements)
36+
- [NFR1](#nfr1)
37+
- [NFR2](#nfr2)
38+
- [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
39+
- [Security Model](#security-model)
40+
- [Risks and Mitigations](#risks-and-mitigations)
41+
- [Alternatives](#alternatives)
42+
- [Upgrade Strategy](#upgrade-strategy)
43+
- [Additional Details](#additional-details)
44+
- [Test Plan [optional]](#test-plan-optional)
45+
- [Graduation Criteria [optional]](#graduation-criteria-optional)
46+
- [Version Skew Strategy [optional]](#version-skew-strategy-optional)
47+
- [Implementation History](#implementation-history)
48+
49+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
50+
51+
## Summary
52+
We need to run multiple CAPI instances in one cluster and divide the namespaces to be watched by given instances.
53+
54+
We want and consider:
55+
- each CAPI instance:
56+
- is running in separate namespace and is using its own service account
57+
- can select by command the line arguments the list of namespaces:
58+
- to watch - e.g.: `--namespace <ns1> --namespace <ns2>`
59+
- to exclude from watching - e.g.: `--excluded-namespace <ns1> --excluded-namespace <ns2>`
60+
- we are not supporting multiple versions of CAPI
61+
- all running CAPI-instances:
62+
- are using the same container image (same version of CAPI)
63+
- are sharing global resources:
64+
- CRDs:
65+
- cluster.x-k8s.io:
66+
- addons.cluster.x-k8s.io: clusterresourcesetbindings, clusterresourcesets
67+
- cluster.x-k8s.io: clusterclasses, clusters, machinedeployments, machinehealthchecks, machinepools, machinesets, machines
68+
- ipam.cluster.x-k8s.io: ipaddressclaims, ipaddresses
69+
- runtime.cluster.x-k8s.io: extensionconfigs
70+
- NOTE: the web-hooks are pointing from the CRDs into the first instance only
71+
- the `ClusterRole/capi-aggregated-manager-role`
72+
- the `ClusterRoleBinding/capi-manager-rolebinding` to bind all service accounts for CAPI instances (e.g. `capi1-system:capi-manager`, ..., `capiN-system:capi-manager`) to the `ClusterRole`
73+
74+
References:
75+
* https://cluster-api.sigs.k8s.io/developer/core/support-multiple-instances
76+
77+
The proposed PRs implementing such a namespace separation:
78+
* https://github.com/kubernetes-sigs/cluster-api/pull/11397 extend the commandline option `--namespace=<ns1, …>`
79+
* https://github.com/kubernetes-sigs/cluster-api/pull/11370 add the new commandline option `--excluded-namespace=<ns1, …>`
80+
81+
## Motivation
82+
Our motivation is to have a provisioning cluster which is provisioned cluster at the same time while using hierarchical structure of clusters.
83+
Two namespaces are used by management cluster and the rest of namespaces are watched by CAPI manager to manage other managed clusters.
84+
85+
Our enhancement is also widely required many times from the CAPI community:
86+
* https://github.com/kubernetes-sigs/cluster-api/issues/11192
87+
* https://github.com/kubernetes-sigs/cluster-api/issues/11193
88+
89+
### Goals
90+
We need to extend the existing feature to limit watching on specified namespace.
91+
We need to run multiple CAPI controller instances:
92+
- each watching only specified namespaces: `capi1-system`, …, `capi$(N-1)-system`
93+
- and the last resort instance to watch the rest of namespaces excluding the namespaces already watched by previously mentioned instances
94+
95+
This change is only a small and strait forward update of the existing feature to limit watching on specified namespace by commandline `--namespace <ns>`
96+
97+
98+
### Non-Goals/Future Work
99+
Non-goals:
100+
* it's not necessary to work with the different versions of CRDs, we consider to:
101+
* use same version of CAPi (the same container image):
102+
* share the same CRDs
103+
* the contract and RBAC need to be solved on specific provider (AWS, AZURE, ...)
104+
105+
106+
## Proposal
107+
We are proposing to:
108+
* enable to select multiple namespaces: add `--namespace=<ns1, …>` to extend `--namespace=<ns>` to watch on selected namespaces
109+
* the code change is only extending an existing hash with one item to multiple items
110+
* the maintenance complexity shouldn't be extended here
111+
* add the new commandline option `--excluded-namespace=<ens1, …>` to define list of excluded namespaces
112+
* the code change is only setting an option `Cache.Options.DefaultFieldSelector` to disable matching with any of specified namespace's names
113+
* the maintenance complexity shouldn't be extended a lot here
114+
115+
### A deployment example
116+
Let's consider an example how to deploy multiple instances:
117+
118+
#### Global resources:
119+
* CRDs (*.cluster.x-k8s.io) - webhooks will point into first instance, e.g.:
120+
```yaml
121+
spec:
122+
conversion:
123+
strategy: Webhook
124+
webhook:
125+
clientConfig:
126+
service:
127+
name: capi-webhook-service
128+
namespace: capi1-system
129+
path: /convert
130+
```
131+
* `ClusterRole/capi-aggregated-manager-role`
132+
* the `ClusterRoleBinding/capi-manager-rolebinding` binding both service accounts `capi1-system:capi-manager` and `capi2-system:capi-manager` to the cluster role
133+
```yaml
134+
subjects:
135+
- kind: ServiceAccount
136+
name: capi-manager
137+
namespace: capi2-system
138+
- kind: ServiceAccount
139+
name: capi-manager
140+
namespace: capi1-system
141+
```
142+
143+
#### Namespace `capi1-system`
144+
* `ServiceAccount/capi-manager`
145+
* `Role/capi-leader-election-role`, `RoleBinding/capi-leader-election-rolebinding`
146+
* `MutatingWebhookConfiguration/capi-mutating-webhook-configuration`, `ValidatingWebhookConfiguration/capi-validating-webhook-configuration`
147+
* `Service/capi-webhook-service`
148+
* `Deployment/capi-controller-manager` with the extra args:
149+
```yaml
150+
containers:
151+
- args:
152+
- --namespace=rosa-a
153+
- --namespace=rosa-b
154+
```
155+
156+
#### Namespace `capi2-system`
157+
* `ServiceAccount/capi-manager`
158+
* `Role/capi-leader-election-role`, `RoleBinding/capi-leader-election-rolebinding`
159+
* `MutatingWebhookConfiguration/capi-mutating-webhook-configuration`, `ValidatingWebhookConfiguration/capi-validating-webhook-configuration`
160+
* `Service/capi-webhook-service`
161+
* `Deployment/capi-controller-manager` with the extra args:
162+
```yaml
163+
containers:
164+
- args:
165+
- --excluded-namespace=rosa-a
166+
- --excluded-namespace=rosa-b
167+
```
168+
169+
### User Stories
170+
We need to deploy two CAPI instances in the same cluster and divide the list of namespaces to assign some well known namespaces to be watched from the first instance and rest of them to assign to the second instace.
171+
172+
#### Story 1 - RedHat Hierarchical deployment using CAPI
173+
Provisioning cluster which is also provisioned cluster at the same time while using hierarchical structure of clusters.
174+
Two namespaces are used by management cluster and the rest of namespaces are watched by CAPI manager to manage other managed clusters.
175+
176+
RedHat Jira Issues:
177+
* [ACM-15441](https://issues.redhat.com/browse/ACM-15441) - CAPI required enabling option for watching multiple namespaces,
178+
* [ACM-14973](https://issues.redhat.com/browse/ACM-14973) - CAPI controllers should enabling option to ignore namespaces
179+
180+
181+
#### Functional Requirements
182+
183+
##### FR1 - watch multiple namespaces
184+
* It's possible to select a namespace using `--namespace <ns>` to select the namespace to watch now.
185+
* We need to select multiple namespaces using `--namespace <ns1>` ... `--namespace <nsN>` to watch multiple namespaces.
186+
187+
##### FR2 - watch on all namespaces excluding multiple namespaces
188+
* We need to selected excluded namespaces using `--excluded-namespace <ens1>` ... `--namespace <ensN>` to watch on all namespaces excluding selected namespaces.
189+
190+
#### Non-Functional Requirements
191+
192+
We consider that all CAPI instances:
193+
- are using the same container image
194+
- are sharing CRDs and ClusterRole
195+
196+
### Implementation Details/Notes/Constraints
197+
198+
There is a hash `watchNamespaces` of `cache.Config{}` objects [here](https://github.com/kubernetes-sigs/cluster-api/pull/11397/files#diff-2873f79a86c0d8b3335cd7731b0ecf7dd4301eb19a82ef7a1cba7589b5252261L319-R319):
199+
```go
200+
var watchNamespaces map[string]cache.Config
201+
...
202+
ctrlOptions := ctrl.Options{
203+
...
204+
Cache: cache.Options{
205+
DefaultNamespaces: watchNamespaces,
206+
....
207+
}
208+
```
209+
210+
#### Current state:
211+
* when `watchNamespaces` == `nil` then all namespaces are watched
212+
* when `watchNamespaces` == `map[string]cache.Config{ watchNamespace: {} }` then only namespace specified by `watchNamespace` is watched
213+
214+
#### Watch on multiple namespaces
215+
We are suggesting to [update](https://github.com/kubernetes-sigs/cluster-api/pull/11397/files#diff-2873f79a86c0d8b3335cd7731b0ecf7dd4301eb19a82ef7a1cba7589b5252261R320-R323) `watchNamespace` to a `watchNamespacesList` list:
216+
```go
217+
if watchNamespacesList != nil {
218+
watchNamespaces = map[string]cache.Config{}
219+
for _, watchNamespace := range watchNamespacesList {
220+
watchNamespaces[watchNamespace] = cache.Config{}
221+
}
222+
}
223+
```
224+
then the namespaces contained in `watchNamespacesList` will be watched by the instance.
225+
226+
#### Exclude watching on selected namespaces
227+
1. We are suggesting to [create the fieldSelector condition](https://github.com/kubernetes-sigs/cluster-api/pull/11370/files#diff-2873f79a86c0d8b3335cd7731b0ecf7dd4301eb19a82ef7a1cba7589b5252261R331-R339) matching all namespaces excluding the list defined in `watchExcludedNamespaces`:
228+
```go
229+
var fieldSelector fields.Selector
230+
if watchExcludedNamespaces != nil {
231+
var conditions []fields.Selector
232+
for i := range watchExcludedNamespaces {
233+
conditions = append(conditions, fields.OneTermNotEqualSelector("metadata.namespace", watchExcludedNamespaces[i]))
234+
}
235+
fieldSelector = fields.AndSelectors(conditions...)
236+
}
237+
```
238+
2. Then we can use the `fieldSelector` to set the `DefaultFieldSelector` value:
239+
```go
240+
ctrlOptions := ctrl.Options{
241+
...
242+
Cache: cache.Options{
243+
DefaultFieldSelector: fieldSelector,
244+
....
245+
}
246+
```
247+
and then the namespaces contained in `watchNamespacesList` will be excluded from watching by the instance.
248+
249+
250+
### Security Model
251+
252+
A service account will be created for each namespace with CAPI instance.
253+
In the simple deployment example we are considering that all CAPI-instances will share the one cluster role `capi-aggregated-manager-role` so all CAPI's service accounts will be bound using then cluster role binding `capi-manager-rolebinding`.
254+
We can also use multiple cluster roles and grant the access more granular only to the namespaces watched by the instance.
255+
256+
### Risks and Mitigations
257+
258+
259+
- What are the risks of this proposal and how do we mitigate? Think broadly.
260+
- How will UX be reviewed and by whom?
261+
- How will security be reviewed and by whom?
262+
- Consider including folks that also work outside the SIG or subproject.
263+
264+
## Alternatives
265+
266+
The `Alternatives` section is used to highlight and record other possible approaches to delivering the value proposed by a proposal.
267+
268+
## Upgrade Strategy
269+
270+
We don't expect any changes while upgrading.
271+
272+
## Additional Details
273+
274+
### Test Plan
275+
276+
Expectations:
277+
* create only E2E testcases using kind
278+
* deploy two instances into `capi1-system` and `capi2-system` namespaces
279+
* crate three namespaces `watch1`, `watch2`, `watch3` for the watching
280+
* test correct watching:
281+
* watch on all namespace (without `--namespace` and `--excluded-namespace`)
282+
* watch on namespaces `watch1`, `watch2` (only events in `watch1` and `watch1` no events from `watch3` are accepted):
283+
* using `--namespace watch1` and `--namespace watch2`
284+
* using `--excluded-namespace watch3`
285+
286+
287+
## Implementation History
288+
289+
<!--
290+
- [ ] MM/DD/YYYY: Proposed idea in an issue or [community meeting]
291+
- [ ] MM/DD/YYYY: Compile a Google Doc following the CAEP template (link here)
292+
- [ ] MM/DD/YYYY: First round of feedback from community
293+
- [ ] MM/DD/YYYY: Present proposal at a [community meeting]
294+
- [ ] MM/DD/YYYY: Open proposal PR
295+
296+
[community meeting](https://www.youtube.com/watch?v=COcM5bpbusI)
297+
-->

0 commit comments

Comments
 (0)