✨ Support for BoostrapSelfManagedAddons flag for EKS cluster creation

[jas-nik]

What type of PR is this?
/kind feature

What this PR does / why we need it:

Add flag to support BootstrapSelfManagedAddons to provision Bare EKS cluster without default addons (coreDNS, kube-proxy, aws-vpc-cni)

https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html

By default, EKS installs multiple networking add-ons during cluster creation. This includes the Amazon VPC CNI, CoreDNS, and kube-proxy.

If you’d like to disable the installation of these default networking add-ons, use the parameter below. This may be used for alternate CNIs, such as Cilium. Review the EKS API reference for more information.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist:

• squashed commits
• includes documentation
• includes emojis
• adds unit tests
• adds or updates e2e tests

Release note:

Add flag to support BootstrapSelfManagedAddons to provision Bare EKS cluster without default addons (coreDNS, kube-proxy, aws-vpc-cni)

[k8s-ci-robot]

Hi @jas-nik. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[adriananeci]

/ok-to-test

[jas-nik]

"failed to create new managed VPC: failed to create vpc: The maximum number of VPCs has been reached"

😞

[jas-nik]

/retest

[JonnieDoe]

/lgtm

[k8s-ci-robot]

@JonnieDoe: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[JonnieDoe]

/retest

[jas-nik]

/retest

[nrb]

@jas-nik Sorry about the delay here, and thank you for this contribution.

I've been troubleshooting EKS CI issues, and this behavior is very welcome :)

I do ask that you add some tests for this case. We'll need at least a cluster template and to point to that template in the e2e test config.

[jas-nik]

@nrb Apologies, this fell off my radar. I'll get them added

[damdo]

Hey @jas-nik CAPA doesn't do merge commits, would you be able to rebase instead?
Thanks!

[nrb]

/test pull-cluster-api-provider-aws-test

Hit VPC limit.

[damdo]

/test pull-cluster-api-provider-aws-test

[jas-nik]

@damdo @nrb @richardcase would you be able to help with the VPC limit issue?

[damdo]

/test pull-cluster-api-provider-aws-test

[nrb]

@jas-nik Unfortunately we can't allocate more VPCs right now. The best we can do is retry the tests at off-peak times.

[nrb]

/retest

[richardcase]

@jas-nik @nrb @damdo - are getting the "VPC limit" error when running just the unit tests or the e2e tests? If it's the unit tests, (i.e. via /test pull-cluster-api-provider-aws-test), which i guess is the case from the discussion, then that should be within our control to change as it shouldn't be fitting AWS and this may be coming from out "resource counting" code.

Also, if it is the e2e then potentially we can increase the service limits.

[richardcase]

Are we sure thats the real error? We have a test the checks for maximum number of VPCs:

cluster-api-provider-aws/controllers/awscluster_controller_test.go

Line 569 in dcea94f

g.Expect(err.Error()).To(ContainSubstring("The maximum number of VPCs has been reached"))

I will have a look at the logs in the morning to check.

[richardcase]

Looking at the logs it seems to be multiple issues with tests:

• FAIL: TestDefaultingWebhook (0.59s) (and sub tests)
• FAIL: TestCreateCluster/cluster_create_with_2_subnets (0.00s)
• FAIL: TestCreateIPv6Cluster (0.01s)

Its worth running just these tests locally to see whats going on.

[jas-nik]

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/5336/pull-cluster-api-provider-aws-test/1887504193065848832 - One of the latest PRs build passed even after VPC limit failure, so it might be a red herring after all. Thank you for chiming in. Still need to investigate what is the actual failure.

[jas-nik]

@nrb @richardcase @damdo Looks like the issue was due to the test cases expecting default value for BootstrapSelfManagedAddons flag. Adjusting expectedSpec for them solves the issue but not sure if that is the right approach here. Should I add a default value for the flag or remove the default value all together since AWS already considers default as True.

[nrb]

Should I add a default value for the flag or remove the default value all together since AWS already considers default as True.

I'm fine with the code you've got as-is. Having it explicit in the CAPA side is nice for users to see at a glance.

Overall this looks good to me, just one question.

[adriananeci]

/lgtm

[nrb]

/approve

I'll make a separate issue for the VPC limits; we seem to hit that a lot, so we likely need to up some limit or adjust the test code that Richard linked to.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nrb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nrb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nrb]

Issue for the maximum VPC flake: #5415