Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 fix: set correct IP address type for machine #3379

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

CharlieR-o-o-t
Copy link

What this PR does / why we need it:
It's incorrect to always set ExternalIP type for all machine IP addresses. Need to fix that.

Which issue(s) this PR fixes
Fixes issue #947

Copy link

linux-foundation-easycla bot commented Mar 6, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: CharlieR-o-o-t / name: Siarhei Rasiukevich (162e9d4)

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign vincepri for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Mar 6, 2025
@k8s-ci-robot
Copy link
Contributor

Welcome @CharlieR-o-o-t!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-vsphere 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-vsphere has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 6, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @CharlieR-o-o-t. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 6, 2025
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Mar 6, 2025
@CharlieR-o-o-t
Copy link
Author

/retest

@k8s-ci-robot
Copy link
Contributor

@CharlieR-o-o-t: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@CharlieR-o-o-t CharlieR-o-o-t changed the title 🐛 fix: set correct IP address type for machine, #947 🐛 fix: set correct IP address type for machine Mar 6, 2025
Copy link
Member

@chrischdi chrischdi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not work like this.

There are places where explicitly MachineExternalIP get's filtered for.

Example:

if machineAddr.Type != clusterv1.MachineExternalIP {

Also we should make sure to not break other potential consumers and this would be a breaking change for them.

Do you know if there is a good definition in upstream documentation of the separation between ExternalIP and InternalIP? This could would rely on the definition of the private subnets.

In general I agree that this should be InternalIP, but we need to be careful.

cc @neolit123 , @rikatz

@chrischdi
Copy link
Member

I think a way forward would be to set both for some time and maybe deprecate setting the ExternalIP, maybe via a feature gate so we could go through normal deprecation lifecycle.

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 7, 2025
Copy link
Member

@neolit123 neolit123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know if there is a good definition in upstream documentation of the separation between ExternalIP and InternalIP? This could would rely on the definition of the private subnets.

node internal ip = ip of the node as accessed from the cluster
node external ip = 'real' node ip for external access

/cc @lubronzhan
since i'm not a network expert

Comment on lines +269 to +272
privateBlocks := []*net.IPNet{
{IP: net.IPv4(10, 0, 0, 0), Mask: net.CIDRMask(8, 32)}, // 10.0.0.0/8
{IP: net.IPv4(172, 16, 0, 0), Mask: net.CIDRMask(12, 32)}, // 172.16.0.0/12
{IP: net.IPv4(192, 168, 0, 0), Mask: net.CIDRMask(16, 32)}, // 192.168.0.0/16
{IP: net.IPv4(127, 0, 0, 1), Mask: net.CIDRMask(8, 32)}, // 127.0.0.0/8 (loopback)
{IP: net.IPv4(169, 254, 0, 0), Mask: net.CIDRMask(16, 32)}, // 169.254.0.0/16 (link-local)
}
Copy link
Member

@neolit123 neolit123 Mar 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that cluster local subnets are not may not be the same as local network subnets.

Copy link
Member

@neolit123 neolit123 Mar 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think that would be the correct way to determine if the ip is internal. it would be an internal network ip, but not necessary the internal cluster ip allocated to a node. to check if the ip is internal as per k8s' definition, i think one would have to check how the controller manager cidr is configured.

e.g. kcm is started with --cluster-cidr=10.244.0.0/16

@CharlieR-o-o-t
Copy link
Author

CharlieR-o-o-t commented Mar 7, 2025

@chrischdi, @neolit123

Hello, thank you for quick reply, agree that we need to be carefull.

About breaking change.
I could be wrong, but I can see that type of IPAddress have no effect on guestinfo.metadata or machine network configuration, it's just status in k8s object, on workload k8s cluster vsphere-cpi will deal with External/Internal IP address types. In vsphere-cpi cloud-config we are able to tell exactly what CIDRs we consider Internal (internalNetworkSubnetCidr) or External (externalNetworkSubnetCidr).

This method is only used to check that we have at least one IP on control-plane bootstrap:

func GetMachinePreferredIPAddress(machine *infrav1.VSphereMachine) (string, error) {

IP output from the function is never used (only in _test.go)

We are using cluster-api-provider-vsphere for long time and only now noticed this issue, when we decided to monitor machines with pubic network.

@lubronzhan
Copy link
Contributor

lubronzhan commented Mar 7, 2025

Do you know if there is a good definition in upstream documentation of the separation between ExternalIP and InternalIP? This could would rely on the definition of the private subnets.

node internal ip = ip of the node as accessed from the cluster node external ip = 'real' node ip for external access

/cc @lubronzhan since i'm not a network expert

Definition from k8s https://github.com/kubernetes/kubernetes/blob/cd060979bfae61630c829d4a88f1fba2c60b3ad1/pkg/apis/core/types.go#L5378-L5389

// NodeInternalIP identifies an IP address which is assigned to one of the node's
	// network interfaces. Every node should have at least one address of this type.
	//
	// An internal IP is normally expected to be reachable from every other node, but
	// may not be visible to hosts outside the cluster. By default it is assumed that
	// kube-apiserver can reach node internal IPs, though it is possible to configure
	// clusters where this is not the case.
	//
	// NodeInternalIP is the default type of node IP, and does not necessarily imply
	// that the IP is ONLY reachable internally. If a node has multiple internal IPs,
	// no specific semantics are assigned to the additional IPs.

The internalIP on the Node.status doesn't always mean it's from the private network cidr, don't think there is a strict limitation in the upstream that where the IP range should be.

The InternalIP is could be from RFC 1918 private IP address ranges:

* 10.0.0.0/8 (10.x.x.x)
* 172.16.0.0/12 (172.16.x.x - 172.31.x.x)
* 192.168.0.0/16 (192.168.x.x)

Still, the the customer could decide what IP range to be used for their internalIP on the node when they configure vsphere-cpi. For example, some lazy customer just choose to use a public IP to pass to CPI.

As for the impact of this change:
Current code change is for MachineInternalIP, which I don't think we use it somewhere in downstream at least. Usually we, or controllers use node's IntermalIP/ExternalIP

For this case, maybe the ideal solution would be, like what vsphere-cpi does, pass a range of IPs as internalMachineRange through cluster object, so this status could use that range to filter what IP to show for MachineInternalIP

@CharlieR-o-o-t
Copy link
Author

@lubronzhan @neolit123
Clearly got your point.

I can rework this as it's done in vsphere-cpi. So user will define CIDRs for internal/external.
The only one question - what to do if nothing defined in config for this?

@sbueringer
Copy link
Member

sbueringer commented Mar 8, 2025

The only one question - what to do if nothing defined in config for this?

Usually, keep the behavior the same to not break anyone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants