🐛 fix: set correct IP address type for machine

[CharlieR-o-o-t]

What this PR does / why we need it:
It's incorrect to always set ExternalIP type for all machine IP addresses. Need to fix that.

Which issue(s) this PR fixes
Fixes issue #947

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: CharlieR-o-o-t / name: Siarhei Rasiukevich (162e9d4)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign vincepri for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Welcome @CharlieR-o-o-t!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-vsphere 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-vsphere has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @CharlieR-o-o-t. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[CharlieR-o-o-t]

/retest

[k8s-ci-robot]

@CharlieR-o-o-t: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[chrischdi]

This does not work like this.

There are places where explicitly MachineExternalIP get's filtered for.

Example:

cluster-api-provider-vsphere/pkg/util/machines.go

Line 69 in 91d6fbb

if machineAddr.Type != clusterv1.MachineExternalIP {

Also we should make sure to not break other potential consumers and this would be a breaking change for them.

Do you know if there is a good definition in upstream documentation of the separation between ExternalIP and InternalIP? This could would rely on the definition of the private subnets.

In general I agree that this should be InternalIP, but we need to be careful.

cc @neolit123 , @rikatz

[chrischdi]

I think a way forward would be to set both for some time and maybe deprecate setting the ExternalIP, maybe via a feature gate so we could go through normal deprecation lifecycle.

/ok-to-test

[neolit123]

Do you know if there is a good definition in upstream documentation of the separation between ExternalIP and InternalIP? This could would rely on the definition of the private subnets.

node internal ip = ip of the node as accessed from the cluster
node external ip = 'real' node ip for external access

/cc @lubronzhan
since i'm not a network expert

[neolit123]

note that cluster local subnets are not may not be the same as local network subnets.

[neolit123]

i don't think that would be the correct way to determine if the ip is internal. it would be an internal network ip, but not necessary the internal cluster ip allocated to a node. to check if the ip is internal as per k8s' definition, i think one would have to check how the controller manager cidr is configured.

e.g. kcm is started with --cluster-cidr=10.244.0.0/16

[CharlieR-o-o-t]

@chrischdi, @neolit123

Hello, thank you for quick reply, agree that we need to be carefull.

About breaking change.
I could be wrong, but I can see that type of IPAddress have no effect on guestinfo.metadata or machine network configuration, it's just status in k8s object, on workload k8s cluster vsphere-cpi will deal with External/Internal IP address types. In vsphere-cpi cloud-config we are able to tell exactly what CIDRs we consider Internal (internalNetworkSubnetCidr) or External (externalNetworkSubnetCidr).

This method is only used to check that we have at least one IP on control-plane bootstrap:

cluster-api-provider-vsphere/pkg/util/machines.go

Line 59 in 91d6fbb

func GetMachinePreferredIPAddress(machine *infrav1.VSphereMachine) (string, error) {

IP output from the function is never used (only in _test.go)

We are using cluster-api-provider-vsphere for long time and only now noticed this issue, when we decided to monitor machines with pubic network.

[lubronzhan]

Do you know if there is a good definition in upstream documentation of the separation between ExternalIP and InternalIP? This could would rely on the definition of the private subnets.

node internal ip = ip of the node as accessed from the cluster node external ip = 'real' node ip for external access

/cc @lubronzhan since i'm not a network expert

Definition from k8s https://github.com/kubernetes/kubernetes/blob/cd060979bfae61630c829d4a88f1fba2c60b3ad1/pkg/apis/core/types.go#L5378-L5389

// NodeInternalIP identifies an IP address which is assigned to one of the node's
	// network interfaces. Every node should have at least one address of this type.
	//
	// An internal IP is normally expected to be reachable from every other node, but
	// may not be visible to hosts outside the cluster. By default it is assumed that
	// kube-apiserver can reach node internal IPs, though it is possible to configure
	// clusters where this is not the case.
	//
	// NodeInternalIP is the default type of node IP, and does not necessarily imply
	// that the IP is ONLY reachable internally. If a node has multiple internal IPs,
	// no specific semantics are assigned to the additional IPs.

The internalIP on the Node.status doesn't always mean it's from the private network cidr, don't think there is a strict limitation in the upstream that where the IP range should be.

The InternalIP is could be from RFC 1918 private IP address ranges:

* 10.0.0.0/8 (10.x.x.x)
* 172.16.0.0/12 (172.16.x.x - 172.31.x.x)
* 192.168.0.0/16 (192.168.x.x)

Still, the the customer could decide what IP range to be used for their internalIP on the node when they configure vsphere-cpi. For example, some lazy customer just choose to use a public IP to pass to CPI.

As for the impact of this change:
Current code change is for MachineInternalIP, which I don't think we use it somewhere in downstream at least. Usually we, or controllers use node's IntermalIP/ExternalIP

For this case, maybe the ideal solution would be, like what vsphere-cpi does, pass a range of IPs as internalMachineRange through cluster object, so this status could use that range to filter what IP to show for MachineInternalIP

[CharlieR-o-o-t]

@lubronzhan @neolit123
Clearly got your point.

I can rework this as it's done in vsphere-cpi. So user will define CIDRs for internal/external.
The only one question - what to do if nothing defined in config for this?

[sbueringer]

The only one question - what to do if nothing defined in config for this?

Usually, keep the behavior the same to not break anyone