Loadbalancer IP won't change after deployment

[evilhamsterman]

If you deploy a LoadBalancer without a loadBalancerIP it properly assigns an IP from the range defined. But if you later add a loadBalancerIP it registers an event that looks like it is changing but it never actually does.

Vice versa if you deploy with a loadBalancerIP and then later remove it again appears to register an event that the IP is changing but never actually changes it.

For example here I try deploying the coredns chart set to use a LoadBalancer service type but I add and remove the loadBalancerIP option (cleaned extra data for brevity)

❯ helm upgrade --install -f values.yaml coredns coredns/coredns
☸ starfleet (default) in homelab/k8s/coredns on  main [?]
❯ k get svc
NAME              TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)        AGE
coredns-coredns   LoadBalancer   10.43.33.1   192.168.2.100   53:31509/UDP   4s
kubernetes        ClusterIP      10.43.0.1    <none>          443/TCP        27h
☸ starfleet (default) in homelab/k8s/coredns on  main [?]
❯ helm upgrade --install -f values.yaml coredns coredns/coredns
☸ starfleet (default) in homelab/k8s/coredns on  main [?]
❯ k get svc
NAME              TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)        AGE
coredns-coredns   LoadBalancer   10.43.33.1   192.168.2.100   53:31509/UDP   46s
kubernetes        ClusterIP      10.43.0.1    <none>          443/TCP        27h
☸ starfleet (default) in homelab/k8s/coredns on  main [?]
❯ k describe svc coredns-coredns
Name:                     coredns-coredns
Namespace:                default
Labels:                   app.kubernetes.io/instance=coredns
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=coredns
                          helm.sh/chart=coredns-1.16.7
                          implementation=kube-vip
                          ipam-address=192.168.2.100
Annotations:              meta.helm.sh/release-name: coredns
                          meta.helm.sh/release-namespace: default
Selector:                 app.kubernetes.io/instance=coredns,app.kubernetes.io/name=coredns
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.33.1
IPs:                      10.43.33.1
IP:                       192.168.2.2
LoadBalancer Ingress:     192.168.2.100
Port:                     udp-53  53/UDP
TargetPort:               53/UDP
NodePort:                 udp-53  31509/UDP
Endpoints:                10.42.0.16:53,10.42.1.73:53,10.42.2.146:53
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                   Age               From                Message
  ----     ------                   ----              ----                -------
  Normal   LoadbalancerIP           49s               service-controller  -> 192.168.2.100
  Normal   EnsuringLoadBalancer     5s (x3 over 49s)  service-controller  Ensuring load balancer
  Warning  UnAvailableLoadBalancer  5s (x3 over 49s)  service-controller  There are no available nodes for LoadBalancer
  Normal   EnsuredLoadBalancer      5s (x3 over 49s)  service-controller  Ensured load balancer
  Normal   LoadbalancerIP           5s                service-controller  192.168.2.100 -> 192.168.2.2
☸ starfleet (default) in homelab/k8s/coredns on  main [?]
❯ helm upgrade --install -f values.yaml coredns coredns/coredns
☸ starfleet (default) in homelab/k8s/coredns on  main [?]
❯ k describe svc coredns-coredns
Name:                     coredns-coredns
Namespace:                default
Labels:                   app.kubernetes.io/instance=coredns
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=coredns
                          helm.sh/chart=coredns-1.16.7
                          implementation=kube-vip
                          ipam-address=192.168.2.101
Annotations:              meta.helm.sh/release-name: coredns
                          meta.helm.sh/release-namespace: default
Selector:                 app.kubernetes.io/instance=coredns,app.kubernetes.io/name=coredns
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.33.1
IPs:                      10.43.33.1
IP:                       192.168.2.101
LoadBalancer Ingress:     192.168.2.100
Port:                     udp-53  53/UDP
TargetPort:               53/UDP
NodePort:                 udp-53  31509/UDP
Endpoints:                10.42.0.16:53,10.42.1.73:53,10.42.2.146:53
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                   Age                 From                Message
  ----     ------                   ----                ----                -------
  Normal   LoadbalancerIP           2m23s               service-controller  -> 192.168.2.100
  Normal   LoadbalancerIP           99s                 service-controller  192.168.2.100 -> 192.168.2.2
  Normal   EnsuringLoadBalancer     0s (x5 over 2m23s)  service-controller  Ensuring load balancer
  Warning  UnAvailableLoadBalancer  0s (x5 over 2m23s)  service-controller  There are no available nodes for LoadBalancer
  Normal   EnsuredLoadBalancer      0s (x5 over 2m23s)  service-controller  Ensured load balancer
  Normal   LoadbalancerIP           0s                  service-controller  192.168.2.2 ->
  Normal   LoadbalancerIP           0s                  service-controller  -> 192.168.2.101
☸ starfleet (default) in homelab/k8s/coredns on  main [?]
❯ k get svc
NAME              TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)        AGE
coredns-coredns   LoadBalancer   10.43.33.1   192.168.2.100   53:31509/UDP   2m39s
kubernetes        ClusterIP      10.43.0.1    <none>          443/TCP        27h

Through out that only 192.168.2.100 ever responded

[habib-the-sweet]

This problem is still relevant. Is there any solution?

[sshambar]

I just hit this problem, but I think the bug is actually in kube-vip (not the cloud-provider)...

Looking at kube-vip/pkg/manager/watch_services.go: servicesWatcher() it appears that modifying the svc.Spec.Type to anything but v1.ServiceTypeLoadBalancer is ignored -- but the existing VIP continues to be tracked and maintained.
Subsequently, setting back to LoadBalancer appears to ignored because the svc.UID is already present in the monitoring list...

Deleting and re-creating the service solves the problem, but isn't ideal.

I might log a bug in kube-vip for this...if one doesn't exist already.