Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: PowerVS cluster creation fails with no fixed IP addresses available #1877

Open
Amulyam24 opened this issue Jul 10, 2024 · 3 comments · May be fixed by #2001
Open

CI: PowerVS cluster creation fails with no fixed IP addresses available #1877

Amulyam24 opened this issue Jul 10, 2024 · 3 comments · May be fixed by #2001
Assignees
@Amulyam24
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Milestone
Next

Comments

@Amulyam24
Copy link
Contributor

In CI, we create 2 clusters as a part of the test suite. After the first cluster is created successfully, during the the second cluster deployment, first machine creation fails with

"message":"Build of instance 70daa327-9052-4168-90ea-27677aac45bc aborted: Failed to allocate the network(s) with error No fixed IP addresses available for network: 8b7e4d36-7142-4216-83d1-4a4849c229c0, not rescheduling."

This is due to improper cleanup of VMs created as a part of the first cluster deployment.

{
  "pvmInstances": [
    {
      "addresses": [],
      "creationDate": "2024-07-08T21:46:35.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.427994",
        "reason": "PENDING",
        "status": "PENDING"
      },
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/70daa327-9052-4168-90ea-27677aac45bc",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "memory": 0,
      "networks": [],
      "osType": "",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "",
      "processors": 0,
      "pvmInstanceID": "70daa327-9052-4168-90ea-27677aac45bc",
      "serverName": "capibm-e2e-ri4t2i-control-plane-4dl6k",
      "srcs": [],
      "status": "BUILD",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "spawning",
      "updatedDate": "2024-07-08T21:46:35.000Z",
      "virtualCores": {
        "assigned": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/5fbf4b71-caad-4e49-9541-bff8b6d6af89/networks/capi-net-CzwSz",
          "ip": "192.168.157.165",
          "ipAddress": "192.168.157.165",
          "macAddress": "fa:7b:e2:cc:61:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:36:00.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428057",
        "reason": "rmc_state - inactive",
        "status": "WARNING"
      },
      "hostID": 4,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/5fbf4b71-caad-4e49-9541-bff8b6d6af89",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/5fbf4b71-caad-4e49-9541-bff8b6d6af89/networks/capi-net-CzwSz",
          "ip": "192.168.157.165",
          "ipAddress": "192.168.157.165",
          "macAddress": "fa:7b:e2:cc:61:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Unknown",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "5fbf4b71-caad-4e49-9541-bff8b6d6af89",
      "serverName": "capibm-e2e-jb3h95-md-0-b7kp5-sbcrc",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:36:00.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/936e6efa-05d4-47dd-ba6c-fca24240bc60/networks/capi-net-CzwSz",
          "ip": "192.168.157.163",
          "ipAddress": "192.168.157.163",
          "macAddress": "fa:63:9b:99:6b:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:35:48.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428106",
        "reason": "rmc_state - inactive",
        "status": "WARNING"
      },
      "hostID": 29,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/936e6efa-05d4-47dd-ba6c-fca24240bc60",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/936e6efa-05d4-47dd-ba6c-fca24240bc60/networks/capi-net-CzwSz",
          "ip": "192.168.157.163",
          "ipAddress": "192.168.157.163",
          "macAddress": "fa:63:9b:99:6b:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Linux/CentOS Stream 4.18.0-552.1.1.el8.ppc64l8, 8",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "936e6efa-05d4-47dd-ba6c-fca24240bc60",
      "serverName": "capibm-e2e-jb3h95-md-0-b7kp5-4mf49",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:35:48.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/4d7910d6-abe7-4a17-ad30-ea27b46e987d/networks/capi-net-CzwSz",
          "ip": "192.168.157.162",
          "ipAddress": "192.168.157.162",
          "macAddress": "fa:19:94:45:15:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:25:49.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428142",
        "status": "OK"
      },
      "hostID": 4,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/4d7910d6-abe7-4a17-ad30-ea27b46e987d",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/4d7910d6-abe7-4a17-ad30-ea27b46e987d/networks/capi-net-CzwSz",
          "ip": "192.168.157.162",
          "ipAddress": "192.168.157.162",
          "macAddress": "fa:19:94:45:15:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Linux/CentOS Stream 4.18.0-552.1.1.el8.ppc64l8, 8",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "4d7910d6-abe7-4a17-ad30-ea27b46e987d",
      "serverName": "capibm-e2e-jb3h95-md-0-b7kp5-crx6p",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:25:49.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/316dc448-77ac-4bdb-8342-7605d95b9c9f/networks/capi-net-CzwSz",
          "ip": "192.168.157.164",
          "ipAddress": "192.168.157.164",
          "macAddress": "fa:fa:b2:69:da:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:16:45.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428172",
        "status": "OK"
      },
      "hostID": 29,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/316dc448-77ac-4bdb-8342-7605d95b9c9f",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/316dc448-77ac-4bdb-8342-7605d95b9c9f/networks/capi-net-CzwSz",
          "ip": "192.168.157.164",
          "ipAddress": "192.168.157.164",
          "macAddress": "fa:fa:b2:69:da:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Linux/CentOS Stream 4.18.0-552.1.1.el8.ppc64l8, 8",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "316dc448-77ac-4bdb-8342-7605d95b9c9f",
      "serverName": "capibm-e2e-jb3h95-control-plane-4w5sw",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:16:45.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    }
  ]
}

Job link - https://prow.ppc64le-cloud.cis.ibm.net/view/gs/ppc64le-kubernetes/logs/periodic-capi-provider-ibmcloud-e2e-powervs/1810418740110561280
@mkumatag mkumatag added this to the Next milestone Jul 23, 2024
@mkumatag mkumatag modified the milestones: Next, v0.9.0 Sep 17, 2024
@Amulyam24 Amulyam24 linked a pull request Oct 9, 2024 that will close this issue
Open
@mkumatag
Copy link
Member

mkumatag commented Nov 14, 2024 •
edited
Loading

is this happening because vm deleting is taking some time and not waiting for vm to be disappear?

@mkumatag mkumatag modified the milestones: v0.9.0, Next Nov 22, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 20, 2025
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Amulyam24 Amulyam24

Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Milestone
Next
4 participants
@mkumatag @k8s-ci-robot @Amulyam24 @k8s-triage-robot