-
Notifications
You must be signed in to change notification settings - Fork 370
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fail to install velero helm chart 3.2.0. Getting error - job failed: BackoffLimitExceeded #550
Comments
We did not touch the release recently. Perhaps the problem is the kubectl image isn't available. |
@jenting Thanks for your response. Were you able to reproduce this issue using above mentioned commands ?
Could you elaborate more on this ? Is there any workaround to specify the kubectl image through configuration ? |
Looks like this issue is coming specifically with velero image versions v1.11.0 & v1.12.0 It is giving the following error.
|
We're also running into this issue and it started at about the same time. We believe that it is due to changes in the We have one cluster that is working and can see
And many others failing consistently with
Overriding the kubectl image digest in the failing clusters to match the digest in the working cluster cause them to start working. We found bitnami/containers#53360 with a comment from yesterday
https://packages.debian.org/source/bullseye/glibc The timing does seem to line up as the last 1.26-debian-11 image was pushed five days ago (last Friday) and we started seeing this problem in builds on Monday. That issue points to https://docs.bitnami.com/tutorials/understand-rolling-tags-containers/. So the default tag used by the velero chart is a rolling tag, not an immutable one.
Which would explain what we're seeing in our clusters since we don't explicitly set that chart value. We're using velero chart version 5.1.0 with default values for image and kubectl. Clusters in both AWS and Azure failing the same way. |
We've confirmed that as the cause of the problem.
The error happens because the upgrade CRDs job copies
And then tries to use it from the velero image.
But the velero 1.20 image is based on debian 11, which isn't compatible with https://github.com/vmware-tanzu/velero/blob/v1.12.0/Dockerfile#L71 That base image changed in velero 1.21 and we think that may be compatible? For now we've just hardcoded the kubectl image tag to |
Fixes BackfOffLimit Exceeded due to libc.so.6: version `GLIBC_2.33' not found, as mentioned upstream at vmware-tanzu/helm-charts#550
Related issue #559, closing it. |
Getting this issue today 14 Mar 2025 with velero 1.15.2 helm chart version 8.5.0
Using: velero-plugin-for-microsoft-azure:v1.11.0
|
What steps did you take and what happened:
This has been working until Feb 19th, 2024.
kubectl create ns velero-320
helm upgrade --debug --install velero vmware-tanzu/velero --version 3.2.0 --namespace velero-320
What did you expect to happen:
We have pinned to version 3.2.0 to avoid breakages and it has been working until yesterday.
The output of the following commands will help us better understand what's going on:
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
helm version
): version.BuildInfo{Version:"v3.12.1", GitCommit:"f32a527a060157990e2aa86bf45010dfb3cc8b8d", GitTreeState:"clean", GoVersion:"go1.20.4"}helm list -n <YOUR NAMESPACE>
):kubectl version
):/etc/os-release
):Mac or LinuxThe text was updated successfully, but these errors were encountered: