🌱 Add KubeVirt support to Tilt dev workflow #11697

johananl · 2025-01-17T11:25:55Z

What this PR does / why we need it:

This PR adds support for local development using KubeVirt as an alternative to CAPD.

This is useful in cases where CAPD can't be used for whatever reason, with one example being developing Ignition-related features (since Ignition runs in early boot and therefore can't be containerized easily).

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): None

/area devtools

sbueringer · 2025-01-21T15:14:25Z

Is there someone who is familiar with KubeVirt that could do a first review?

nunnatsa · 2025-01-22T14:48:33Z

Hi @johananl

Thanks for this PR. As far as I can see, this is based on the KubeVirt sections in the quickstart guide.

I just opened a PR to update it, as it didn't work for me now (I was working when I wrote it...). Maybe you want to take a look?

#11734

fabriziopandini

Thanks for writing this down.
It will be great if we can do a further step and re-use the existig scrpt for creating kind + registry as well as create a new script for automating most of the steps described in this doc

docs/book/src/developer/core/tilt.md

johananl · 2025-01-30T15:17:14Z

@fabriziopandini I've refactored the PR and converted most of the instructions to code. We now have make kind-cluster-kubevirt which does almost everything. I've edited the guide accordingly.

Thanks for the suggestion 👍

mboersma

/lgtm

The changes make sense to me. I tried this out with make kind-cluster-kubevirt and everything seems to work as advertised in Tilt. I did make serve to follow through the docs, and I think they're clear and the tab formatting works fine.

k8s-ci-robot · 2025-01-31T16:47:53Z

LGTM label has been added.

Git tree hash: b986b25002289d3e30b1412e76e01616859efe9d

hack/kind-install-for-capd.sh

sbueringer · 2025-02-04T12:00:20Z

Thx!

/lgtm

/assign @nunnatsa @fabriziopandini

chrischdi

Just some comments on the doc.

Great work on the bash scripts!

docs/book/src/developer/core/tilt.md

hack/kind-install-for-capd.sh

johananl · 2025-02-21T13:13:34Z

I think I've addressed all the feedback. Happy to get a final review 🙏

fabriziopandini

just few nits otherwise lgtm, but I'm not sure when I will have bandwidth to test, hopefully someone could get there before me

docs/book/src/developer/core/tilt.md

hack/kind-install-for-capk.sh

dmvolod · 2025-03-02T17:31:55Z

hack/kind-install-for-capk.sh

+while [[ -z $(kubectl -n metallb-system get pods \
+  -l app=metallb,component=controller -o jsonpath="{.items[0].metadata.name}" 2>/dev/null) ]]; do
+  sleep 2
+done


It seems to waiting for MetalLB Deployment and DaemonSet pods can be a bit simplified with kubectl wait option.

kubectl wait --for condition=available deployment controller -n metallb-system --timeout 150s kubectl wait --for condition=ready pod -l app=metallb,component=speaker -n metallb-system --timeout 150s

I used the loop because with kubectl wait I kept getting errors because the script was trying to check for Ready status on a non-existent pod. We need to allow some time for the deployment or daemon set controller to create the pods.

Good point regarding checking for available status on the deployment rather than the pods. This works. But looks like we don't have an equivalent for daemon sets.

We could do what you suggested and check for available deployment and ready pod, however there will be a race between speaker pod creation and the wait on speaker pod.

I've adopted your suggestion for now to make progress. Maybe we can leave the race be since it's probably quite rare that it would trip someone.

Yes, totally agree about potential race between controller and speaker pods.

Probably, I've found solution for the DaemonSet wait for ready condition:

kubectl wait -n metallb-system daemonsets.apps speaker --for=jsonpath='{.status.numberReady}'=$(kubectl get daemonsets.apps speaker -n metallb-system -ojsonpath='{.status.currentNumberScheduled}'

Thanks, but it doesn't work: When trying to run

kubectl wait pods -n metallb-system -l app=metallb,component=speaker --for condition=Ready --timeout 5m

right after

kubectl wait -n metallb-system daemonsets.apps speaker \ --for jsonpath='{.status.numberReady}'="$(kubectl get daemonsets.apps speaker -n metallb-system -ojsonpath='{.status.currentNumberScheduled}')" --timeout 5m

error: no matching resources found

It's strange, but let's keep the original approach. Thank you.

Actually I've already adopted your suggestion partially, just without waiting for the daemonset (see current state of code). Think we should go back to the sleep loops?

No, adopted part is good. Without waiting for the DaemonSet approach, as it's seems to not working in some cases.

Probably this code should work. I tried to run it locally and it's working for me

kubectl wait -n metallb-system daemonsets.apps -l app=metallb,component=speaker --for=jsonpath='{.status.numberReady}'=$(kubectl get daemonsets.apps -l app=metallb,component=speaker -n metallb-system -ojsonpath='{.items[0].status.currentNumberScheduled}')

It's "working" for me only because by the time the controller deployment has converged the speaker DS has converged, too. I ran

kubectl wait -n metallb-system daemonsets.apps -l app=metallb,component=speaker \ --for=jsonpath='{.status.numberReady}'="$(kubectl get daemonsets.apps -l app=metallb,component=speaker -n metallb-system -ojsonpath='{.items[0].status.currentNumberScheduled}')" kubectl wait pods -n metallb-system -l app=metallb,component=speaker --for condition=Ready --timeout 5m

for testing and got:

error: no matching resources found

Let's keep the current state and if anyone ever complains about the race we switch to sleep.

johananl · 2025-03-07T13:52:19Z

I'd like to merge this ASAP since we depend on this PR for the Node Bootstrapping working group (https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/community/20241112-node-bootstrapping.md). We can't test Ignition functionality using CAPD so we need the CAPK Tilt workflow.

I think we can follow up with fixes or enhancements at any point later on but for now let's have something that works. Also, this is a dev tool rather than a user-facing feature so IMO it's OK if it's not 100% polished.

chrischdi

Last nit, happy to merge afterwards :-)

hack/kind-install.sh

chrischdi

Last nit, happy to merge afterwards :-)

Signed-off-by: Johanan Liebermann <jliebermann@microsoft.com>

johananl · 2025-03-20T09:30:32Z

Thanks @chrischdi. I think we should be good to go.

sbueringer · 2025-03-20T09:35:22Z

/assign

(I want to take a look at the delta since my last lgtm)

chrischdi · 2025-03-20T16:03:48Z

lgtm from my side, pending stefans review :-)

sbueringer · 2025-03-20T17:55:20Z

Thank you!

/lgtm
/approve

k8s-ci-robot · 2025-03-20T17:55:26Z

LGTM label has been added.

Git tree hash: e48a1686713852f0d48b09e29f0462214759ac18

k8s-ci-robot · 2025-03-20T17:55:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the area/devtools label Jan 17, 2025

k8s-ci-robot requested a review from jackfrancis January 17, 2025 11:26

k8s-ci-robot added the cncf-cla: yes label Jan 17, 2025

k8s-ci-robot requested a review from richardcase January 17, 2025 11:26

k8s-ci-robot added the size/L label Jan 17, 2025

johananl changed the title ~~Add KubeVirt instructions to tilt.md~~ 📖 Add KubeVirt instructions to tilt.md Jan 17, 2025

fabriziopandini reviewed Jan 27, 2025

View reviewed changes

docs/book/src/developer/core/tilt.md Outdated Show resolved Hide resolved

docs/book/src/developer/core/tilt.md Outdated Show resolved Hide resolved

johananl force-pushed the tilt-kubevirt branch 9 times, most recently from a560441 to 7bbd562 Compare January 30, 2025 15:01

johananl changed the title ~~📖 Add KubeVirt instructions to tilt.md~~ 🌱 Add KubeVirt support to Tilt dev workflow Jan 30, 2025

mboersma approved these changes Jan 31, 2025

View reviewed changes

k8s-ci-robot assigned mboersma Jan 31, 2025

k8s-ci-robot added the lgtm label Jan 31, 2025

sbueringer reviewed Feb 3, 2025

View reviewed changes

hack/kind-install-for-capd.sh Outdated Show resolved Hide resolved

k8s-ci-robot removed the lgtm label Feb 4, 2025

k8s-ci-robot requested a review from mboersma February 4, 2025 10:51

sbueringer added the tide/merge-method-squash label Feb 4, 2025

k8s-ci-robot assigned fabriziopandini Feb 4, 2025

johananl force-pushed the tilt-kubevirt branch 2 times, most recently from 56bb5e7 to fe92230 Compare February 4, 2025 14:56

chrischdi reviewed Feb 18, 2025

View reviewed changes

docs/book/src/developer/core/tilt.md Outdated Show resolved Hide resolved

docs/book/src/developer/core/tilt.md Outdated Show resolved Hide resolved

docs/book/src/developer/core/tilt.md Outdated Show resolved Hide resolved

docs/book/src/developer/core/tilt.md Show resolved Hide resolved

chrischdi reviewed Feb 18, 2025

View reviewed changes

hack/kind-install-for-capd.sh Outdated Show resolved Hide resolved

johananl force-pushed the tilt-kubevirt branch from fe92230 to eb53bc0 Compare February 21, 2025 13:10

fabriziopandini reviewed Feb 24, 2025

View reviewed changes

dmvolod reviewed Mar 2, 2025

View reviewed changes

johananl force-pushed the tilt-kubevirt branch 2 times, most recently from 2b628c3 to f45295f Compare March 6, 2025 15:36

johananl requested review from fabriziopandini, dmvolod and chrischdi March 6, 2025 15:38

chrischdi reviewed Mar 14, 2025

View reviewed changes

hack/kind-install.sh Show resolved Hide resolved

chrischdi reviewed Mar 14, 2025

View reviewed changes

johananl added 2 commits March 20, 2025 11:28

Update Kind scripts to support KubeVirt

f67f302

Signed-off-by: Johanan Liebermann <jliebermann@microsoft.com>

Add KubeVirt instructions to tilt.md

f73af9d

Signed-off-by: Johanan Liebermann <jliebermann@microsoft.com>

johananl force-pushed the tilt-kubevirt branch from f45295f to f73af9d Compare March 20, 2025 09:28

johananl requested a review from chrischdi March 20, 2025 09:30

k8s-ci-robot added the lgtm label Mar 20, 2025

k8s-ci-robot added the approved label Mar 20, 2025

k8s-ci-robot merged commit 129b395 into kubernetes-sigs:main Mar 20, 2025
19 checks passed

k8s-ci-robot added this to the v1.11 milestone Mar 20, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 Add KubeVirt support to Tilt dev workflow #11697

🌱 Add KubeVirt support to Tilt dev workflow #11697

johananl commented Jan 17, 2025 •

edited

Loading

sbueringer commented Jan 21, 2025

nunnatsa commented Jan 22, 2025 •

edited

Loading

fabriziopandini left a comment

johananl commented Jan 30, 2025

mboersma left a comment •

edited

Loading

k8s-ci-robot commented Jan 31, 2025

sbueringer commented Feb 4, 2025

chrischdi left a comment

johananl commented Feb 21, 2025

fabriziopandini left a comment

dmvolod Mar 2, 2025

johananl Mar 6, 2025 •

edited

Loading

johananl Mar 6, 2025

dmvolod Mar 7, 2025

johananl Mar 7, 2025

dmvolod Mar 7, 2025

johananl Mar 7, 2025

dmvolod Mar 7, 2025

johananl Mar 7, 2025

johananl Mar 7, 2025

johananl commented Mar 7, 2025 •

edited

Loading

chrischdi left a comment

chrischdi left a comment

johananl commented Mar 20, 2025

sbueringer commented Mar 20, 2025 •

edited

Loading

chrischdi commented Mar 20, 2025

sbueringer commented Mar 20, 2025

k8s-ci-robot commented Mar 20, 2025

k8s-ci-robot commented Mar 20, 2025

🌱 Add KubeVirt support to Tilt dev workflow #11697

🌱 Add KubeVirt support to Tilt dev workflow #11697

Conversation

johananl commented Jan 17, 2025 • edited Loading

sbueringer commented Jan 21, 2025

nunnatsa commented Jan 22, 2025 • edited Loading

fabriziopandini left a comment

Choose a reason for hiding this comment

johananl commented Jan 30, 2025

mboersma left a comment • edited Loading

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 31, 2025

sbueringer commented Feb 4, 2025

chrischdi left a comment

Choose a reason for hiding this comment

johananl commented Feb 21, 2025

fabriziopandini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johananl Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johananl commented Mar 7, 2025 • edited Loading

chrischdi left a comment

Choose a reason for hiding this comment

chrischdi left a comment

Choose a reason for hiding this comment

johananl commented Mar 20, 2025

sbueringer commented Mar 20, 2025 • edited Loading

chrischdi commented Mar 20, 2025

sbueringer commented Mar 20, 2025

k8s-ci-robot commented Mar 20, 2025

k8s-ci-robot commented Mar 20, 2025

johananl commented Jan 17, 2025 •

edited

Loading

nunnatsa commented Jan 22, 2025 •

edited

Loading

mboersma left a comment •

edited

Loading

johananl Mar 6, 2025 •

edited

Loading

johananl commented Mar 7, 2025 •

edited

Loading

sbueringer commented Mar 20, 2025 •

edited

Loading