OCPNODE-2982: add a OCP enhancement for Kueue

[kannon92]

No description provided.

[openshift-ci-robot]

@kannon92: This pull request references OCPNODE-2982 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign joepvd for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[varshaprasad96]

Though UX is out - we would have to discuss on how we would like to migrate Distributed Workloads UI which majorly has Kueue metrics. We can keep that as a part of RHOAI, but it probably makes sense to expose those metrics in OCP dashboard instead for standalone installations of Kueue without RHOAI.

[kannon92]

I think you could maybe reuse the same Prometheus Rules that you use for these metrics.

[varshaprasad96]

Yes, but having kueue-metrics on OCP dashboard seems to make sense for users trying to install Kueue in a stand-alone mode without RHOAI. We would have to discuss the role of DW dashboard in the long term.

[kannon92]

I agree but I'm not sure every customer installation would want the same prometheus rules that RHOAI has.

This bullet point was mostly in mind for how to improve Kueue APIs or add easier ways to create kueue resources. We want to call that out of scope for this operator.

Metrics will be exported into OCP dashboard as part of our deployment. What RHOAI chooses to build for prometheus rules or dashboards may need some discussion if its general for all kinds of Kueue installations or tailor made for RHOAI.

[varshaprasad96]

There is going to be a small discrepancy in here - currently in RHOAI Kueue, we restrict the creation of RayClusters without the use of local queue label by default by setting up a VAP that the RHOAI operator applies.

manageJobsWithoutQueueName is still set to false (https://github.com/opendatahub-io/kueue/blob/1c69a687d3ea92eb353c69ccc9c091f400383be9/config/components/manager/controller_manager_config.yaml#L29) as this is to restrict creation of ray clusters alone for now, and then scale it for other resources as we see fit.

The option of managingJobsWithoutQueueName should either be made more granular so that the Kueue operator can instead apply those VAP resources for specific frameworks, or we would have to leave it to RHOAI platforrm team to add this binding to specific namespaces.

[kannon92]

I added this mostly for MLBatch use case.

Personally I think it would be best to not allow this in the API but IBM had examples of using this so I decided to add it in.

[varshaprasad96]

Fair. To enable the feature of restricting workloads that don't have kueue labels, we would probably have to make kueue-operator apply the VAP manifests instead of the RHOAI operator.
cc: @ChristianZaccaria fyi

[kannon92]

We should discuss what kind of flavors you'd want to apply on top of our operator. I tend to think that maybe there is some kind of special operator that RHOAI has for Kueue that can apply VAP for RHOAI. This could also be an option for the work that @KPostOffice is doing for role binding for local queues.

We probably do not want VAP rules for Ray/Kubeflow in our operator as we most likely won't have these APIs in all openshift clusters.

[varshaprasad96]

tend to think that maybe there is some kind of special operator that RHOAI has for Kueue that can apply VAP for RHOAI.

This logic is implemented in the RHOAI operator itself (https://github.com/opendatahub-io/opendatahub-operator/pull/1480/files). What happens currently is that there is check in place to ensure that VAP API is available (in clusters 4.16+) and if so there are manifests applied to cluster that ensure that any ray cluster created has a local queue label.

[ChristianZaccaria]

We probably do not want VAP rules for Ray/Kubeflow in our operator as we most likely won't have these APIs in all openshift clusters.

This shouldn't be an issue. VAP resources can live in the OpenShift cluster without the need of any workload API being present. The policy will exist, and take effect on any incoming request to create a workload such as RayClusters or PyTorchJobs for example. I think having all the policies in place does no harm.

[kannon92]

Its more on who will support/test these VAPs. Openshift does not release Ray/Kubeflow so we would not test these VAPs.

This seems to be more of a flavor of RHOAI versus native core OCP.

VAPs will be in the openshift cluster regardless but it is more on who will actually maintain these and test.

[varshaprasad96]

The other side is - given RHOAI/platform operator would not manage Kueue anymore, having these tests on platform side may not be relevant. We would have to discuss with platform team on where these manifests would live.

[JoelSpeed]

This sounds like RHOAI operator is extending the functionality of the Kueue operator and not something core to Kueue operators goals.

One of the goals of VAP was to allow extending validation of things you don't own, so these extended VAPs that cover multiple components that make up RHOAI definitely seem like a RHOAI concern to me.

[JoelSpeed]

How do we expect resource flavours to work in HCP clusters? Will they be within the workload cluster API rather than the management cluster?

IIUC, autoscaling exists in the management plane, and so, does the autoscaler look at provisioning requests in the HCP workload cluster API, or management cluster API?

[kannon92]

Kueue would create provision requests in the worker cluster. ProvRequests are scoped to the namespace of the workload so I would expect these to be coming from the customer environment.

[kannon92]

I have a section on hypershift below where I mention that this would probably live in a infra node on the customer clusters in HCP cases.

[JoelSpeed]

@elmiko Do you happen to know if the ProvisioningRequest is expected to be in the workload cluster from the perspective of the Cluster Autoscaler, where it distinguishes different clients for workload and management clusters?

[elmiko]

hmm, it would appear that the cluster autoscaler is going to use which kubeconfig is specified in the --kubeconfig flag (or similar mechanism eg env variable or service account). so we might need to rethink how this works if we want to support multiple topologies in the upstream.

for example, users have some discretion with the clusterapi provider as to where they put the clusterapi resources, this would definitely impact how the autoscaler is deployed with respect to provisioningrequests.

[enxebre]

How do we expect resource flavours to work in HCP clusters? Will they be within the workload cluster API rather than the management cluster?
IIUC, autoscaling exists in the management plane, and so, does the autoscaler look at provisioning requests in the HCP workload cluster API, or management cluster API?

Do you happen to know if the ProvisioningRequest is expected to be in the workload cluster from the perspective of the Cluster Autoscaler, where it distinguishes different clients for workload and management clusters?

I see kueue+provioningRequest setup as a Cluster Admin task. From kueue pov it's just an Openshift cluster. I don't foresee anything specific about hcp beyond the aforementioned webhook performance considerations.

[elmiko]

i looked at the cluster-autoscaler code a little more to confirm that provisioing request is using the kubeconfig from the flagged argument, and it appears that it does.

this shouldn't be a problem on openshift as we setup these kubeconfig for the user, but is something that will require some documentation in upstream.

[JoelSpeed]

Ok, so which kubeconfig is that? The same one it uses to fetch pods and nodes, or the one it uses to fetch CAPI resources?

[enxebre]

this is a core cluster autoscaler feature so it must be able to use the guest cluster config, just like Pods and Nodes. Whether we also want to enable the option to watch this management side when in conjunction with capi provider I see as an orthogonal topic

[JoelSpeed]

I wasn't planning for them to be viewed from the management side at present, just wanted to confirm that the CAPI autoscaler expectation is that these are guest cluster side, as that is what is needed for this use case

[JoelSpeed]

How does the support get defined for this bundle? Is there a way to stop the bundle from being upgraded once an alpha feature is used?

[kannon92]

I asked this to @joelanford and his team on ways to block upgrades for operators. I don't think there is really anyway to block an upgrade.

We were thinking that there could be an expermental bundle for operators and a stable stream. Experimental support is limited and users can upgrade within their stream.

But this is something we need to think through the design and I have it in open questions.

[JoelSpeed]

Do we have a timeline for this?

[kannon92]

As with any upstream project, that is dictated by the maintainers.

We know that there may be a new api bump from v1beta1 -> v1beta2 in 2025 timeframe. I think maybe 2026 there are discussions to promote Kueue to v1.
But there are also a lot of existing features in Kueue so there was push back to graduate the APIs to v1.

[JoelSpeed]

Is maintaining this as TP for an extended period going to cause more or less pain for us then GA'ing on the beta APIs and taking the hit of having to lifecycle and support the APIs as they evolve?

[kannon92]

I pushed some options. There was a discussion yesterday where we GA the GAish apis like ClusterQueues, LocalQueues, Workloads etc.

cc @mrunalp

[varshaprasad96]

I believe it would be the responsibility of the cluster admin to be able to assign admin and user roles. Though not ideal, in RHOAI space, this was handled while creating a new project (namespace).

This would again be an implementation detail - but we would have to discuss how the permission model would work out with this migration. This deserves a user story though - in the sense, whenever a new project is created from RHOAI's end (translating to creating a new namespace), the admin should have the option for the user to assign whether they would be a user or an admin.

[kannon92]

This is something we should define and potentially create a RFE once you have a better idea of how you want this.

Maybe a design doc with your requirements here would be helpful.

[varshaprasad96]

Upgrade and support cycle are going to be a part of this enhancement?

[kannon92]

Yes, but that is an open question right now.

We have three phases:

1. Tech Preview release
2. RHOAI integration
3. GA

For 1 it doesn't make sense to have upgrades and support cycles on a tech preview.

[kannon92]

For OCP features we wouldn't allow upgrades for tech preview. The feature set is TechPreviewNoUpgrade actually.

[varshaprasad96]

Fair. To enable the feature of restricting workloads that don't have kueue labels, we would probably have to make kueue-operator apply the VAP manifests instead of the RHOAI operator.
cc: @ChristianZaccaria fyi

[varshaprasad96]

Is there a way to specify only a subset of FGs that can strictly make the cluster non-upgradable, and allow cluster upgrades for others? Is this a pattern that is supported by OCP, and are there any other layered operators that do this?

[varshaprasad96]

Following up from the previous comment - we probably should also chart out release schedule for Kueue operator and map Kueue <-> Kueue operator versions?

[kannon92]

We have three phases for this and right now I am working on getting a tech preview out that would focus on a single kueue version to operator. For a tech preview of an operator we won't necessary handle upgrades (there is no version less than this).

Phase 2 would be RHOAI integration where we figure out versioning and we should have a better idea of how to use Konflux to release Kueue + Kueue Operator. We still need to create file based cataloges and figure out what kind of stream we want our operator to be.

Right now, 0.1 of the operator would be using Kueue 0.10 or 0.11 depending on when we get everything signed off for Kueue. We are still waiting on production tasks to official release Kueue.

[varshaprasad96]

Just curious for my understanding - does OLM v1 currently also create jobs to unpack bundles and install. We frequently were having issues with OLM tending to be stuck during upgrades, as Kueue webhooks stopped them from getting created. There is a fix for this that was added. Just want to know if it would still remain an issue with v1.

[kannon92]

cc @joelanford?

[joelanford]

No, OLMv1 now uses containers/image to directly communicate with the image registry for pulling bundles and catalogs.

[KPostOffice]

waitForPodsReady makes it so that Kueue only considers a workload successfully admitted if all the pods for that workload get to the ready state. Kueue Workloads are enough to ensure gang admission. I find the Kueue documentation here confusing.

[JoelSpeed]

Given you have experience here, could you explain, as an end user, in what situation would I/would I not want this? Any concrete use cases you can talk to?

[kannon92]

In my experience, users usually want workloads to be admitted if pods are ready.

My mental model of Kube:

• schedule
• init containers succeed
• main containers pull
• probes pass
• pod is ready for action

My experience is that users assume that if a pod is scheduled it means that it is ready for action. Scheduling means that a pod has been assigned a node and then it starts running the pod lifecycle. So in most cases a pod is scheduled but it could take a long time to have the pod actually ready.

So most workloads I've seen start to consider "WaitForReady" as a way to enforce that pods are sufficient to run.

We did this in JobSet to make sure that leader jobs are ready before going to the worker jobs.

This doc seems to explain it better well. https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready/

[KPostOffice]

I think @kannon92 can speak more to the concrete use cases. I think the main benefit is the backoff/retry semantics introduced for groups of pods. It can capture cases where pods may never get to the ready state due to some service being offline. In that case, Kueue can deactivate the workload after a certain number of retries so that the capacity being used for the admitted workload can be used for workloads that will progress.

[kannon92]

@KPostOffice Is there any plans to use this right now? Or could we just remove it from operator?

Default this to disabled and leave it as a future addition if there is a need for it?

[JoelSpeed]

Right so IIUC, WaitForPodsReady tells kueue that it must wait for all of the pods it just created to be ready, before it considers the capacity to be used? And it if decides that some of the pods never become ready, it will kill the workloads after ????, and that means it can then schedule something else instead?

Kueue will only admit the workload if all pods are able to be admitted at once.

What does it mean for Kueue to admit a workload, because this doesn't seem to tally with what I've just written above if I use my mental model of admission. From this sentence I expect that Kueue is blocking the pod scheduling until it can see that there's enough capacity for all of the pods in the group to be scheduled simultaneously, which I guess prevents some being scheduled earlier than others and therefore minimises the time they must wait?

It almost sounds like this option is more, wait for capacity, than wait for pods ready? Or am I confusing two different concepts here?

[JoelSpeed]

Reviewing the docs from upstream, it seems really like this is a batch scheduling problem, it's like they want an option of jobScheduling: Parallel | Sequential, where Parallel means that it will just schedule all pods from all jobs at once, and Sequential is where it will wait for the pods ready from a particular job, before moving to the next

The option as it's described currently really focuses on the implementation and not the visible feature for the end user, which is why I think it's hard to understand what it's achieving

[kannon92]

Correct. Essentially the problem is that Kubernetes has not yet added Gang scheduling as a core feature. There is a scheduling plugin called CoScheduling that people have used but plugins are a bit more difficult to support and people don't usually mess with the kube scheduler.

So kueue defined there method to at least have some basic support for gang scheduling by waiting for ready pods.

Sequential is where it will wait for the pods ready from a particular job, before moving to the next

That sounds more like you are thinking of a workflow (ie argo Workflows, Tekton, or Kubeflow pipelines). Kueue is aiming to focus on a single workload and assuming that they should all run together. Kueue has some issues around proper workflow support but they have not made much progress in that avenue.

[JoelSpeed]

Can you define what you mean by workload in this context? I think the term is fairly overloaded and I'm not sure I see the difference between what the two of us are saying

[KPostOffice]

If waitForPodsReady is enabled and Parallel then Kubernetes will admit a Workload[^1], but it also watch the resulting pods and evict and requeue the Workload if the timeout is exceeded. If waitForPodsReady and Sequential is enabled then Kueue will wait until the pods either get to a ready state or surpass their # of retries before moving onto the next Workload in that ClusterQueue

[^1] All references to Workload are a reference to Kueue's Workload CR

[JoelSpeed]

Do you mean all here?

[JoelSpeed]

These are all v1beta1 right now, it's probably worth including that, and also why we are considering these tier 1

[JoelSpeed]

Not even in 4.20 as TP still? Is there a timeline for any of these features? Would be good to distinguish those we intend to support over time vs not at all

[JoelSpeed]

Do we have a section/should we have a section talking about how we will support the migration from the RHOAI version to this version?

[kannon92]

Yea, I think once we figure out how that looks. I gave rhoai team a dev preview of RHOAI to try and figure out the integration plan.

I worry about putting too much details in RHOAI side as I am not an expert or have ever really dived into that.

[JoelSpeed]

Maybe we just need to add something that explains that further investigation is required for the migration specifics will be a future update to the EP? I vaguely recall maybe we have that already?

[JoelSpeed]

Are you concerned about the users installing the alpha APIs themselves? The Gateway API project are preventing other actors from installing CRDs that they consider part of their ecosystem.

For you, that would mean a VAP preventing writes to any Kueue API once your operator is installed, apart from where the write comes from your operator.

That would give you a way in the future to be confident in introducing other Kueue APIs without worry that users have got the old alpha versions already installed?

[kannon92]

I didn't consider that but worst case the feature gates are turned off for these alpha features so the functionality would not be used.

If a user as the ability to install CRDs then they would also have the ability to delete them. They could also delete the deployment..

So I'm not sure if this is worth guarding against.

[kannon92]

To be clear, we are not registered the alpha APIS as a Custom Resource in the APIServer. So APIServer will reject those CRs.

I think that is sufficient in this case. We also disable the feature gates for these features as they are alpha.

[JoelSpeed]

Fair, I think that's a perfectly reasonable way to go, just thought I'd mention the Gateway API example as they went the other way, they're concerned about the upgrade story and want to make sure it's safe to take over management of the CRDs in all OpenShift clusters. As this is an add-on, it is a slightly different approach though

[openshift-ci]

@kannon92: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.