Criticality Discussion/ Improvements

[kfswain]

Criticality has been discussed quite a bit, Ex: here and here, as well as in the Inf-GW weekly meeting.

We recognize that this field may be imperfect, but without user feedback its currently difficult to iterate in the proper direction. So to centralize discussion, we are creating this issue.

[shaneutt]

One of my questions is about how Criticality functions during the entire lifecycle of models:

In practice will operators have to update large groups of InferenceModels to accommodate for the criticality of new InferenceModels? For example let's say I have 60 models out there, 10 of them Critical, 30 Default and 20 Sheddable. I have 20 new InferenceModels to deploy which now need to be the only Critical ones. So... do I have to "downgrade" other existing models in order to achieve this? Is this example a reasonable one, and am I understanding correctly?

More essentially: Criticality is a specification of relationships across multiple resources to "rank" them, and any kind of cross-resource relationship in Kubernetes has the potential to introduce complexity. Do we see potential for ways (like the above) in which this could become painful for operators on clusters?

[nirrozenbaum]

I agree with @shaneutt.
I was also trying to understand - what is this field trying to model?
I was expecting to see prioritization of requests according to the requesting user or something equivalent.
what is the use case where this field is used?

[kfswain]

I converted this to a discussion as its been a highly trafficked conversation point. Love all the feedback! I will try to engage within the next few days.

[ahg-g]

My 2 cents: Criticality predefines a few bands of priority, the expectation is that most setups need 2 or 3 classes of priorities, and so what we have now makes it straightforward for what is thought to be the common scenario.

However, the Criticality API is future proof in that we can expand it to allow for finer grained prioritization within each band. Either based on service objectives (latency targets) or explicit priority number.

[kfswain]

This topic was discussed in our OSS meeting last week (Feb 6th). The recording is not up quite yet, but will show up here: https://www.youtube.com/@kubernetes-wg-serving-ue7yb/videos when uploaded. That discussion there, and follow ups, are great context

[mwm5945]

For others looking for the recording of the criticality convo from 2/6/25, here's the link with timestamp --> https://youtu.be/YAgKUa-3ZrY?t=993

I too am interested in request based prioritization, but also agree that it mixing business logic here would not be a great idea. I also am not a huge fan of managing a large number of mappings between client + model + criticality, seems like a headache for platform maintainers.

Would some sort of configurable plugin/service in the scheduler be of interest?

The other thing i thought of is an additional upstream ext_proc service that does this mapping, that is it looks at the request, and re-writes/assigns the model based on some sort of configuration that the EPP would handle normally. I wonder if it could even be finagled
through adding headers normally via HTTPRoute...