v0.3.0 Release Tracker

[kfswain]

v0.3.0

Following our roadmap. v0.3.0 will begin work to implement prefix cache aware routing, and continue to improve reliability and flexibility of our offerings

Required

• OpenAI API conformance tests #513
• Prefix Cache Aware Proposal #498
• Support setting fallback endpoints in EPP picking #414
• Refactor the vllm specific code to become model server agnostic #383
• Handle response body parsing for both streaming and non-streaming cases #178
• InferenceModel needs custom printer columns #504
• Fine grained handling of metrics scraping failures #495

Should have

• Create env vars for the algorithm's scheduling parameters #447
• Add helm chart to simplify creating an InferencePool + EPP deployment #381

Nice to Have

• Prefix Cache Aware Routing #497
• GW Conformance Testing #500
• Support endpoint subsetting #415

[ahg-g]

Top of mind for me:

• Create env vars for the algorithm's scheduling parameters #447
• Support endpoint subsetting #415
• Support setting fallback endpoints in EPP picking #414
• Add helm chart to simplify creating an InferencePool + EPP deployment #381
• Refactor the vllm specific code to become model server agnostic #383
• Fine grained handling of metrics scraping failures #495
• Handle response body parsing for both streaming and non-streaming cases #178

[kfswain]

SG, added those and some others, will continue to populate

[ahg-g]

I think #447 is likely going to be required, the current values are tuned for H100 and llama 7b, any other accelerator will likely behave differently.

[ahg-g]

Another note: #414 should probably be required, but #415 can be a nice to have.