Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Groundwork to support OpenAI API endpoints that vLLM supports #526

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kfswain
Copy link
Collaborator

@kfswain kfswain commented Mar 18, 2025

This PR sets up envoy to share request path attributes with EPP, and creates a set of maps to determine which routes we allow, and which are just passthrough.

Optionally, we can make the decision that EPP should do no route enforcement, and then we only map the RoutableEndpoints and assume any other route is just passthrough.

The route list was made from an intersection of the endpoints in:

@k8s-ci-robot k8s-ci-robot requested a review from liu-cong March 18, 2025 18:02
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested a review from robscott March 18, 2025 18:02
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. approved Indicates a PR has been approved by an approver from all required OWNERS files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 18, 2025
Copy link

netlify bot commented Mar 18, 2025

Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit d604a20
🔍 Latest deploy log https://app.netlify.com/sites/gateway-api-inference-extension/deploys/67d9b5e6b4512300088ab14c
😎 Deploy Preview https://deploy-preview-526--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@kfswain kfswain force-pushed the open-ai-compliance branch from 389ef35 to a62a169 Compare March 18, 2025 18:04
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 18, 2025
@kfswain kfswain force-pushed the open-ai-compliance branch from a62a169 to d604a20 Compare March 18, 2025 18:05
@hzxuzhonghu
Copy link
Member

For these passthrough api sendpoints, users should setup policies to tell envoy not send the request to epp at all. It's better to shortcircuit in envoy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants