Skip to content

Latest commit

 

History

History
173 lines (152 loc) · 6.19 KB

ai-proxy.md

File metadata and controls

173 lines (152 loc) · 6.19 KB
title keywords description
ai-proxy
Apache APISIX
API Gateway
Plugin
ai-proxy
This document contains information about the Apache APISIX ai-proxy Plugin.

Description

The ai-proxy plugin simplifies access to LLM providers and models by defining a standard request format that allows key fields in plugin configuration to be embedded into the request.

Proxying requests to OpenAI is supported now. Other LLM services will be supported soon.

Request Format

OpenAI

  • Chat API
Name Type Required Description
messages Array Yes An array of message objects
messages.role String Yes Role of the message (system, user, assistant)
messages.content String Yes Content of the message

Plugin Attributes

Field Required Type Description
auth Yes Object Authentication configuration
auth.header No Object Authentication headers. Key must match pattern ^[a-zA-Z0-9._-]+$.
auth.query No Object Authentication query parameters. Key must match pattern ^[a-zA-Z0-9._-]+$.
model.provider Yes String Name of the AI service provider (openai).
model.name Yes String Model name to execute.
model.options No Object Key/value settings for the model
override.endpoint No String Override the endpoint of the AI provider
timeout No Integer Timeout in milliseconds for requests to LLM. Range: 1 - 60000. Default: 30000
keepalive No Boolean Enable keepalive for requests to LLM. Default: true
keepalive_timeout No Integer Keepalive timeout in milliseconds for requests to LLM. Minimum: 1000. Default: 60000
keepalive_pool No Integer Keepalive pool size for requests to LLM. Minimum: 1. Default: 30
ssl_verify No Boolean SSL verification for requests to LLM. Default: true

Example usage

Create a route with the ai-proxy plugin like so:

curl "http://127.0.0.1:9180/apisix/admin/routes/1" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/anything",
    "plugins": {
      "ai-proxy": {
        "auth": {
          "header": {
            "Authorization": "Bearer <some-token>"
          }
        },
        "model": {
          "provider": "openai",
          "name": "gpt-4",
          "options": {
            "max_tokens": 512,
            "temperature": 1.0
          }
        }
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "somerandom.com:443": 1
      },
      "scheme": "https",
      "pass_host": "node"
    }
  }'

Upstream node can be any arbitrary value because it won't be contacted.

Now send a request:

curl http://127.0.0.1:9080/anything -i -XPOST  -H 'Content-Type: application/json' -d '{
        "messages": [
            { "role": "system", "content": "You are a mathematician" },
            { "role": "user", "a": 1, "content": "What is 1+1?" }
        ]
    }'

You will receive a response like this:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "The sum of \\(1 + 1\\) is \\(2\\).",
        "role": "assistant"
      }
    }
  ],
  "created": 1723777034,
  "id": "chatcmpl-9whRKFodKl5sGhOgHIjWltdeB8sr7",
  "model": "gpt-4o-2024-05-13",
  "object": "chat.completion",
  "system_fingerprint": "fp_abc28019ad",
  "usage": { "completion_tokens": 15, "prompt_tokens": 23, "total_tokens": 38 }
}

Send request to an OpenAI compatible LLM

Create a route with the ai-proxy plugin with provider set to openai-compatible and the endpoint of the model set to override.endpoint like so:

curl "http://127.0.0.1:9180/apisix/admin/routes/1" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/anything",
    "plugins": {
      "ai-proxy": {
        "auth": {
          "header": {
            "Authorization": "Bearer <some-token>"
          }
        },
        "model": {
          "provider": "openai-compatible",
          "name": "qwen-plus"
        },
        "override": {
          "endpoint": "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions"
        }
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "somerandom.com:443": 1
      },
      "scheme": "https",
      "pass_host": "node"
    }
  }'