Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AI Extensions - strict schema support for OpenAI models #6062

Open
danielwinkler opened this issue Mar 10, 2025 · 2 comments
Open

AI Extensions - strict schema support for OpenAI models #6062

danielwinkler opened this issue Mar 10, 2025 · 2 comments
Labels
area-ai Microsoft.Extensions.AI libraries bug This issue describes a behavior which is not expected - a bug.

Comments

@danielwinkler
Copy link

Description

I noticed that the structured output of OpenAI chat completions did not match the schema reliably after switching from the native openai client to the AI extensions library.

One such example is:

{
  "summary": "....",
  "question": "....",
  "answer": "....",
  "title": "...."
}

sometimes gets returned as

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "summary": "....",
    "question": "....",
    "answer": "....",
    "title": "...."
  },
  "required": [
    "summary",
    "question",
    "answer",
    "title"
  ],
  "additionalProperties": false
}

Investigating the library, a possible reason for this could be in the mapping to OpenAI options:

            else if (options.ResponseFormat is ChatResponseFormatJson jsonFormat)
            {
                result.ResponseFormat = jsonFormat.Schema is { } jsonSchema ?
                    OpenAI.Chat.ChatResponseFormat.CreateJsonSchemaFormat(
                        jsonFormat.SchemaName ?? "json_schema",
                        BinaryData.FromBytes(
                            JsonSerializer.SerializeToUtf8Bytes(jsonSchema, OpenAIJsonContext.Default.JsonElement)),
                        jsonFormat.SchemaDescription) :
                    OpenAI.Chat.ChatResponseFormat.CreateJsonObjectFormat();
            }

namely, the jsonSchemaIsStrict parameter of CreateJsonSchemaFormat receives the default value null, see

<param name="jsonSchemaIsStrict">
    <para>
        Whether to enable strict schema adherence when generating the response. If set to <c>true</c>, the
        model will follow the exact schema defined in <paramref name="jsonSchema"/>.
    </para>
    <para>
        Only a subset of the JSON schema specification is supported when this is set to <c>true</c>. Learn more
        in the
        <see href="https://platform.openai.com/docs/guides/structured-outputs">structured outputs guide</see>.
    </para>
</param> 
public static ChatResponseFormat CreateJsonSchemaFormat(string jsonSchemaFormatName, BinaryData jsonSchema, string jsonSchemaFormatDescription = null, bool? jsonSchemaIsStrict = null)

Reproduction Steps

  • Create a record for a schema
  • private record ExampleSchema(string summary, string question, string answer, string title)
  • Create a schema
  • AIJsonUtilities.CreateJsonSchema
  • Create options
  • ChatResponseFormat.ForJsonSchema
  • get a chat completion

Expected behavior

The provided schema should always be returned

Actual behavior

The returned data sometimes doesn't adhere to the schema

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

@danielwinkler danielwinkler added bug This issue describes a behavior which is not expected - a bug. untriaged labels Mar 10, 2025
@github-actions github-actions bot added the area-ai Microsoft.Extensions.AI libraries label Mar 10, 2025
@danielwinkler
Copy link
Author

danielwinkler commented Mar 10, 2025

Ah sorry ... just found useNativeJsonSchema, which does the job

@danielwinkler
Copy link
Author

Accidentally closed by reading and relying on the documentation of useNativeJsonSchema

However, according to the documentation of OpenAI, the strict flags has to be set, see e.g. https://openai.com/index/introducing-structured-outputs-in-the-api/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-ai Microsoft.Extensions.AI libraries bug This issue describes a behavior which is not expected - a bug.
Projects
None yet
Development

No branches or pull requests

1 participant