AI Extensions - strict schema support for OpenAI models

[danielwinkler]

Description

I noticed that the structured output of OpenAI chat completions did not match the schema reliably after switching from the native openai client to the AI extensions library.

One such example is:

{
  "summary": "....",
  "question": "....",
  "answer": "....",
  "title": "...."
}

sometimes gets returned as

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "summary": "....",
    "question": "....",
    "answer": "....",
    "title": "...."
  },
  "required": [
    "summary",
    "question",
    "answer",
    "title"
  ],
  "additionalProperties": false
}

Investigating the library, a possible reason for this could be in the mapping to OpenAI options:

            else if (options.ResponseFormat is ChatResponseFormatJson jsonFormat)
            {
                result.ResponseFormat = jsonFormat.Schema is { } jsonSchema ?
                    OpenAI.Chat.ChatResponseFormat.CreateJsonSchemaFormat(
                        jsonFormat.SchemaName ?? "json_schema",
                        BinaryData.FromBytes(
                            JsonSerializer.SerializeToUtf8Bytes(jsonSchema, OpenAIJsonContext.Default.JsonElement)),
                        jsonFormat.SchemaDescription) :
                    OpenAI.Chat.ChatResponseFormat.CreateJsonObjectFormat();
            }

namely, the jsonSchemaIsStrict parameter of CreateJsonSchemaFormat receives the default value null, see

<param name="jsonSchemaIsStrict">
    <para>
        Whether to enable strict schema adherence when generating the response. If set to <c>true</c>, the
        model will follow the exact schema defined in <paramref name="jsonSchema"/>.
    </para>
    <para>
        Only a subset of the JSON schema specification is supported when this is set to <c>true</c>. Learn more
        in the
        <see href="https://platform.openai.com/docs/guides/structured-outputs">structured outputs guide</see>.
    </para>
</param> 

public static ChatResponseFormat CreateJsonSchemaFormat(string jsonSchemaFormatName, BinaryData jsonSchema, string jsonSchemaFormatDescription = null, bool? jsonSchemaIsStrict = null)

Reproduction Steps

• Create a record for a schema
• private record ExampleSchema(string summary, string question, string answer, string title)
• Create a schema
• AIJsonUtilities.CreateJsonSchema
• Create options
• ChatResponseFormat.ForJsonSchema
• get a chat completion

Expected behavior

The provided schema should always be returned

Actual behavior

The returned data sometimes doesn't adhere to the schema

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[danielwinkler]

Ah sorry ... just found useNativeJsonSchema, which does the job

[danielwinkler]

Accidentally closed by reading and relying on the documentation of useNativeJsonSchema

However, according to the documentation of OpenAI, the strict flags has to be set, see e.g. https://openai.com/index/introducing-structured-outputs-in-the-api/