-
Notifications
You must be signed in to change notification settings - Fork 848
Open
Labels
Description
extensions/test/Libraries/Microsoft.Extensions.AI.Integration.Tests/ChatClientIntegrationTests.cs
Lines 1092 to 1117 in cc328a7
| public virtual async Task GetResponseAsync_StructuredOutput_NonNative() | |
| { | |
| SkipIfNotEnabled(); | |
| var capturedOptions = new List<ChatOptions?>(); | |
| var captureOutputChatClient = ChatClient.AsBuilder() | |
| .Use((messages, options, nextAsync, cancellationToken) => | |
| { | |
| capturedOptions.Add(options); | |
| return nextAsync(messages, options, cancellationToken); | |
| }) | |
| .Build(); | |
| var response = await captureOutputChatClient.GetResponseAsync<Person>(""" | |
| Supply an object to represent Jimbo Smith from Cardiff. | |
| """, useJsonSchemaResponseFormat: false); | |
| Assert.Equal("Jimbo Smith", response.Result.FullName); | |
| Assert.Contains("Cardiff", response.Result.HomeTown); | |
| // Verify it used *non-native* structured output, i.e., response format Json with no schema | |
| var responseFormat = Assert.IsType<ChatResponseFormatJson>(Assert.Single(capturedOptions)!.ResponseFormat); | |
| Assert.Null(responseFormat.Schema); | |
| Assert.Null(responseFormat.SchemaName); | |
| Assert.Null(responseFormat.SchemaDescription); | |
| } |
Microsoft.Extensions.AI.OpenAIAssistantChatClientIntegrationTests.GetResponseAsync_StructuredOutput_NonNative will occasionally fail with the following exception:
Microsoft.Extensions.AI.OpenAIAssistantChatClientIntegrationTests.GetResponseAsync_StructuredOutput_NonNative [FAIL]
System.Text.Json.JsonException : '`' is an invalid start of a value. Path: $ | LineNumber: 0 | BytePositionInLine: 0.
---- System.Text.Json.JsonReaderException : '`' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
Stack Trace:
at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)
at System.Text.Json.JsonSerializer.GetReaderScopedToNextValue(Utf8JsonReader& reader, ReadStack& state)
at System.Text.Json.JsonSerializer.Read[TValue](Utf8JsonReader& reader, JsonTypeInfo`1 jsonTypeInfo)
at System.Text.Json.JsonSerializer.Deserialize[TValue](Utf8JsonReader& reader, JsonTypeInfo`1 jsonTypeInfo)
C:\src\dotnet\extensions\src\Libraries\Microsoft.Extensions.AI\ChatCompletion\ChatResponse{T}.cs(107,0): at Microsoft.Extensions.AI.ChatResponse`1.DeserializeFirstTopLevelObject(String json, JsonTypeInfo`1 typeInfo)
C:\src\dotnet\extensions\src\Libraries\Microsoft.Extensions.AI\ChatCompletion\ChatResponse{T}.cs(124,0): at Microsoft.Extensions.AI.ChatResponse`1.GetResultCore(Nullable`1& failureReason)
C:\src\dotnet\extensions\src\Libraries\Microsoft.Extensions.AI\ChatCompletion\ChatResponse{T}.cs(60,0): at Microsoft.Extensions.AI.ChatResponse`1.get_Result()
C:\src\dotnet\extensions\test\Libraries\Microsoft.Extensions.AI.Integration.Tests\ChatClientIntegrationTests.cs(1109,0): at Microsoft.Extensions.AI.ChatClientIntegrationTests.GetResponseAsync_StructuredOutput_NonNative()
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
at System.Text.Json.Utf8JsonReader.Read()
at System.Text.Json.JsonSerializer.GetReaderScopedToNextValue(Utf8JsonReader& reader, ReadStack& state)
I can make it fail about 2 out of 5 times. When it fails I debugged to see what the model returned. It looks like it returned a markdown block instead of raw JSON. The response was the following.
```json
{
"fullName": "Jimbo Smith",
"ageInYears": 34,
"homeTown": "Cardiff",
"job": "Programmer"
}
```