-
Notifications
You must be signed in to change notification settings - Fork 848
Open
Labels
Description
I found this when testing the latest (with OpenAI 2.8.0) however I reverted that change and still reproduced. I was also able to repro with 10.1 and 10.0 so it's not a recent regression.
extensions/test/Libraries/Microsoft.Extensions.AI.Integration.Tests/ChatClientIntegrationTests.cs
Lines 1275 to 1317 in cc328a7
| public virtual async Task SummarizingChatReducer_Streaming() | |
| { | |
| SkipIfNotEnabled(); | |
| var chatClient = new TestSummarizingChatClient(ChatClient, targetCount: 2, threshold: 0); | |
| List<ChatMessage> messages = | |
| [ | |
| new(ChatRole.User, "I'm Bob and I work as a software engineer at a startup."), | |
| new(ChatRole.Assistant, "Nice to meet you, Bob! Working at a startup must be exciting. What kind of software do you develop?"), | |
| new(ChatRole.User, "We build AI-powered tools for education."), | |
| new(ChatRole.Assistant, "That sounds impactful! AI in education has so much potential."), | |
| new(ChatRole.User, "Yes, we focus on personalized learning experiences."), | |
| new(ChatRole.Assistant, "Personalized learning is the future of education!"), | |
| new(ChatRole.User, "Was anyone named in the conversation? Provide their name and job.") | |
| ]; | |
| StringBuilder sb = new(); | |
| await foreach (var chunk in chatClient.GetStreamingResponseAsync(messages)) | |
| { | |
| sb.Append(chunk.Text); | |
| } | |
| // The summarizer should have reduced the conversation | |
| Assert.Equal(1, chatClient.SummarizerCallCount); | |
| Assert.NotNull(chatClient.LastSummarizedConversation); | |
| Assert.Equal(3, chatClient.LastSummarizedConversation.Count); | |
| Assert.Collection(chatClient.LastSummarizedConversation, | |
| m => | |
| { | |
| Assert.Equal(ChatRole.Assistant, m.Role); // Summary | |
| Assert.Contains("Bob", m.Text); | |
| }, | |
| m => Assert.StartsWith("Personalized learning", m.Text, StringComparison.Ordinal), | |
| m => Assert.Equal("Was anyone named in the conversation? Provide their name and job.", m.Text)); | |
| string responseText = sb.ToString(); | |
| Assert.Contains("Bob", responseText); | |
| Assert.True( | |
| responseText.IndexOf("software", StringComparison.OrdinalIgnoreCase) >= 0 || | |
| responseText.IndexOf("engineer", StringComparison.OrdinalIgnoreCase) >= 0, | |
| $"Expected 'software' or 'engineer' in response: {responseText}"); | |
| } |
Microsoft.Extensions.AI.OpenAIAssistantChatClientIntegrationTests.SummarizingChatReducer_Streaming (8s 326ms): Error Message: Assert.Collection() Failure: Item comparison failure
↓ (pos 0)
Collection: [Personalized learning can really enhance student engagement and outcomes. How do you implement that in your tools?, Personalized learning is the future of education!, Was anyone named in the conversation? Provide their name and job.]
Error: Assert.Contains() Failure: Sub-string not found
String: "Personalized learning can really enhance "···
Not found: "Bob"
Stack Trace:
C:\src\dotnet\extensions\test\Libraries\Microsoft.Extensions.AI.Integration.Tests\ChatClientIntegrationTests.cs(1305,0): at Microsoft.Extensions.AI.ChatClientIntegrationTests.<>c.<SummarizingChatReducer_Streaming>b__66_0(ChatMessage m)
Stack Trace:
at Microsoft.Extensions.AI.ChatClientIntegrationTests.<SummarizingChatReducer_Streaming>d__66.MoveNext() in C:\src\dotnet\extensions\test\Libraries\Microsoft.Extensions.AI.Integration.Tests\ChatClientIntegrationTests.cs:line 1301
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)