Skip to content

SummarizingChatReducer_Streaming test consistently failing. Omitting the initial user info #7145

@ericstj

Description

@ericstj

I found this when testing the latest (with OpenAI 2.8.0) however I reverted that change and still reproduced. I was also able to repro with 10.1 and 10.0 so it's not a recent regression.

public virtual async Task SummarizingChatReducer_Streaming()
{
SkipIfNotEnabled();
var chatClient = new TestSummarizingChatClient(ChatClient, targetCount: 2, threshold: 0);
List<ChatMessage> messages =
[
new(ChatRole.User, "I'm Bob and I work as a software engineer at a startup."),
new(ChatRole.Assistant, "Nice to meet you, Bob! Working at a startup must be exciting. What kind of software do you develop?"),
new(ChatRole.User, "We build AI-powered tools for education."),
new(ChatRole.Assistant, "That sounds impactful! AI in education has so much potential."),
new(ChatRole.User, "Yes, we focus on personalized learning experiences."),
new(ChatRole.Assistant, "Personalized learning is the future of education!"),
new(ChatRole.User, "Was anyone named in the conversation? Provide their name and job.")
];
StringBuilder sb = new();
await foreach (var chunk in chatClient.GetStreamingResponseAsync(messages))
{
sb.Append(chunk.Text);
}
// The summarizer should have reduced the conversation
Assert.Equal(1, chatClient.SummarizerCallCount);
Assert.NotNull(chatClient.LastSummarizedConversation);
Assert.Equal(3, chatClient.LastSummarizedConversation.Count);
Assert.Collection(chatClient.LastSummarizedConversation,
m =>
{
Assert.Equal(ChatRole.Assistant, m.Role); // Summary
Assert.Contains("Bob", m.Text);
},
m => Assert.StartsWith("Personalized learning", m.Text, StringComparison.Ordinal),
m => Assert.Equal("Was anyone named in the conversation? Provide their name and job.", m.Text));
string responseText = sb.ToString();
Assert.Contains("Bob", responseText);
Assert.True(
responseText.IndexOf("software", StringComparison.OrdinalIgnoreCase) >= 0 ||
responseText.IndexOf("engineer", StringComparison.OrdinalIgnoreCase) >= 0,
$"Expected 'software' or 'engineer' in response: {responseText}");
}

      Microsoft.Extensions.AI.OpenAIAssistantChatClientIntegrationTests.SummarizingChatReducer_Streaming (8s 326ms): Error Message: Assert.Collection() Failure: Item comparison failure
                   ↓ (pos 0)
      Collection: [Personalized learning can really enhance student engagement and outcomes. How do you implement that in your tools?, Personalized learning is the future of education!, Was anyone named in the conversation? Provide their name and job.]
      Error:      Assert.Contains() Failure: Sub-string not found
                  String:    "Personalized learning can really enhance "···
                  Not found: "Bob"
                  Stack Trace:
                    C:\src\dotnet\extensions\test\Libraries\Microsoft.Extensions.AI.Integration.Tests\ChatClientIntegrationTests.cs(1305,0): at Microsoft.Extensions.AI.ChatClientIntegrationTests.<>c.<SummarizingChatReducer_Streaming>b__66_0(ChatMessage m)
      Stack Trace:
         at Microsoft.Extensions.AI.ChatClientIntegrationTests.<SummarizingChatReducer_Streaming>d__66.MoveNext() in C:\src\dotnet\extensions\test\Libraries\Microsoft.Extensions.AI.Integration.Tests\ChatClientIntegrationTests.cs:line 1301
      --- End of stack trace from previous location where exception was thrown ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      --- End of stack trace from previous location where exception was thrown ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      --- End of stack trace from previous location where exception was thrown ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-aiMicrosoft.Extensions.AI librariesuntriaged

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions