Add browser automation recording and tests (#54415)

* Add sample and test for Browser automotion tool

* Record Browser Automation tests.

* Fix

sdk/ai/Azure.AI.Projects.OpenAI/README.md

@@ -37,6 +37,7 @@ Develop Agents using the Azure AI Foundry platform, leveraging an extensive ecos
   - [MCP tool with project connection](#mcp-tool-with-project-connection)
   - [OpenAPI tool](#openapi-tool)
   - [OpenAPI tool with project connection](#openapi-tool-project-connection)
+  - [Browser automation](#browser-automation)
   - [SharePoint tool](#sharepoint-tool)
 - [Tracing](#tracing)
   - [Tracing to Azure Monitor](#tracing-to-azure-monitor)
@@ -1093,6 +1094,79 @@ OpenAIResponse response = await responseClient.CreateResponseAsync(
 Console.WriteLine(response.GetOutputText());
 ```
 
+## Browser automation
+
+Playwright is a Node.js library for browser automation. Microsoft provides the [Azure Playwright workspace](https://learn.microsoft.com/javascript/api/overview/azure/playwright-readme), which can execute Playwright-based tasks triggered by an Agent using the BrowserAutomationAgentTool.
+
+### Create Azure Playwright workspace
+
+1. Deploy an Azure Playwright workspace.
+2. In the **Get started** section, open **2. Set up authentication**.
+3. **Select Service Access Token**, then choose **Generate Token**. **Save the token immediately-once you close the page, it cannot be viewed again.**
+
+### Configure Microsoft Foundry
+
+1. Open the left navigation and select **Management center**.
+2. Choose **Connected resources**.
+3. Create a new connection of type **Serverless Model**.
+4. Provide a name, then paste your Access Token into the **Key** field.
+5. Set the Playwright Workspace Browser endpoint as the **Target URI**. You can find this endpoint on the Workspace **Overview page**. It begins with `wss://`.
+
+Please note that Browser automation operations may take longer than typical calls to process. Using background mode for Responses or applying a network timeout of at least five minutes for non-background calls is highly recommended.
+
+```C# Snippet:Sample_CreateProjectClient_BrowserAutomotion
+var projectEndpoint = System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
+var modelDeploymentName = System.Environment.GetEnvironmentVariable("MODEL_DEPLOYMENT_NAME");
+var playwrightConnectionName = System.Environment.GetEnvironmentVariable("PLAYWRIGHT_CONNECTION_NAME");
+AIProjectClientOptions options = new()
+{
+    NetworkTimeout = TimeSpan.FromMinutes(5)
+};
+AIProjectClient projectClient = new(endpoint: new Uri(projectEndpoint), tokenProvider: new DefaultAzureCredential(), options: options);
+```
+
+To use Azure Playwright workspace we need to create agent with `BrowserAutomationAgentTool`.
+
+```C# Snippet:Sample_CreateAgent_BrowserAutomotion_Async
+AIProjectConnection playwrightConnection = await projectClient.Connections.GetConnectionAsync(playwrightConnectionName);
+BrowserAutomationAgentTool playwrightTool = new(
+    new BrowserAutomationToolParameters(
+        new BrowserAutomationToolConnectionParameters(playwrightConnection.Id)
+    ));
+
+PromptAgentDefinition agentDefinition = new(model: modelDeploymentName)
+{
+    Instructions = "You are an Agent helping with browser automation tasks.\n" +
+    "You can answer questions, provide information, and assist with various tasks\n" +
+    "related to web browsing using the Browser Automation tool available to you.",
+    Tools = {playwrightTool}
+};
+AgentVersion agentVersion = await projectClient.Agents.CreateAgentVersionAsync(
+    agentName: "myAgent",
+    options: new(agentDefinition));
+```
+
+Streaming response outputs with browser automation provides incremental updates as the automation is processed. This is advised for interactive scenarios, as browser automation can require several minutes to fully complete.
+
+```C# Snippet:Sample_CreateResponse_BrowserAutomotion_Async
+ProjectResponsesClient responseClient = projectClient.OpenAI.GetProjectResponsesClientForAgent(agentVersion.Name);
+ResponseCreationOptions responseOptions = new()
+{
+    ToolChoice = ResponseToolChoice.CreateRequiredChoice()
+};
+await foreach (StreamingResponseUpdate update in responseClient.CreateResponseStreamingAsync(
+        userInputText: "Your goal is to report the percent of Microsoft year-to-date stock price change.\n" +
+        "To do that, go to the website finance.yahoo.com.\n" +
+        "At the top of the page, you will find a search bar.\n" +
+        "Enter the value 'MSFT', to get information about the Microsoft stock price.\n" +
+        "At the top of the resulting page you will see a default chart of Microsoft stock price.\n" +
+        "Click on 'YTD' at the top of that chart, and report the percent value that shows up just below it.",
+        options: responseOptions))
+{
+    ParseResponse(update);
+}
+```
+
 ## SharePoint tool
 `SharepointAgentTool` allows Agent to access SharePoint pages to get the data context. Use the SharePoint connection name as it is shown in the connections section of Microsoft Foundry to get the connection. Get the connection ID to initialize the `SharePointGroundingToolOptions`, which will be used to create `SharepointAgentTool`.
 

sdk/ai/Azure.AI.Projects.OpenAI/samples/Sample23_BrowserAutomationTool.md

@@ -0,0 +1,152 @@
+# Sample for use of `BrowserAutomationAgentTool` and Agents in Azure.AI.Projects.OpenAI.
+
+Playwright is a Node.js library for browser automation. Microsoft provides the [Azure Playwright workspace](https://learn.microsoft.com/javascript/api/overview/azure/playwright-readme), which can execute Playwright-based tasks triggered by an Agent using the BrowserAutomationAgentTool.
+
+## Create Azure Playwright workspace
+
+1. Deploy an Azure Playwright workspace.
+2. In the **Get started** section, open **2. Set up authentication**.
+3. **Select Service Access Token**, then choose **Generate Token**. **Save the token immediately-once you close the page, it cannot be viewed again.**
+
+## Configure Microsoft Foundry
+
+1. Open the left navigation and select **Management center**.
+2. Choose **Connected resources**.
+3. Create a new connection of type **Serverless Model**.
+4. Provide a name, then paste your Access Token into the **Key** field.
+5. Set the Playwright Workspace Browser endpoint as the **Target URI**. You can find this endpoint on the Workspace **Overview page**. It begins with `wss://`.
+
+## Run this sample.
+
+1. Begin by creating the Agent client and reading the required environment variables. Please note that the Browser automation operations may take longer than usual and requiring request timeout to be at least 5 minutes.
+
+```C# Snippet:Sample_CreateProjectClient_BrowserAutomotion
+var projectEndpoint = System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
+var modelDeploymentName = System.Environment.GetEnvironmentVariable("MODEL_DEPLOYMENT_NAME");
+var playwrightConnectionName = System.Environment.GetEnvironmentVariable("PLAYWRIGHT_CONNECTION_NAME");
+AIProjectClientOptions options = new()
+{
+    NetworkTimeout = TimeSpan.FromMinutes(5)
+};
+AIProjectClient projectClient = new(endpoint: new Uri(projectEndpoint), tokenProvider: new DefaultAzureCredential(), options: options);
+```
+
+2. Create an Agent with  `BrowserAutomationAgentTool`. Use the serverless connection name to get the connection from the project and use the connection ID to create the tool.
+
+Synchronous sample:
+```C# Snippet:Sample_CreateAgent_BrowserAutomotion_Sync
+AIProjectConnection playwrightConnection = projectClient.Connections.GetConnection(playwrightConnectionName);
+BrowserAutomationAgentTool playwrightTool = new(
+    new BrowserAutomationToolParameters(
+        new BrowserAutomationToolConnectionParameters(playwrightConnection.Id)
+    ));
+
+PromptAgentDefinition agentDefinition = new(model: modelDeploymentName)
+{
+    Instructions = "You are an Agent helping with browser automation tasks.\n" +
+    "You can answer questions, provide information, and assist with various tasks\n" +
+    "related to web browsing using the Browser Automation tool available to you.",
+    Tools = { playwrightTool }
+};
+AgentVersion agentVersion = projectClient.Agents.CreateAgentVersion(
+    agentName: "myAgent",
+    options: new(agentDefinition));
+```
+
+Asynchronous sample:
+```C# Snippet:Sample_CreateAgent_BrowserAutomotion_Async
+AIProjectConnection playwrightConnection = await projectClient.Connections.GetConnectionAsync(playwrightConnectionName);
+BrowserAutomationAgentTool playwrightTool = new(
+    new BrowserAutomationToolParameters(
+        new BrowserAutomationToolConnectionParameters(playwrightConnection.Id)
+    ));
+
+PromptAgentDefinition agentDefinition = new(model: modelDeploymentName)
+{
+    Instructions = "You are an Agent helping with browser automation tasks.\n" +
+    "You can answer questions, provide information, and assist with various tasks\n" +
+    "related to web browsing using the Browser Automation tool available to you.",
+    Tools = {playwrightTool}
+};
+AgentVersion agentVersion = await projectClient.Agents.CreateAgentVersionAsync(
+    agentName: "myAgent",
+    options: new(agentDefinition));
+```
+
+3. To parse the stream, obtained from the Agent, we will create a helper method `ParseResponse`.
+
+```C# Snippet:Sample_ParseResponse_BrowserAutomotion
+private static void ParseResponse(StreamingResponseUpdate streamResponse)
+{
+    if (streamResponse is StreamingResponseCreatedUpdate createUpdate)
+    {
+        Console.WriteLine($"Stream response created with ID: {createUpdate.Response.Id}");
+    }
+    else if (streamResponse is StreamingResponseOutputTextDeltaUpdate textDelta)
+    {
+        Console.WriteLine($"Delta: {textDelta.Delta}");
+    }
+    else if (streamResponse is StreamingResponseOutputTextDoneUpdate textDoneUpdate)
+    {
+        Console.WriteLine($"Response done with full message: {textDoneUpdate.Text}");
+    }
+    else if (streamResponse is StreamingResponseErrorUpdate errorUpdate)
+    {
+        throw new InvalidOperationException($"The stream has failed with the error: {errorUpdate.Message}");
+    }
+}
+```
+
+4. Create the response stream. We also make sure that the agent using tool by setting `ToolChoice = ResponseToolChoice.CreateRequiredChoice()` on the `ResponseCreationOptions`.
+
+Synchronous sample:
+```C# Snippet:Sample_CreateResponse_BrowserAutomotion_Sync
+ProjectResponsesClient responseClient = projectClient.OpenAI.GetProjectResponsesClientForAgent(agentVersion.Name);
+ResponseCreationOptions responseOptions = new()
+{
+    ToolChoice = ResponseToolChoice.CreateRequiredChoice()
+};
+foreach (StreamingResponseUpdate update in responseClient.CreateResponseStreaming(
+        userInputText: "Your goal is to report the percent of Microsoft year-to-date stock price change.\n" +
+        "To do that, go to the website finance.yahoo.com.\n" +
+        "At the top of the page, you will find a search bar.\n" +
+        "Enter the value 'MSFT', to get information about the Microsoft stock price.\n" +
+        "At the top of the resulting page you will see a default chart of Microsoft stock price.\n" +
+        "Click on 'YTD' at the top of that chart, and report the percent value that shows up just below it.",
+        options: responseOptions))
+{
+    ParseResponse(update);
+}
+```
+
+Asynchronous sample:
+```C# Snippet:Sample_CreateResponse_BrowserAutomotion_Async
+ProjectResponsesClient responseClient = projectClient.OpenAI.GetProjectResponsesClientForAgent(agentVersion.Name);
+ResponseCreationOptions responseOptions = new()
+{
+    ToolChoice = ResponseToolChoice.CreateRequiredChoice()
+};
+await foreach (StreamingResponseUpdate update in responseClient.CreateResponseStreamingAsync(
+        userInputText: "Your goal is to report the percent of Microsoft year-to-date stock price change.\n" +
+        "To do that, go to the website finance.yahoo.com.\n" +
+        "At the top of the page, you will find a search bar.\n" +
+        "Enter the value 'MSFT', to get information about the Microsoft stock price.\n" +
+        "At the top of the resulting page you will see a default chart of Microsoft stock price.\n" +
+        "Click on 'YTD' at the top of that chart, and report the percent value that shows up just below it.",
+        options: responseOptions))
+{
+    ParseResponse(update);
+}
+```
+
+9. After the sample is completed, delete the Agent we have created.
+
+Synchronous sample:
+```C# Snippet:Sample_Cleanup_BrowserAutomotion_Sync
+projectClient.Agents.DeleteAgentVersion(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
+```
+
+Asynchronous sample:
+```C# Snippet:Sample_Cleanup_BrowserAutomotion_Async
+await projectClient.Agents.DeleteAgentVersionAsync(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
+```

sdk/ai/Azure.AI.Projects.OpenAI/tests/ProjectsOpenAITestEnvironment.cs

@@ -29,6 +29,7 @@ public class ProjectsOpenAITestEnvironment : TestEnvironment
         public string CUSTOM_BING_CONNECTION_NAME => GetRecordedVariable("CUSTOM_BING_CONNECTION_NAME");
         public string BING_CUSTOM_SEARCH_INSTANCE_NAME => GetRecordedVariable("BING_CUSTOM_SEARCH_INSTANCE_NAME");
         public string MCP_PROJECT_CONNECTION_NAME => GetRecordedOptionalVariable("MCP_PROJECT_CONNECTION_NAME");
+        public string PLAYWRIGHT_CONNECTION_NAME => GetRecordedOptionalVariable("PLAYWRIGHT_CONNECTION_NAME");
         public string SHAREPOINT_CONNECTION_NAME => GetRecordedOptionalVariable("SHAREPOINT_CONNECTION_NAME");
         public string WrappedGetRecordedVariable(string key, bool isSecret = true)
         {