Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion eng/Packages.Data.props
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@
<PackageReference Update="OpenAI" Version="[2.7.0,3.0)" />
<!-- Foundry libs depend on OpenAI; float SCM with upstream version -->
<PackageReference Update="System.ClientModel" Version="*" />
<PackageReference Update="System.Text.Json" Version="10.0.0" />
</ItemGroup>

<ItemGroup Condition="$(MSBuildProjectName.StartsWith('Azure.AI.AgentServer'))">
Expand Down Expand Up @@ -501,4 +502,4 @@
<UnbrandedGeneratorVersion>1.0.0-alpha.20251203.1</UnbrandedGeneratorVersion>
<AzureGeneratorVersion>1.0.0-alpha.20251125.3</AzureGeneratorVersion>
</PropertyGroup>
</Project>
</Project>
138 changes: 138 additions & 0 deletions sdk/ai/Azure.AI.Projects.OpenAI/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ Develop Agents using the Azure AI Foundry platform, leveraging an extensive ecos
- [MCP tool with project connection](#mcp-tool-with-project-connection)
- [OpenAPI tool](#openapi-tool)
- [OpenAPI tool with project connection](#openapi-tool-project-connection)
- [Browser automation](#browser-automation)
- [SharePoint tool](#sharepoint-tool)
- [Tracing](#tracing)
- [Tracing to Azure Monitor](#tracing-to-azure-monitor)
- [Tracing to Console](#tracing-to-console)
Expand Down Expand Up @@ -1092,6 +1094,141 @@ OpenAIResponse response = await responseClient.CreateResponseAsync(
Console.WriteLine(response.GetOutputText());
```

## Browser automation

Playwright is a Node.js library for browser automation. Microsoft provides the [Azure Playwright workspace](https://learn.microsoft.com/javascript/api/overview/azure/playwright-readme), which can execute Playwright-based tasks triggered by an Agent using the BrowserAutomationAgentTool.

### Create Azure Playwright workspace

1. Deploy an Azure Playwright workspace.
2. In the **Get started** section, open **2. Set up authentication**.
3. **Select Service Access Token**, then choose **Generate Token**. **Save the token immediately-once you close the page, it cannot be viewed again.**

### Configure Microsoft Foundry

1. Open the left navigation and select **Management center**.
2. Choose **Connected resources**.
3. Create a new connection of type **Serverless Model**.
4. Provide a name, then paste your Access Token into the **Key** field.
5. Set the Playwright Workspace Browser endpoint as the **Target URI**. You can find this endpoint on the Workspace **Overview page**. It begins with `wss://`.

Please note that Browser automation operations may take longer than typical calls to process. Using background mode for Responses or applying a network timeout of at least five minutes for non-background calls is highly recommended.

```C# Snippet:Sample_CreateProjectClient_BrowserAutomotion
var projectEndpoint = System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
var modelDeploymentName = System.Environment.GetEnvironmentVariable("MODEL_DEPLOYMENT_NAME");
var playwrightConnectionName = System.Environment.GetEnvironmentVariable("PLAYWRIGHT_CONNECTION_NAME");
AIProjectClientOptions options = new()
{
NetworkTimeout = TimeSpan.FromMinutes(5)
};
AIProjectClient projectClient = new(endpoint: new Uri(projectEndpoint), tokenProvider: new DefaultAzureCredential(), options: options);
```

To use Azure Playwright workspace we need to create agent with `BrowserAutomationAgentTool`.

```C# Snippet:Sample_CreateAgent_BrowserAutomotion_Async
AIProjectConnection playwrightConnection = await projectClient.Connections.GetConnectionAsync(playwrightConnectionName);
BrowserAutomationAgentTool playwrightTool = new(
new BrowserAutomationToolParameters(
new BrowserAutomationToolConnectionParameters(playwrightConnection.Id)
));

PromptAgentDefinition agentDefinition = new(model: modelDeploymentName)
{
Instructions = "You are an Agent helping with browser automation tasks.\n" +
"You can answer questions, provide information, and assist with various tasks\n" +
"related to web browsing using the Browser Automation tool available to you.",
Tools = {playwrightTool}
};
AgentVersion agentVersion = await projectClient.Agents.CreateAgentVersionAsync(
agentName: "myAgent",
options: new(agentDefinition));
```

Streaming response outputs with browser automation provides incremental updates as the automation is processed. This is advised for interactive scenarios, as browser automation can require several minutes to fully complete.

```C# Snippet:Sample_CreateResponse_BrowserAutomotion_Async
ProjectResponsesClient responseClient = projectClient.OpenAI.GetProjectResponsesClientForAgent(agentVersion.Name);
ResponseCreationOptions responseOptions = new()
{
ToolChoice = ResponseToolChoice.CreateRequiredChoice()
};
await foreach (StreamingResponseUpdate update in responseClient.CreateResponseStreamingAsync(
userInputText: "Your goal is to report the percent of Microsoft year-to-date stock price change.\n" +
"To do that, go to the website finance.yahoo.com.\n" +
"At the top of the page, you will find a search bar.\n" +
"Enter the value 'MSFT', to get information about the Microsoft stock price.\n" +
"At the top of the resulting page you will see a default chart of Microsoft stock price.\n" +
"Click on 'YTD' at the top of that chart, and report the percent value that shows up just below it.",
options: responseOptions))
{
ParseResponse(update);
}
```

## SharePoint tool
`SharepointAgentTool` allows Agent to access SharePoint pages to get the data context. Use the SharePoint connection name as it is shown in the connections section of Microsoft Foundry to get the connection. Get the connection ID to initialize the `SharePointGroundingToolOptions`, which will be used to create `SharepointAgentTool`.

```C# Snippet:Sample_CreateAgent_Sharepoint_Async
AIProjectConnection sharepointConnection = await projectClient.Connections.GetConnectionAsync(sharepointConnectionName);
SharePointGroundingToolOptions sharepointToolOption = new()
{
ProjectConnections = { new ToolProjectConnection(projectConnectionId: sharepointConnection.Id) }
};
PromptAgentDefinition agentDefinition = new(model: modelDeploymentName)
{
Instructions = "You are a helpful assistant.",
Tools = { new SharepointAgentTool(sharepointToolOption), }
};
AgentVersion agentVersion = await projectClient.Agents.CreateAgentVersionAsync(
agentName: "myAgent",
options: new(agentDefinition));
```

Create the response and make sure we are always using tool.

```C# Snippet:Sample_CreateResponse_Sharepoint_Async
ProjectResponsesClient responseClient = projectClient.OpenAI.GetProjectResponsesClientForAgent(agentVersion.Name);
ResponseCreationOptions responseOptions = new()
{
ToolChoice = ResponseToolChoice.CreateRequiredChoice()
};
OpenAIResponse response = await responseClient.CreateResponseAsync("What is Contoso whistleblower policy", options: responseOptions);
```

SharePoint tool can create the reference to the page, grounding the data. We will create the `GetFormattedAnnotation` method to get the URI annotation.

```C# Snippet:Sample_FormatReference_Sharepoint
private static string GetFormattedAnnotation(OpenAIResponse response)
{
foreach (ResponseItem item in response.OutputItems)
{
if (item is MessageResponseItem messageItem)
{
foreach (ResponseContentPart content in messageItem.Content)
{
foreach (ResponseMessageAnnotation annotation in content.OutputTextAnnotations)
{
if (annotation is UriCitationMessageAnnotation uriAnnotation)
{
return $" [{uriAnnotation.Title}]({uriAnnotation.Uri})";
}
}
}
}
}
return "";
}
```

Print the Agent output and add the annotation at the end.

```C# Snippet:Sample_WaitForResponse_Sharepoint
Assert.That(response.Status, Is.EqualTo(ResponseStatus.Completed));
Console.WriteLine($"{response.GetOutputText()}{GetFormattedAnnotation(response)}");
```

## Tracing
**Note:** The tracing functionality is currently in preview with limited scope. Only agent creation operations generate dedicated gen_ai traces currently. As a preview feature, the trace structure including spans, attributes, and events may change in future releases.

Expand Down Expand Up @@ -1131,6 +1268,7 @@ var tracerProvider = Sdk.CreateTracerProviderBuilder()
.AddAzureMonitorTraceExporter().Build();
```


### Tracing to Console

For tracing to console from your application, install the OpenTelemetry.Exporter.Console with [NuGet](https://www.nuget.org/ ):
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
# Sample for use of `BrowserAutomationAgentTool` and Agents in Azure.AI.Projects.OpenAI.

Playwright is a Node.js library for browser automation. Microsoft provides the [Azure Playwright workspace](https://learn.microsoft.com/javascript/api/overview/azure/playwright-readme), which can execute Playwright-based tasks triggered by an Agent using the BrowserAutomationAgentTool.

## Create Azure Playwright workspace

1. Deploy an Azure Playwright workspace.
2. In the **Get started** section, open **2. Set up authentication**.
3. **Select Service Access Token**, then choose **Generate Token**. **Save the token immediately-once you close the page, it cannot be viewed again.**

## Configure Microsoft Foundry

1. Open the left navigation and select **Management center**.
2. Choose **Connected resources**.
3. Create a new connection of type **Serverless Model**.
4. Provide a name, then paste your Access Token into the **Key** field.
5. Set the Playwright Workspace Browser endpoint as the **Target URI**. You can find this endpoint on the Workspace **Overview page**. It begins with `wss://`.

## Run this sample.

1. Begin by creating the Agent client and reading the required environment variables. Please note that the Browser automation operations may take longer than usual and requiring request timeout to be at least 5 minutes.

```C# Snippet:Sample_CreateProjectClient_BrowserAutomotion
var projectEndpoint = System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
var modelDeploymentName = System.Environment.GetEnvironmentVariable("MODEL_DEPLOYMENT_NAME");
var playwrightConnectionName = System.Environment.GetEnvironmentVariable("PLAYWRIGHT_CONNECTION_NAME");
AIProjectClientOptions options = new()
{
NetworkTimeout = TimeSpan.FromMinutes(5)
};
AIProjectClient projectClient = new(endpoint: new Uri(projectEndpoint), tokenProvider: new DefaultAzureCredential(), options: options);
```

2. Create an Agent with `BrowserAutomationAgentTool`. Use the serverless connection name to get the connection from the project and use the connection ID to create the tool.

Synchronous sample:
```C# Snippet:Sample_CreateAgent_BrowserAutomotion_Sync
AIProjectConnection playwrightConnection = projectClient.Connections.GetConnection(playwrightConnectionName);
BrowserAutomationAgentTool playwrightTool = new(
new BrowserAutomationToolParameters(
new BrowserAutomationToolConnectionParameters(playwrightConnection.Id)
));

PromptAgentDefinition agentDefinition = new(model: modelDeploymentName)
{
Instructions = "You are an Agent helping with browser automation tasks.\n" +
"You can answer questions, provide information, and assist with various tasks\n" +
"related to web browsing using the Browser Automation tool available to you.",
Tools = { playwrightTool }
};
AgentVersion agentVersion = projectClient.Agents.CreateAgentVersion(
agentName: "myAgent",
options: new(agentDefinition));
```

Asynchronous sample:
```C# Snippet:Sample_CreateAgent_BrowserAutomotion_Async
AIProjectConnection playwrightConnection = await projectClient.Connections.GetConnectionAsync(playwrightConnectionName);
BrowserAutomationAgentTool playwrightTool = new(
new BrowserAutomationToolParameters(
new BrowserAutomationToolConnectionParameters(playwrightConnection.Id)
));

PromptAgentDefinition agentDefinition = new(model: modelDeploymentName)
{
Instructions = "You are an Agent helping with browser automation tasks.\n" +
"You can answer questions, provide information, and assist with various tasks\n" +
"related to web browsing using the Browser Automation tool available to you.",
Tools = {playwrightTool}
};
AgentVersion agentVersion = await projectClient.Agents.CreateAgentVersionAsync(
agentName: "myAgent",
options: new(agentDefinition));
```

3. To parse the stream, obtained from the Agent, we will create a helper method `ParseResponse`.

```C# Snippet:Sample_ParseResponse_BrowserAutomotion
private static void ParseResponse(StreamingResponseUpdate streamResponse)
{
if (streamResponse is StreamingResponseCreatedUpdate createUpdate)
{
Console.WriteLine($"Stream response created with ID: {createUpdate.Response.Id}");
}
else if (streamResponse is StreamingResponseOutputTextDeltaUpdate textDelta)
{
Console.WriteLine($"Delta: {textDelta.Delta}");
}
else if (streamResponse is StreamingResponseOutputTextDoneUpdate textDoneUpdate)
{
Console.WriteLine($"Response done with full message: {textDoneUpdate.Text}");
}
else if (streamResponse is StreamingResponseErrorUpdate errorUpdate)
{
throw new InvalidOperationException($"The stream has failed with the error: {errorUpdate.Message}");
}
}
```

4. Create the response stream. We also make sure that the agent using tool by setting `ToolChoice = ResponseToolChoice.CreateRequiredChoice()` on the `ResponseCreationOptions`.

Synchronous sample:
```C# Snippet:Sample_CreateResponse_BrowserAutomotion_Sync
ProjectResponsesClient responseClient = projectClient.OpenAI.GetProjectResponsesClientForAgent(agentVersion.Name);
ResponseCreationOptions responseOptions = new()
{
ToolChoice = ResponseToolChoice.CreateRequiredChoice()
};
foreach (StreamingResponseUpdate update in responseClient.CreateResponseStreaming(
userInputText: "Your goal is to report the percent of Microsoft year-to-date stock price change.\n" +
"To do that, go to the website finance.yahoo.com.\n" +
"At the top of the page, you will find a search bar.\n" +
"Enter the value 'MSFT', to get information about the Microsoft stock price.\n" +
"At the top of the resulting page you will see a default chart of Microsoft stock price.\n" +
"Click on 'YTD' at the top of that chart, and report the percent value that shows up just below it.",
options: responseOptions))
{
ParseResponse(update);
}
```

Asynchronous sample:
```C# Snippet:Sample_CreateResponse_BrowserAutomotion_Async
ProjectResponsesClient responseClient = projectClient.OpenAI.GetProjectResponsesClientForAgent(agentVersion.Name);
ResponseCreationOptions responseOptions = new()
{
ToolChoice = ResponseToolChoice.CreateRequiredChoice()
};
await foreach (StreamingResponseUpdate update in responseClient.CreateResponseStreamingAsync(
userInputText: "Your goal is to report the percent of Microsoft year-to-date stock price change.\n" +
"To do that, go to the website finance.yahoo.com.\n" +
"At the top of the page, you will find a search bar.\n" +
"Enter the value 'MSFT', to get information about the Microsoft stock price.\n" +
"At the top of the resulting page you will see a default chart of Microsoft stock price.\n" +
"Click on 'YTD' at the top of that chart, and report the percent value that shows up just below it.",
options: responseOptions))
{
ParseResponse(update);
}
```

9. After the sample is completed, delete the Agent we have created.

Synchronous sample:
```C# Snippet:Sample_Cleanup_BrowserAutomotion_Sync
projectClient.Agents.DeleteAgentVersion(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
```

Asynchronous sample:
```C# Snippet:Sample_Cleanup_BrowserAutomotion_Async
await projectClient.Agents.DeleteAgentVersionAsync(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
```
Loading