Playwright Azure Reporter

A Playwright reporter that publishes test results to Azure DevOps Test Plans.

Table of Contents

• Quick Start
• Defining Test Case IDs
  • In Test Title
  • Using Tags
  • Using Annotations
• Configuration
  • Required Options
  • General Options
  • Authentication
  • Publishing Modes
  • Retry Results
  • Attachments & Logs
  • Test Point Mapping
  • Test Case ID Matching
  • Test Case Summary
  • Auto-Mark as Automated
• CI/CD Integration
• Releases

Quick Start

Install the package:

npm install @alex_neo/playwright-azure-reporter

Add the reporter to your playwright.config.ts:

import { defineConfig } from '@playwright/test';
import type { AzureReporterOptions } from '@alex_neo/playwright-azure-reporter';

export default defineConfig({
  reporter: [
    ['list'],
    [
      '@alex_neo/playwright-azure-reporter',
      {
        orgUrl: 'https://dev.azure.com/your-organization-name',
        token: process.env.AZURE_TOKEN,
        planId: 44,
        projectName: 'SampleProject',
        publishTestResultsMode: 'testRun',
      } as AzureReporterOptions,
    ],
  ],
});

Each test must reference an existing Azure DevOps test case ID. See Defining Test Case IDs for all supported formats.

Defining Test Case IDs

You must link each test to an existing test case in Azure DevOps before running tests. There are three ways to do this.

In Test Title

Wrap test case IDs in square brackets inside the test name:

• [1] Test name - single test case
• [1,2,3] Test name - multiple test cases
• [1, 2, 3] Test name [4] another [5][6] combined - mixed formats

test.describe('Test suite', () => {
  test('[1] First Test', () => {
    expect(true).toBe(true);
  });

  test('[10,11,12] Multiple test cases', () => {
    expect(true).toBe(true);
  });
});

Using Tags

Use Playwright's tag syntax with the format @[ID]:

test.describe('Test suite', () => {
  test('Test name', {
    tag: ['@[1]', '@smoke', '@slow'],
  }, () => {
    expect(true).toBe(true);
  });
});

The @ prefix is required by Playwright. Non-ID tags (like @smoke) are ignored by the reporter.

Using Annotations

Use Playwright's annotation API with testCaseIdZone: 'annotation':

test('Test case', {
  annotation: { type: 'TestCase', description: '12345' },
}, () => {
  expect(true).toBe(true);
});

See Test Case ID Matching for advanced matching patterns with annotations.

Configuration

Required Options

Option	Type	Description
orgUrl	string	Full URL for your organization. Example: https://dev.azure.com/your-organization-name
projectName	string	Name of your Azure DevOps project
planId	number	ID of the test plan (from the test plan URL: planId=4)

Note: Some APIs (e.g. ProfileApi) must be hit at the deployment level: https://vssps.dev.azure.com/{yourorgname}

General Options

Option	Type	Default	Description
isDisabled	boolean	false	Disable the reporter entirely
environment	string	undefined	Prefix for test run names (e.g. 'QA')
logging	boolean	false	Enable debug logging
testRunTitle	string	'Playwright Test Run'	Title for newly created test runs
testRunConfig	object	undefined	Extra data passed when creating a test run. See API docs

Authentication

The reporter supports three authentication methods:

Personal Access Token (PAT) - Default

{
  token: 'your-personal-access-token',
  authType: 'pat' // Optional, this is the default
}

Access Token

{
  token: 'your-access-token',
  authType: 'accessToken'
}

Azure Managed Identity

{
  token: 'not-used', // Token field is required but ignored for managedIdentity
  authType: 'managedIdentity',
  applicationIdURI: '499b84ac-1321-427f-aa17-267ca6975798/.default'
}

When to use each:

• PAT: Personal Access Token from Azure DevOps. Suitable for most use cases.
• Access Token: OAuth access token. Suitable for CI/CD environments with token-based authentication.
• Managed Identity: Azure environments with managed identity. Uses DefaultAzureCredential supporting Azure CLI, managed identity, environment variables, etc.

For more examples, see Authentication Examples.

Required Token Scopes

Scope	Access	Purpose
Test Management	Read & Write	Creating/updating test runs, publishing results, uploading attachments
Work Items	Read & Write	Marking test cases as automated (only if autoMarkTestCasesAsAutomated is enabled)
Project and Team	Read	Accessing project information

Alternatively, use a token with Full access scope.

Note: If your token lacks Work Items write permissions but autoMarkTestCasesAsAutomated is enabled, the reporter will log errors for automation updates but continue publishing test results normally.

Publishing Modes

Control how and when test results are sent to Azure DevOps with publishTestResultsMode:

Mode	Default	Description
'testResult'	Yes	Results published after each test completes, in parallel with test execution
'testRun'		Results batched and published at the end of the test run
'testRunADO'		Updates existing results in an Azure DevOps test run (requires isExistingTestRun: true and testRunId)

Note: In testRun mode, if the same test case ID appears in multiple tests, only the last result is kept.

Existing Test Runs

To publish results to an existing test run instead of creating a new one:

{
  publishTestResultsMode: 'testRunADO',
  isExistingTestRun: true,
  testRunId: 12345, // or set AZURE_PW_TEST_RUN_ID env variable
}

• isExistingTestRun [boolean] - Publish to an existing test run without creating/completing it. Default: false.
• testRunId [number] - ID of the existing test run. Can also be set via AZURE_PW_TEST_RUN_ID env variable. Reporter options take precedence over the env variable.

Note: When using isExistingTestRun, the test run is not completed automatically. You must complete it manually.

Retry Results

Control how retry attempts are published with publishRetryResults:

Mode	Description
'all'	Every retry attempt is published as a separate test result (default)
'last'	Only the final attempt is published. Intermediate failures are skipped

{
  publishRetryResults: 'last',
  // works with Playwright's retries setting
}

Attachments & Logs

Option	Type	Default	Description
uploadAttachments	boolean	false	Upload attachments (screenshots/video) after each test
attachmentsType	(string|RegExp)[]	['screenshot']	Filter which attachments to upload by type or name pattern
uploadLogs	boolean	false	Upload stdout/stderr logs (independent of uploadAttachments)

Test Point Mapping

testPointMapper

A callback to map tests to specific test configurations (e.g. by browser):

import { TestCase } from '@playwright/test/reporter';
import { TestPoint } from 'azure-devops-node-api/interfaces/TestInterfaces';

{
  testPointMapper: async (testCase: TestCase, testPoints: TestPoint[]) => {
    switch (testCase.parent.project()?.use.browserName) {
      case 'chromium':
        return testPoints.filter((tp) => tp.configuration.id === '3');
      case 'firefox':
        return testPoints.filter((tp) => tp.configuration.id === '4');
      case 'webkit':
        return testPoints.filter((tp) => tp.configuration.id === '5');
      default:
        throw new Error('invalid test configuration!');
    }
  },
  testRunConfig: {
    // Get configuration IDs from:
    // https://dev.azure.com/{organization}/{project}/_apis/test/configurations
    configurationIds: [3, 4, 5],
  },
}

Important: Define testRunConfig.configurationIds and/or testPointMapper, otherwise results are published for all configurations.

rootSuiteId

Restricts test point resolution to a specific suite. Useful when different suites (e.g. smoke, integration) contain the same test case IDs. Can also be set via AZURE_PW_ROOT_SUITE_ID env variable. Reporter options take precedence.

Automation Tests
  - Smoke Tests (suiteId: 5)
    - Test 1 (caseId: 1)
    - Test 2 (caseId: 2)
  - Integration Tests (suiteId: 6)
    - Test 1 (caseId: 1)
    - Test 2 (caseId: 2)

Without rootSuiteId, running smoke tests would also match test points in Integration Tests. Set rootSuiteId: 5 to scope results to Smoke Tests only.

Test Case ID Matching

testCaseIdMatcher

A string, RegExp, or array of either, used to extract test case IDs. Default: /\[([\d,\s]+)\]/

Examples with test titles:

Test Title	Matcher	Extracted IDs
Test case @tag1=123	/@tag1=(\d+)/	['123']
Test case @TC=123 [@TC=456]	/@TC=(\d+)/	['123', '456']
Test case test123 TEST456	[/[a-z]+(\d+)/, /[A-Z]+(\d+)/]	['123', '456']

If an invalid matcher is provided (e.g. a number), an error is thrown: "Invalid testCaseIdMatcher. Must be a string or RegExp.".

testCaseIdZone

Specifies where to look for test case IDs: 'title' (default) or 'annotation'.

• 'title' - Extracts IDs from the test title and tags
• 'annotation' - Extracts IDs from test annotations only

Important: When using testCaseIdZone: 'annotation', you must also define testCaseIdMatcher.

When using annotation zone, the matcher works in two steps:

1. First matcher matches the annotation type (e.g. /(TestCase)/)
2. Subsequent matchers extract IDs from the annotation description

Annotation examples:

Simple description:

// testCaseIdZone: 'annotation', testCaseIdMatcher: /(TestCase)/
test('Test case', {
  annotation: { type: 'TestCase', description: '12345' },
}, () => { /* ... */ });
// Extracted: ['12345']

Azure DevOps URLs:

// testCaseIdZone: 'annotation', testCaseIdMatcher: [/(Test Case)/, /\/(\d+)/]
test('Test case', {
  annotation: {
    type: 'Test Case',
    description: 'https://dev.azure.com/org/project/_workitems/edit/12345, https://dev.azure.com/org/project/_workitems/edit/54321',
  },
}, () => { /* ... */ });
// Extracted: ['12345', '54321']

Bracketed format:

// testCaseIdZone: 'annotation', testCaseIdMatcher: [/(Test Case)/, /\[([\d,\s]+)\]/]
test('Test case', {
  annotation: { type: 'Test Case', description: '[12345, 67890]' },
}, () => { /* ... */ });
// Extracted: ['12345', '67890']

Test Case Summary

Generate a report of test cases that don't match the test plan. Helps identify missing, misconfigured, or unassigned test cases.

{
  testCaseSummary: {
    enabled: true,
    outputPath: './test-case-summary.md',
    consoleOutput: true,
    publishToRun: true,
  },
}

Sub-option	Type	Default	Description
enabled	boolean	false	Enable summary generation
outputPath	string	undefined	File path for the summary report
consoleOutput	boolean	true	Print summary to console
publishToRun	boolean	false	Upload summary as a Markdown attachment to the ADO test run

Output channels work independently - enable any combination:

consoleOutput	outputPath	publishToRun	Result
true	-	-	Console only
true	set	-	Console + file
-	-	true	ADO attachment only
true	set	true	All three channels

When all test cases match, a success message is logged instead.

Note: When publishToRun: true, the attachment is only uploaded if unmatched test points exist. Key summary lines are always logged regardless of the logging setting to ensure CI visibility.

Example output:

# Test Case Summary

Found 1 test(s) with test case IDs that don't match the test plan:

## Tests with No Matching Test Points (1)

- **[777] Test with file output**
  - File: tests/example.spec.ts:3
  - Test Case IDs: [777]

## Recommendations

- Verify test case IDs exist in Azure DevOps test plan 4
- Check that test cases are assigned to configurations: [10, 20] (Firefox, Safari)
- Ensure test cases are included in the test plan suite structure

Auto-Mark as Automated

Automatically update Azure DevOps test case work items to reflect their automation status when tests are executed.

{
  autoMarkTestCasesAsAutomated: {
    enabled: true,
    updateAutomatedTestName: true,
    updateAutomatedTestStorage: true,
  },
}

Sub-option	Type	Default	Description
enabled	boolean	false	Enable automatic marking
updateAutomatedTestName	boolean	true	Set AutomatedTestName field
updateAutomatedTestStorage	boolean	true	Set AutomatedTestStorage field
automatedTestNameFormat	string	'title'	'title' or 'titleWithParent' (includes parent suite name)
automatedTestStoragePath	boolean | function	false	false: filename only, true: full path, function: custom callback
automatedTestType	function	undefined	Callback to set AutomatedTestType (e.g. 'Unit Test', 'E2E Test')

What it does:

1. Checks each test case's automation status in Azure DevOps
2. If "Not Automated": sets AutomationStatus to "Automated", updates name/storage/type fields, generates a new AutomatedTestId GUID
3. If already "Automated": optionally updates name and storage fields if they differ

Advanced example:

{
  autoMarkTestCasesAsAutomated: {
    enabled: true,
    updateAutomatedTestName: true,
    updateAutomatedTestStorage: true,
    automatedTestNameFormat: 'titleWithParent',
    automatedTestStoragePath: (testCase) => {
      // Use relative path from project root
      const parts = testCase.location.file.split('/');
      const idx = parts.indexOf('my-project');
      return idx >= 0 ? parts.slice(idx + 1).join('/') : testCase.location.file;
    },
    automatedTestType: (testCase) => {
      if (testCase.location.file.includes('/e2e/')) return 'End-to-End Test';
      if (testCase.location.file.includes('/integration/')) return 'Integration Test';
      return 'Functional Test';
    },
  },
}

Tip: Use automatedTestNameFormat: 'titleWithParent' to prevent name collisions across suites. Use automatedTestStoragePath callback to avoid exposing absolute paths.

Note: Requires Work Items write permissions. Works in all publishing modes (testResult, testRun, testRunADO).

CI/CD Integration

Environment Variables

Variable	Description
AZURE_PW_TEST_RUN_ID	Set automatically after the test run is created. Available as process.env.AZURE_PW_TEST_RUN_ID. In testResult mode it's set at the start; in testRun mode at the end; in testRunADO mode it contains the existing run ID.
AZURE_PW_ROOT_SUITE_ID	Alternative to the rootSuiteId option
AZUREPWDEBUG	Enable debug logging: '1' = enabled, '0' = disabled (default)

Azure DevOps Pipeline Example

- script: npx playwright test
  displayName: 'Run Playwright tests'
  name: 'playwright'
  env:
    CI: 'true'
    AZUREPWDEBUG: '1'

# Access the test run ID from subsequent steps (since v1.10.0)
- script: echo $(playwright.AZURE_PW_TEST_RUN_ID)
  displayName: 'Print test run id'

Releases

For detailed information about stable releases, beta releases, and version management, see RELEASES.md.