-
-
Notifications
You must be signed in to change notification settings - Fork 21
Add support for Microsoft.Testing.Platform
#53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
38febc9
Migrate from `vstest` to `Microsoft.Testing.Platform` (#43)
Evangelink 3fe7b21
Format
Tyrrrz a2b5bd6
wip
Tyrrrz 3cd65bd
Format
Tyrrrz e42c6a6
asd
Tyrrrz ad398b2
asd
Tyrrrz d1c23fa
asd
Tyrrrz 09cb57f
asd
Tyrrrz 4e8b5df
Retrofit support for VSTest via an abstraction layer
Tyrrrz 0fa368d
asd
Tyrrrz ee603f1
asd
Tyrrrz 0ff54ee
asd
Tyrrrz b0959a1
asd
Tyrrrz 2a04ac6
asd
Tyrrrz b375377
asd
Tyrrrz c3baa5e
asd
Tyrrrz 7e73608
asd
Tyrrrz 1ec70f8
asd
Tyrrrz baa088e
asd
Tyrrrz e202765
asd
Tyrrrz 05fc4e5
asd
Tyrrrz 3896f9a
asd
Tyrrrz 4555bd6
asd
Tyrrrz d34bd30
asd
Tyrrrz 86232b4
asd
Tyrrrz d418098
asd
Tyrrrz d680c00
asd
Tyrrrz 1084ae2
MTP demo works
Tyrrrz af06bef
asd
Tyrrrz 23601ff
asd
Tyrrrz ed6a032
asd
Tyrrrz 201dcae
asd
Tyrrrz 460ef7b
Upgrade to MTP v2
Tyrrrz 77f8f51
asd
Tyrrrz 12e4e65
Merge
Tyrrrz ab562cc
asd
Tyrrrz c367f57
Use peer deps
Tyrrrz f4582b9
tests
Tyrrrz 80ff683
Async github interactions
Tyrrrz 65a9c48
Add MTP tests
Tyrrrz 3c4f5d5
asd
Tyrrrz 29a8325
asd
Tyrrrz b6a8c73
Migrate demo projects to MSTest
Tyrrrz b25749c
asd
Tyrrrz 423a9d9
asd
Tyrrrz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
asd
- Loading branch information
commit 23601ff9ff8712f00e9e94a89d5004d9d7c93538
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| using System.IO; | ||
| using System.Threading.Tasks; | ||
| using FluentAssertions; | ||
| using GitHubActionsTestLogger.Tests.VsTest; | ||
| using Microsoft.Testing.Platform.Builder; | ||
| using Microsoft.VisualStudio.TestPlatform.ObjectModel; | ||
| using Xunit; | ||
|
|
||
| namespace GitHubActionsTestLogger.Tests; | ||
|
|
||
| public class MtpAnnotationSpecs | ||
| { | ||
| [Fact] | ||
| public async Task I_can_use_the_logger_to_produce_annotations_for_failed_tests() | ||
| { | ||
| // Arrange | ||
| var builder = await TestApplication.CreateBuilderAsync([]); | ||
|
|
||
| await using var commandWriter = new StringWriter(); | ||
|
|
||
| builder.AddGitHubActionsReporting(commandWriter, TextWriter.Null); | ||
| var app = await builder.BuildAsync(); | ||
|
|
||
| // Act | ||
| events.SimulateTestRun( | ||
| new VsTestResultBuilder() | ||
| .SetDisplayName("Test1") | ||
| .SetOutcome(TestOutcome.Failed) | ||
| .SetErrorMessage("ErrorMessage") | ||
| .Build(), | ||
| new VsTestResultBuilder() | ||
| .SetDisplayName("Test2") | ||
| .SetOutcome(TestOutcome.Passed) | ||
| .Build(), | ||
| new VsTestResultBuilder() | ||
| .SetDisplayName("Test3") | ||
| .SetOutcome(TestOutcome.Skipped) | ||
| .Build() | ||
| ); | ||
|
|
||
| // Assert | ||
| var output = commandWriter.ToString().Trim(); | ||
|
|
||
| output.Should().StartWith("::error "); | ||
| output.Should().Contain("Test1"); | ||
| output.Should().Contain("ErrorMessage"); | ||
|
|
||
| output.Should().NotContain("Test2"); | ||
| output.Should().NotContain("Test3"); | ||
|
|
||
| testOutput.WriteLine(output); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| using System.Threading.Tasks; | ||
| using Microsoft.Testing.Platform.Builder; | ||
| using Xunit; | ||
|
|
||
| namespace GitHubActionsTestLogger.Tests; | ||
|
|
||
| public class MtpInitializationSpecs | ||
| { | ||
| [Fact] | ||
| public async Task I_can_use_the_logger_with_the_default_configuration() | ||
| { | ||
| // Arrange | ||
| var builder = await TestApplication.CreateBuilderAsync([]); | ||
|
|
||
| // Act & assert | ||
| builder.AddGitHubActionsReporting(); | ||
|
|
||
| // Can't perform a more meaningful assertion here without | ||
| // accessing internal members of the reporter. | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,9 +19,8 @@ | |
| <img src="favicon.png" alt="Icon" /> | ||
| </p> | ||
|
|
||
| **GitHub Actions Test Logger** is a custom logger for `dotnet test` that integrates with GitHub Actions. | ||
| When using this logger, failed tests are listed in job annotations and highlighted in code diffs. | ||
| Additionally, this logger also generates a job summary that contains detailed information about the executed test run. | ||
| **GitHub Actions Test Logger** is an extension for VSTest and MTP that reports test results via GitHub Actions. | ||
| It lists failed tests in job annotations, highlights them in code diffs, and produces a detailed job summary with information about the executed test run. | ||
|
|
||
| ## Terms of use<sup>[[?]](https://github.com/Tyrrrz/.github/blob/master/docs/why-so-political.md)</sup> | ||
|
|
||
|
|
@@ -45,48 +44,35 @@ To learn more about the war and how you can help, [click here](https://tyrrrz.me | |
|
|
||
| ## Usage | ||
|
|
||
| To use **GitHub Actions Test Logger**, install it in your test project and modify your GitHub Actions workflow by adding `--logger GitHubActions` to `dotnet test`: | ||
| **GitHub Actions Test Logger** is available for both the classic **VSTest** test runner and the newer **Microsoft.Testing.Platform**. | ||
|
|
||
| ```yaml | ||
| name: main | ||
| on: [push, pull_request] | ||
| ### [Microsoft.Testing.Platform](https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-intro) | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
| Simply install the package in your test project. | ||
| The provided test reporter will automatically be detected and enabled when running in a GitHub Actions environment. | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Install .NET | ||
| uses: actions/setup-dotnet@v4 | ||
|
|
||
| - name: Build & test | ||
| run: dotnet test --configuration Release --logger GitHubActions | ||
| ``` | ||
| ### [VSTest](https://github.com/microsoft/vstest) | ||
|
|
||
| By default, the logger will only report failed tests in the job summary and annotations. | ||
| If you want the summary to include detailed information about passed and skipped tests as well, update the workflow as follows: | ||
| Install the package in your test project and specify the logger when running `dotnet test`: | ||
|
|
||
| ```yaml | ||
| name: main | ||
| on: [push, pull_request] | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
| build: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| # ... | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Build & test | ||
| run: > | ||
| dotnet test | ||
| --configuration Release | ||
| --logger "GitHubActions;summary.includePassedTests=true;summary.includeSkippedTests=true" | ||
| ``` | ||
| - name: Install .NET | ||
| uses: actions/setup-dotnet@v4 | ||
|
|
||
| > **Warning**: | ||
| > The new testing platform (i.e. `Microsoft.Testing.Platform`) [is not yet supported](https://github.com/Tyrrrz/GitHubActionsTestLogger/issues/41). This is because VSTest and MTP are using different extensibility models, and this project existed before MTP existed. | ||
| > To use **GitHub Actions Test Logger**, make sure to use the classic testing experience (`vstest`) instead. | ||
| - name: Build & test | ||
| run: dotnet test --configuration Release --logger GitHubActions | ||
| ``` | ||
|
|
||
| > **Important**: | ||
| > Ensure that your test project references the latest version of **Microsoft.NET.Test.Sdk**. | ||
|
|
@@ -95,10 +81,10 @@ jobs: | |
| > **Important**: | ||
| > If you are using **.NET SDK v2.2 or lower**, you need to [set the `<CopyLocalLockFileAssemblies>` property to `true` in your test project](https://github.com/Tyrrrz/GitHubActionsTestLogger/issues/5#issuecomment-648431667). | ||
|
|
||
| ### Collecting source information | ||
| #### Collecting source information | ||
|
|
||
| **GitHub Actions Test Logger** can leverage source information to link reported test results to the locations in the source code where the corresponding tests are defined. | ||
| By default, `dotnet test` does not collect source information, so the logger relies on stack traces to extract it manually. | ||
| By default, VSTest does not collect source information, so the logger relies on stack traces to extract it manually. | ||
| This approach only works for failed tests, and even then may not always be fully accurate. | ||
|
|
||
| To instruct the runner to collect source information, add the `RunConfiguration.CollectSourceInformation=true` argument to the command as shown below: | ||
|
|
@@ -129,7 +115,29 @@ jobs: | |
|
|
||
| ### Customizing behavior | ||
|
|
||
| When running `dotnet test`, you can customize the logger's behavior by passing additional options: | ||
| When running `dotnet test`, you can customize the logger's behavior by passing additional options on the command line: | ||
|
|
||
| **Microsoft.Testing.Platform**: | ||
|
|
||
| ```yml | ||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| # ... | ||
|
|
||
| - name: Build & test | ||
| run: > | ||
| dotnet test | ||
| --configuration Release | ||
| -- | ||
| --report-github | ||
| --report-github-annotations-title=@test | ||
| --report-github-annotations-message=@error | ||
| ``` | ||
|
|
||
| **VSTest**: | ||
|
|
||
| ```yml | ||
| jobs: | ||
|
|
@@ -148,7 +156,7 @@ jobs: | |
|
|
||
| #### Custom annotation title | ||
|
|
||
| Use the `annotations-title` option to specify the annotation title format used for reporting test failures. | ||
| Use the `[--report-github-]annotations-title` option to specify the annotation title format used for reporting test failures. | ||
|
|
||
| The following replacement tokens are available: | ||
|
|
||
|
|
@@ -168,7 +176,7 @@ The following replacement tokens are available: | |
|
|
||
| #### Custom annotation message | ||
|
|
||
| Use the `annotations-message` option to specify the annotation message format used for reporting test failures. | ||
| Use the `[--report-github-]annotations-message` option to specify the annotation message format used for reporting test failures. | ||
| Supports the same replacement tokens as [`annotations-title-format`](#custom-annotation-title). | ||
|
||
|
|
||
| **Default**: `@error`. | ||
|
|
@@ -180,13 +188,13 @@ Supports the same replacement tokens as [`annotations-title-format`](#custom-ann | |
|
|
||
| #### Allow empty test summaries | ||
|
|
||
| Use the `summary-allow-empty` option to specify whether empty test runs should be included in the summary, for example as a result of using [test filters](https://learn.microsoft.com/dotnet/core/testing/selective-unit-tests). | ||
| Use the `[--report-github-]summary-allow-empty` option to specify whether empty test runs should be included in the summary, for example as a result of using [test filters](https://learn.microsoft.com/dotnet/core/testing/selective-unit-tests). | ||
|
|
||
| **Default**: `false`. | ||
|
|
||
| #### Include passed tests in the summary | ||
|
|
||
| Use the `summary-include-passed` option to specify whether passed tests should be included in the summary. | ||
| Use the `[--report-github-]summary-include-passed` option to specify whether passed tests should be included in the summary. | ||
| If you want to link passed tests to their corresponding source definitions, make sure to also enable [source information collection](#collecting-source-information). | ||
|
|
||
| **Default**: `true`. | ||
|
|
@@ -196,7 +204,7 @@ If you want to link passed tests to their corresponding source definitions, make | |
|
|
||
| #### Include skipped tests in the summary | ||
|
|
||
| Use the `summary-include-skipped` option to specify whether skipped tests should be included in the summary. | ||
| Use the `[--report-github-]summary-include-skipped` option to specify whether skipped tests should be included in the summary. | ||
| If you want to link skipped tests to their corresponding source definitions, make sure to also enable [source information collection](#collecting-source-information). | ||
|
|
||
| **Default**: `true`. | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.