perf: filter-aware test collection for faster filtered runs

[thomhurst]

Summary

• Implements pre-filtering of test sources by type before metadata materialization
• Significantly improves performance for filtered test runs (e.g., --treenode-filter)
• Reduces redundant metadata collection by passing pre-collected metadata directly

Key Changes

1. FilterHints struct (MetadataFilterMatcher.cs): Extracts assembly, namespace, class, and method name hints from TreeNodeFilter for pre-filtering

2. CouldTypeMatch method: Quickly filters types based on hints without full metadata processing

3. Filter-aware collection (ITestDataCollector.cs, AotTestDataCollector.cs): New overload that accepts filter and pre-filters Sources.TestSources by type

4. BuildTestsFromMetadataAsync (TestBuilderPipeline.cs): New method to build tests from pre-collected metadata, avoiding redundant collection

5. TestDiscoveryService optimization: Uses BuildTestsFromMetadataAsync to avoid collecting metadata twice

6. MetadataDependencyExpander optimization: Added indexed lookups for O(1) dependency resolution

Performance Results

Metric	Before	After	Improvement
Filtered run (301→1 types)	~1366ms	~596ms	56% faster
Metadata collection	847ms	11ms	77x faster

Test plan

• All existing tests pass (10,452 tests)
• Filtered tests work correctly (60 filtered tests verified)
• Performance verified with profiling traces

🤖 Generated with Claude Code

[thomhurst]

Summary

This PR implements filter-aware test collection to significantly improve performance for filtered test runs by pre-filtering test sources by type before metadata materialization.

Critical Issues

1. Method Overload Ambiguity in TestBuilderPipeline.cs

The new BuildTestsFromMetadataAsync method creates an ambiguous overload:

Existing method (line 121):

public async Task<IEnumerable<AbstractExecutableTest>> BuildTestsFromMetadataAsync(
    IEnumerable<TestMetadata> testMetadata, 
    TestBuildingContext buildingContext)

New method added:

public async Task<IEnumerable<AbstractExecutableTest>> BuildTestsFromMetadataAsync(
    IEnumerable<TestMetadata> metadata,
    TestBuildingContext buildingContext,
    CancellationToken cancellationToken = default)

When called with just 2 parameters, the compiler cannot determine which overload to use. This will cause compilation errors at call sites.

Fix: Either:

• Remove the existing 2-parameter method and update all call sites to use the new 3-parameter version
• Rename the new method (e.g., BuildTestsFromMetadataAsync2, BuildTestsFromMetadataWithCancellationAsync)
• Make the cancellation token required (not optional) to differentiate the signatures

The existing method is called at:

• TUnit.Engine/Building/TestBuilderPipeline.cs:68
• TUnit.Engine/Building/TestBuilderPipeline.cs:133
• TUnit.Engine/Building/TestBuilderPipeline.cs:357

Suggestions

None - the optimization approach is sound and follows TUnit's performance-first principles.

Verdict

⚠️ REQUEST CHANGES - Method overload ambiguity must be fixed before merge

[thomhurst]

Fixed the method overload ambiguity:

• Removed the simpler duplicate BuildTestsFromMetadataAsync method
• Added CancellationToken cancellationToken = default parameter to the existing comprehensive method
• Now there's only one method with better error handling for dynamic tests

The existing method already had proper error handling and IDynamicTestMetadata support, so consolidating into that method keeps the behavior consistent.

Performance optimization is retained - verified that filtered test runs still use the filter-aware CollectTestMetadataAsync → BuildTestsFromMetadataAsync path.

[thomhurst]

Summary

This PR adds filter-aware pre-filtering optimization to test discovery, using filter hints to skip non-matching test sources by type before materializing metadata, plus O(1) indexed dependency lookup to replace O(n) scanning.

Critical Issues

None found ✅

Observations

Dual-Mode Implementation ✅

The changes correctly implement dual-mode support:

• AOT mode (AotTestDataCollector.cs): Pre-filters Sources.TestSources by type using FilterHints.CouldTypeMatch() before enumerating test sources
• Reflection mode (ReflectionTestDataCollector.cs): Implements the new interface method but delegates to standard collection since reflection mode doesn't have type-indexed sources

Both paths maintain behavioral equivalence as required.

Performance Optimizations

Two significant improvements:

1. Filter hint extraction (MetadataFilterMatcher.cs): Parses TreeNodeFilter to extract assembly/namespace/class/method patterns for O(1) type matching
2. Dependency expansion indexing (MetadataDependencyExpander.cs): Replaces O(n²) dependency scanning with O(1) lookups using three dictionaries indexed by class type, (class+method), and method name

Design Quality

• Early return optimization when no dependencies exist
• Proper handling of generic types in class name matching
• Wildcard detection to avoid false positives
• CancellationToken propagation added to BuildTestsFromMetadataAsync

Minor Observations

1. Regex usage (MetadataFilterMatcher.cs:99): Using System.Text.RegularExpressions.Regex.Replace - consider replacing with simple string operations for better AOT performance
2. Dictionary initialization syntax: Minor inconsistency between new Dictionary() and [] collection expressions

Verdict

✅ APPROVE - No critical issues. The implementation is sound, follows TUnit's dual-mode requirements, and delivers meaningful performance improvements without breaking changes.

[thomhurst]

Fixed the CI test failures. The failing tests were DynamicTestIndexTests.Test and DynamicTests.Test, which were failing because tests like TestWithCustomDisplayName and TestWithParameterizedDisplayName weren't getting their custom display names applied.

Root cause: The GenerateDynamicTests method in TestBuilderPipeline.cs was not handling custom DisplayName from IDynamicTestMetadata. This was an oversight in PR #4058 which added DisplayName support to BuildTestsFromSingleMetadataAsync but missed the GenerateDynamicTests method.

Fix: Updated GenerateDynamicTests to:

1. Extract dynamicTestMetadata at the start for consistent access
2. Use dynamicTestMetadata?.DisplayName ?? metadata.TestName for display name (instead of just metadata.TestName)
3. Set context.Metadata.DisplayName when a custom display name is specified

All dynamic tests now pass locally (18/18).

[thomhurst]

Summary

Implements filter-aware test collection that pre-filters test sources by type before metadata materialization, improving filtered test run performance by 56%.

Critical Issues

None found ✅

Analysis

TUnit Rule Compliance

✅ Dual-Mode Implementation

• AOT mode (AotTestDataCollector.cs:25-51): Pre-filters Sources.TestSources by type using FilterHints.CouldTypeMatch()
• Reflection mode (ReflectionTestDataCollector.cs:136-143): Implements interface but delegates to standard collection
• Both modes maintain behavioral equivalence as required

✅ Performance First
Two optimization strategies:

1. Filter hint extraction (MetadataFilterMatcher.cs:83-148): O(1) type matching before metadata collection
2. Dependency expansion indexing (MetadataDependencyExpander.cs:66-125): O(1) dictionary lookups replacing O(n²) scanning

✅ AOT Compatible

• No new reflection usage in hot paths
• Uses source-generated metadata in AOT mode
• Suppression attributes properly applied

Code Quality

Interface Design (ITestDataCollector.cs:17-24)
Clean extension of existing interface with optional filter parameter. Maintains backwards compatibility.

DisplayName Support (TestBuilderPipeline.cs:165-237)
The fix for DynamicTests properly handles IDynamicTestMetadata.DisplayName:

• Line 168: Extract dynamicTestMetadata once for consistency
• Line 174-176: Use custom DisplayName if available, falls back to TestName
• Line 232-235: Set DisplayName on context metadata

Regex Usage Note (MetadataFilterMatcher.cs:99)
System.Text.RegularExpressions.Regex.Replace is used to strip property filters. Consider replacing with IndexOf/Substring for better AOT performance, though this is non-hot-path code so impact is minimal.

Previous Comments Addressed

Per your comments:

• ✅ Method overload ambiguity fixed by consolidating into single method with CancellationToken
• ✅ DisplayName support added to GenerateDynamicTests
• ✅ DynamicTestIndexTests and DynamicTests now passing

Suggestions

Optional: String manipulation over Regex

// MetadataFilterMatcher.cs:99 - Replace regex with simple string ops
if (filterString.Contains('['))
{
    var bracketIndex = filterString.IndexOf('[');
    filterString = filterString.Substring(0, bracketIndex);
}

This avoids regex compilation overhead and is more AOT-friendly, but the impact is negligible since this isn't a hot path.

Verdict

✅ APPROVE - Excellent performance optimization that follows all TUnit critical rules. The 56% improvement for filtered runs and 77x faster metadata collection are significant wins. Implementation is clean, maintains dual-mode compatibility, and all identified issues have been resolved.

[thomhurst]

Summary

Implements filter-aware test collection that pre-filters test sources by type before metadata materialization, improving filtered test run performance by 56%.

Critical Issues

None found ✅

Analysis

TUnit Rule Compliance ✅

Dual-Mode Implementation
The PR correctly implements dual-mode support:

• AOT mode (AotTestDataCollector.cs:15-51): Adds CollectTestsAsync(string, ITestExecutionFilter?) overload that pre-filters Sources.TestSources by type using FilterHints.CouldTypeMatch() before enumerating test sources
• Reflection mode (ReflectionTestDataCollector.cs:136-143): Implements the new interface method but delegates to standard collection (no type-indexed sources in reflection mode)
• Both modes maintain behavioral equivalence as required by TUnit critical rules

Performance-First Approach
Two complementary optimizations:

1. Filter hint extraction (MetadataFilterMatcher.cs:83-148): Parses TreeNodeFilter to extract assembly/namespace/class/method patterns for O(1) type matching before metadata collection
2. Dependency expansion indexing (MetadataDependencyExpander.cs:83-197): Replaces O(n²) dependency scanning with O(1) dictionary lookups using three indexes (by class type, by class+method tuple, by method name)

AOT Compatibility

• No new reflection usage in hot paths ✅
• Uses source-generated metadata in AOT mode ✅
• Suppression attributes properly applied ✅

Code Quality

Interface Extension (ITestDataCollector.cs:17-24)
Clean addition of optional filter parameter while maintaining backwards compatibility. The new overload enables filter-aware collection without breaking existing callers.

DisplayName Fix (TestBuilderPipeline.cs:165-237)
Addresses previous test failures by properly handling IDynamicTestMetadata.DisplayName:

• Line 194: Extract dynamicTestMetadata once for consistency
• Line 197-199: Use custom DisplayName if available, falls back to TestName
• Line 233-236: Set DisplayName on context metadata

Dependency Expansion Optimization (MetadataDependencyExpander.cs:83-197)
Excellent optimization:

• Early return when no dependencies exist (lines 85-99)
• Three specialized indexes for different dependency types (lines 103-138)
• Proper fallback logic for generic types and inheritance (lines 152-164)

TestDiscoveryService Integration (TestDiscoveryService.cs:77-88)
The change from:

var allMetadata = await _testBuilderPipeline.CollectTestMetadataAsync(testSessionId);
var tests = await _testBuilderPipeline.BuildTestsStreamingAsync(..., metadataFilter: ..., ...)

To:

var allMetadata = await _testBuilderPipeline.CollectTestMetadataAsync(testSessionId, filter);
var tests = await _testBuilderPipeline.BuildTestsFromMetadataAsync(metadataToInclude, buildingContext, ...)

This avoids collecting metadata twice - once for filtering, once for building. Smart optimization.

Previous Review Status

All issues from previous reviews have been addressed:

• ✅ Method overload ambiguity fixed by consolidating into single method with CancellationToken parameter
• ✅ DisplayName support added to GenerateDynamicTests
• ✅ DynamicTestIndexTests and DynamicTests now passing (18/18 tests)

Minor Observations

1. Regex usage (MetadataFilterMatcher.cs:62): Uses System.Text.RegularExpressions.Regex.Replace to strip property filters. This is non-hot-path code so the AOT impact is negligible, but could theoretically be replaced with simple string operations if desired.

2. DynamicTestIndex in equality (MetadataDependencyExpander.cs:19-26): Good catch on including DynamicTestIndex in the equality comparer to distinguish multiple dynamic tests targeting the same method.

3. CancellationToken propagation (TestBuilderPipeline.cs:133): Proper cancellation support added to BuildTestsFromMetadataAsync.

Performance Impact

Per the PR description:

• Filtered run (301→1 types): ~1366ms → ~596ms (56% faster)
• Metadata collection: 847ms → 11ms (77x faster)

These are significant improvements that align with TUnit's performance-first principle.

Verdict

✅ APPROVE - Excellent performance optimization that follows all TUnit critical rules. The implementation is clean, maintains dual-mode compatibility, properly handles edge cases, and all identified issues from previous reviews have been resolved. The 56% improvement for filtered runs and 77x faster metadata collection are meaningful wins for the test framework.