perf: lazy test materialization with two-phase discovery

[thomhurst]

Summary

This PR implements lazy test materialization through a two-phase discovery architecture that significantly improves test discovery performance, especially when running filtered tests.

Key Changes

• TestDescriptor struct - Lightweight test descriptor containing only filter-relevant metadata (class name, method name, categories, properties, dependencies) without full test materialization
• ITestDescriptorSource interface - Enables test sources to provide fast descriptor enumeration via EnumerateTestDescriptors()
• Source generator updates - Generates EnumerateTestDescriptors() implementation with pre-computed filter hints extracted at compile time
• Two-phase discovery in pipeline - When filters are present:
  1. Enumerate lightweight descriptors
  2. Apply filter + expand dependencies at descriptor level
  3. Materialize only matching tests
• Descriptor-level dependency expansion - DependsOn property on descriptors enables dependency resolution before materialization, ensuring filtered tests include their dependencies

Performance Impact

When running filtered tests (e.g., --treenode-filter "/*/*/MyClass/MyTest"), only tests matching the filter (plus their dependencies) are fully materialized. This avoids the cost of:

• Instantiating data source attributes
• Evaluating data generators
• Creating full TestMetadata objects

For unfiltered runs, the traditional collection path is used.

Backward Compatibility

• Sources implementing only ITestSource continue to work (fallback to full materialization)
• No breaking changes to public APIs

Test plan

• All 290 engine tests pass (including FilteredDependencyTests)
• All 372 source generator snapshot tests pass
• Verified filtered test execution works correctly with dependencies
• Run benchmarks to measure performance improvement

🤖 Generated with Claude Code

[thomhurst]

Summary

Implements lazy test materialization through a two-phase discovery architecture (descriptors → filter → materialize) to improve performance when running filtered tests.

Critical Issues

None found ✅

Suggestions

1. Missing DependsOn information in ReflectionTestDataCollector descriptors

In ReflectionTestDataCollector.EnumerateDescriptors(), the generated descriptors have empty DependsOn arrays. This means dependency expansion in two-phase discovery won't work correctly for reflection mode.

Why this matters: The two-phase discovery in AotTestDataCollector.CollectTestsWithTwoPhaseDiscoveryAsync() relies on descriptor.DependsOn to expand dependencies before materialization. Without this, filtered tests won't include their dependencies in reflection mode.

Suggested fix: Either extract DependsOn values, document that dependency expansion is source-gen-only, or add a TODO comment explaining the limitation.

2. Consider ConfigureAwait consistency

In AotTestDataCollector.CollectTestsWithTwoPhaseDiscoveryAsync(), the code uses ConfigureAwait(false) but the method signature doesn't accept a CancellationToken parameter itself.

TUnit Rules Compliance

✅ Rule 1 - Dual-Mode: Both AotTestDataCollector and ReflectionTestDataCollector updated
✅ Rule 2 - Snapshot Testing: All 100 .verified.txt files updated correctly
✅ Rule 3 - No VSTest: No VSTest dependencies added
✅ Rule 4 - Performance First: This is a performance optimization PR
✅ Rule 5 - AOT Compatibility: Proper annotations on new methods

Verdict

✅ APPROVE - No critical issues

Excellent performance optimization that maintains backward compatibility. The architecture is clean, implementation is thorough, and all snapshot tests updated properly.

[thomhurst]

Summary

This PR implements lazy test materialization through a two-phase discovery architecture that defers creation of full TestMetadata objects until after filtering, improving performance when running filtered tests.

Critical Issues

None found - all TUnit critical rules are properly followed.

Suggestions

1. Dependency parsing uses IndexOf/Substring repeatedly. For complex dependency graphs, consider pre-parsing to tuples, though current approach is reasonable given dependencies are rare.

2. Property extraction assumes values do not contain equals signs. Consider documenting this limitation or using a more robust separator, though PropertyAttribute values are typically simple.

3. Reflection mode cannot pre-extract filter hints efficiently (documented limitation). Consider adding comment that it still avoids data source materialization which is the bigger win.

TUnit Rules Compliance

• Dual-Mode: Both source-gen and reflection implement ITestDescriptorSource correctly
• Snapshot Testing: All 372 .verified.txt files updated properly
• AOT Compatible: Proper DynamicallyAccessedMembers annotations present
• Performance First: Excellent - readonly structs, compile-time hints, static delegates
• No VSTest: Uses Microsoft.Testing.Platform exclusively

Architecture Highlights

• Backward compatible fallback for ITestSource without ITestDescriptorSource
• Descriptor-level dependency expansion before materialization
• Single-pass enumeration with indexing and filtering
• Conditional optimization only when filterHints.HasHints is true

Verdict

Approve - Excellent performance optimization with proper dual-mode support, AOT compatibility, and backward compatibility.

[thomhurst]

Summary

This PR implements lazy test materialization through a two-phase discovery architecture to improve test discovery performance when running filtered tests.

Critical Issues

None found ✅

Observations

Excellent Design ✅

1. Dual-mode implementation - Both source-gen (ITestDescriptorSource.EnumerateTestDescriptors) and reflection (ReflectionTestDataCollector.EnumerateDescriptors) paths are implemented, adhering to TUnit's dual-mode requirement.

2. Snapshot tests properly updated - All 97 .verified.txt files correctly show the new EnumerateTestDescriptors() method being generated. No .received.txt files committed.

3. AOT compatibility maintained - Proper [DynamicallyAccessedMembers] annotations present in reflection code paths.

4. No blocking async - All async operations use proper async/await patterns. No .Result, .Wait(), or .GetAwaiter().GetResult().

5. No VSTest dependencies - Exclusively uses Microsoft.Testing.Platform.

6. Performance-first design:

   • Lightweight readonly struct TestDescriptor minimizes allocations
   • Pre-computed filter hints (categories, properties, dependencies) extracted at compile time
   • Two-phase discovery avoids materializing tests that won't run
   • Dependency expansion at descriptor level before materialization
   • Sequential processing for small test sets (from PR perf: use sequential processing for small test sets #4299)

7. String escaping - Proper escaping in generated code (EscapeString handles backslashes, quotes, newlines, etc.)

8. Backward compatibility - Sources implementing only ITestSource continue to work via fallback to traditional collection path.

Code Quality

• Clean separation of concerns between descriptor enumeration and materialization
• Well-documented with XML comments explaining the two-phase approach
• Efficient dependency resolution using dictionaries for O(1) lookups
• Smart optimization: only expands dependencies if any matching descriptor has them

Verdict

✅ APPROVE - No critical issues. This is a well-designed performance optimization that follows all TUnit mandatory rules.

[thomhurst]

Summary

This PR implements lazy test materialization through a two-phase discovery architecture that improves test discovery performance when running filtered tests.

Critical Issues

None found ✅

Observations

TUnit Rules Compliance

✅ Dual-Mode Implementation - The PR correctly implements the feature in both:

• Source generator mode: TUnit.Core.SourceGenerator/Generators/TestMetadataGenerator.cs generates ITestDescriptorSource implementation with EnumerateTestDescriptors()
• Reflection mode: TUnit.Engine/Discovery/ReflectionTestDataCollector.cs implements EnumerateDescriptors() and MaterializeFromDescriptorsAsync()

✅ Snapshot Testing - All .verified.txt files updated correctly (372 source generator snapshots). No .received.txt files committed.

✅ Public API Changes - New public types properly tracked:

• TestDescriptor struct (TUnit.Core:11711-11870)
• ITestDescriptorSource interface (TUnit.Core/Interfaces/SourceGenerator:11624-11676)
• API snapshots updated for all target frameworks

✅ AOT Compatible - Proper [DynamicallyAccessedMembers] annotations on reflection code:

• CreateReflectionMaterializer (TUnit.Engine/Discovery/ReflectionTestDataCollector.cs:12289-12295)
• MaterializeSingleTestAsync (TUnit.Engine/Discovery/ReflectionTestDataCollector.cs:12297-12316)

✅ Performance First - Excellent optimization:

• Descriptors are lightweight structs
• Filter hints pre-computed at compile time (categories, properties, dependencies)
• Two-phase discovery only triggered when filters present
• Dependency expansion at descriptor level (before materialization)

Architecture Strengths

1. Backward Compatible - Sources implementing only ITestSource continue to work via fallback path
2. Smart Filtering - The CouldDescriptorMatch method (TUnit.Engine/Services/MetadataFilterMatcher.cs:12359-12384) correctly filters by class/method name before materialization
3. Dependency Handling - Pre-extracted DependsOn property enables dependency resolution at descriptor level (TUnit.Engine/Building/Collectors/AotTestDataCollector.cs:11960-12015)
4. Single-Pass Enumeration - Clever optimization in CollectTestsWithTwoPhaseDiscoveryAsync that indexes AND filters descriptors in one pass (TUnit.Engine/Building/Collectors/AotTestDataCollector.cs:11923-11958)

Minor Observations

1. Assembly Loader Optimization (TUnit.Core.SourceGenerator/CodeGenerators/AssemblyLoaderGenerator.cs:11149-11245) - Nice addition to only register assemblies that reference TUnit.Core, reducing reflection overhead

2. Reflection Mode Limitation - As documented, reflection mode cannot pre-extract filter hints without instantiating attributes. This is an acceptable trade-off and properly documented.

3. Test Coverage - PR description mentions "All 290 engine tests pass" and "All 372 source generator snapshot tests pass". The unchecked benchmark item is reasonable for a separate follow-up.

Verdict

✅ APPROVE - No critical issues

This is a well-architected performance optimization that:

• Follows all TUnit critical rules
• Maintains dual-mode compatibility
• Uses modern C# features appropriately
• Is properly documented
• Has comprehensive test coverage

The implementation demonstrates careful attention to performance, backward compatibility, and TUnit's architectural constraints.