Skip to content

[azsdk-cli] Add compile-time analyzer to enforce Tool method return types#11565

Merged
benbp merged 11 commits intomainfrom
copilot/fix-11564
Aug 7, 2025
Merged

[azsdk-cli] Add compile-time analyzer to enforce Tool method return types#11565
benbp merged 11 commits intomainfrom
copilot/fix-11564

Conversation

Copy link
Contributor

Copilot AI commented Aug 6, 2025

This PR implements a new Roslyn analyzer to enforce that all public non-static methods in the Azure.Sdk.Tools.Cli.Tools namespace return only approved types at compile time.

Problem

Tool methods were returning inconsistent types, including raw collections and custom objects that don't follow the established response pattern. This made the API inconsistent and harder to consume.

Solution

Added a new analyzer EnforceToolsReturnTypesAnalyzer (MCP003) that enforces the following return type rules:

Allowed Return Types

  1. Response Types: Classes implementing Azure.Sdk.Tools.Cli.Models.Response
    • ValidationResponse, DefaultCommandResponse, LogAnalysisResponse, etc.
  2. Built-in Value Types: string, int, bool, DateTime, etc.
  3. Async Variations: Task<T> and ValueTask<T> where T is any allowed type above
  4. Void Async: Task for async methods that don't return values

Analyzer Features

  • Automatically runs during compilation with clear error messages
  • Excludes framework methods like GetCommand() and HandleCommand()
  • Excludes abstract/virtual/override methods from base classes
  • Properly handles inheritance chains to detect Response implementations
  • Supports both generic and non-generic Task types

Current Violations Detected

The analyzer identifies 6 methods that need to be updated:

  • Methods returning List<FailedTestRunResponse> should return a Response containing the list
  • Methods returning List<int> or List<string> should return a Response containing the collection
  • Methods returning IList<string> should return a Response containing the collection

Example

// ❌ Before - Invalid return type
public async Task<List<int>> GetIds() { }

// ✅ After - Valid return type  
public async Task<IdsResponse> GetIds() { }

public class IdsResponse : Response 
{
    public List<int> Ids { get; set; }
}

The analyzer provides immediate feedback with error message:

error MCP003: Method 'GetIds' in Tools namespace must return a class implementing Response, a built-in value type, or string. Current return type: 'System.Collections.Generic.List<int>'.

All existing tests pass and the analyzer correctly validates both positive and negative cases without false positives.

Fixes #11564.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits August 6, 2025 20:16
Co-authored-by: benbp <1020379+benbp@users.noreply.github.com>
Co-authored-by: benbp <1020379+benbp@users.noreply.github.com>
Copilot AI changed the title [WIP] [azsdk-cli] Ensure all Tool responses are of a certain type [azsdk-cli] Add compile-time analyzer to enforce Tool method return types Aug 6, 2025
Copilot AI requested a review from benbp August 6, 2025 20:19
Co-authored-by: benbp <1020379+benbp@users.noreply.github.com>
Co-authored-by: benbp <1020379+benbp@users.noreply.github.com>
@benbp benbp added the azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli label Aug 7, 2025
@benbp benbp force-pushed the copilot/fix-11564 branch from 6e5d6c8 to d2b51bc Compare August 7, 2025 22:00
Copy link
Member

@praveenkuttappan praveenkuttappan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@benbp benbp marked this pull request as ready for review August 7, 2025 22:15
Copilot AI review requested due to automatic review settings August 7, 2025 22:15
@benbp benbp requested review from jeo02, maririos and smw-ms as code owners August 7, 2025 22:15
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a new Roslyn analyzer to enforce that all public non-static methods in the Azure.Sdk.Tools.Cli.Tools namespace return only approved types at compile time, addressing the issue of inconsistent return types across tool methods.

  • Adds new EnforceToolsReturnTypesAnalyzer (MCP003) to validate tool method return types
  • Restructures tool namespaces to consolidate all tools under Azure.Sdk.Tools.Cli.Tools
  • Updates .editorconfig to enforce braces on single-line if statements

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Analyzer/EnforceToolsReturnTypes.cs New analyzer implementation that validates tool method return types
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Analyzer/README.md Documentation for the new MCP003 analyzer with usage examples
tools/azsdk-cli/Azure.Sdk.Tools.Cli/Tools/*/Tool.cs Namespace consolidation moving all tools to Azure.Sdk.Tools.Cli.Tools
tools/azsdk-cli/Azure.Sdk.Tools.Cli/Commands/SharedOptions.cs Removal of specific tool namespace imports after consolidation
Multiple tool and service files Code style updates adding braces to single-line if statements
tools/azsdk-cli/.editorconfig Style enforcement configuration updates
Test files Updated imports to reflect namespace changes
tools/azsdk-cli/Azure.Sdk.Tools.Cli/Azure.Sdk.Tools.Cli.csproj Enabled code style enforcement in build

case SpecialType.System_String:
case SpecialType.System_Boolean:
case SpecialType.System_Byte:
case SpecialType.System_Char: // NOTE: this seems to be matching against 'string' for some reason
Copy link

Copilot AI Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment suggests System_Char is matching against 'string', which indicates a potential logic error. System_Char should match char types, not string. If this is unexpected behavior, the logic needs investigation and the comment should be clarified or the case removed if incorrect.

Suggested change
case SpecialType.System_Char: // NOTE: this seems to be matching against 'string' for some reason

Copilot uses AI. Check for mistakes.
Comment on lines +102 to +103
(namedType.ConstructedFrom?.ToDisplayString() == "System.Threading.Tasks.Task<T>" ||
namedType.Name == "Task" && namedType.ContainingNamespace?.ToDisplayString() == "System.Threading.Tasks"))
Copy link

Copilot AI Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The Task detection logic checks both the constructed form string and separate name/namespace conditions. This is redundant and could be simplified to use either the ToDisplayString() approach or the name/namespace approach consistently.

Suggested change
(namedType.ConstructedFrom?.ToDisplayString() == "System.Threading.Tasks.Task<T>" ||
namedType.Name == "Task" && namedType.ContainingNamespace?.ToDisplayString() == "System.Threading.Tasks"))
namedType.ConstructedFrom?.ToDisplayString() == "System.Threading.Tasks.Task<T>")

Copilot uses AI. Check for mistakes.
Comment on lines +202 to +208
var ienumerableInterface = returnType.AllInterfaces.FirstOrDefault(
i => i.IsGenericType &&
i.ConstructedFrom?.ToDisplayString() == "System.Collections.Generic.IEnumerable<T>");

if (ienumerableInterface != null && ienumerableInterface.TypeArguments.Length > 0)
{
var elementType = ienumerableInterface.TypeArguments[0];
Copy link

Copilot AI Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using FirstOrDefault() with a complex predicate can be inefficient. Consider using a more specific lookup or caching the constructed generic type definition for IEnumerable to improve performance when analyzing large codebases.

Suggested change
var ienumerableInterface = returnType.AllInterfaces.FirstOrDefault(
i => i.IsGenericType &&
i.ConstructedFrom?.ToDisplayString() == "System.Collections.Generic.IEnumerable<T>");
if (ienumerableInterface != null && ienumerableInterface.TypeArguments.Length > 0)
{
var elementType = ienumerableInterface.TypeArguments[0];
var ienumerableTypeSymbol = compilation.GetTypeByMetadataName("System.Collections.Generic.IEnumerable`1");
if (ienumerableTypeSymbol == null)
{
return false;
}
var ienumerableInterface = returnType.AllInterfaces
.FirstOrDefault(i => i is INamedTypeSymbol named &&
named.IsGenericType &&
SymbolEqualityComparer.Default.Equals(named.OriginalDefinition, ienumerableTypeSymbol));
if (ienumerableInterface != null && ((INamedTypeSymbol)ienumerableInterface).TypeArguments.Length > 0)
{
var elementType = ((INamedTypeSymbol)ienumerableInterface).TypeArguments[0];

Copilot uses AI. Check for mistakes.
@benbp benbp added the Central-EngSys This issue is owned by the Engineering System team. label Aug 7, 2025
@benbp benbp moved this from 🤔 Triage to 🔬 Dev in PR in Azure SDK EngSys 📆🎇 Aug 7, 2025
@benbp benbp merged commit 1d438cc into main Aug 7, 2025
16 checks passed
@benbp benbp deleted the copilot/fix-11564 branch August 7, 2025 22:16
@kurtzeborn kurtzeborn moved this from 🔬 Dev in PR to 🎊 Closed in Azure SDK EngSys 📆🎇 Aug 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli Central-EngSys This issue is owned by the Engineering System team.

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

[azsdk-cli] Ensure all Tool responses are of a certain type

5 participants