Management API Readiness - feedback from experience with service partner and bugs/function not working

[ladonnaq]

I was working with service partner today and thought it would be a good opportunity to show them the release planner beta for management plane. They are using Cadl so perfect team to use the release planner app.

This was our experience in attempting to use the API Readiness App for management plane.

1. When we got to the API Readiness step for Cadl validation all of the status was showing as unavailable. I am not sure if this is because the app is not working or if it is something to do with the PR they provided.

2. When we selected details - we don't end up exactly where the information related to what is being validated. For example, select details next Cadl validation directs us here.

3. When I expand the "openapi-pipeline-app" and search, I do find cadl validation and the message said it succeeded.

4. Adding initial package for readme verification. #1, Adding status badges #2, and Readme Content Verification #3 apply to the Review breaking changes and legacy spec issues screen/blade as well.
5. Setup API docs is also displaying unavailable but if I go look in the PR, I found that is available. Here is the link - https://review.learn.microsoft.com/en-us/rest/api/azure-rest-preview/?branch=result_12688ab9-a7ed-4e44-9da6-e1bf5f6fd08b&view=azure-rest-preview
   
6. The service partner engineers had never seen this information before and did not know what to do with it.
   
7. They found this page confusing. The engineers are working a new API version in the private repo for Public Preview. There is another API Version that was approved by ARM for Private Preview and they met with ARM Review Board. Also, the bot adds the label to initiate the ARM review.
   

8. Generate SDK screen/blade had no data.
9. There is no information to explain why something failed and where to go to fix it.

• The service partner told me that there would be new initial management libraries, so naming is required but in the app it status displays that it is not required.
• The screen displays SDK generation is completed but on the Generate SDK screen it was blank (no SDKs seen).
• Spec validation displays failed but there is no mention of the Spec anywhere else. Only Cadl validation.

[live1206]

1. When we got to the API Readiness step for Cadl validation all of the status was showing as unavailable. I am not sure if this is because the app is not working or if it is something to do with the PR they provided.

This PR was created from forked repo to private repo, this is not supported in OpenAPI Hub anymore, but the user can still manually create this kind of PR. The implementation of getting validation results in Release Planner App is via OpenAPI Hub, so this kind of PR is not supported. However, I think this is a valid case for Release Planner to support.
@honghr We need to figure out a way to get the validation result for PR created from forked repo.

[live1206]

2. When we selected details - we don't end up exactly where the information related to what is being validated. For example, select details next Cadl validation directs us here.

This is the same reason as the previous issue, we could not get CI result from PR of forked repo, the link is missing check_run_id.

3. When I expand the "openapi-pipeline-app" and search, I do find cadl validation and the message said it succeeded.

This is the same reason as the previous issue.

4. Adding initial package for readme verification. #1, Adding status badges #2, and Readme Content Verification #3 apply to the Review breaking changes and legacy spec issues screen/blade as well.

The mentioned PRs above are all created from forked repo.

5. Setup API docs is also displaying unavailable but if I go look in the PR, I found that is available. Here is the link - https://review.learn.microsoft.com/en-us/rest/api/azure-rest-preview/?branch=result_12688ab9-a7ed-4e44-9da6-e1bf5f6fd08b&view=azure-rest-preview

The same reason as previous, unable to get CI result from PR of forked repo.

[live1206]

6. The service partner engineers had never seen this information before and did not know what to do with it.
   

@josefree I know you had some internal discussion regarding this, any updates?

[live1206]

7. They found this page confusing. The engineers are working a new API version in the private repo for Public Preview. There is another API Version that was approved by ARM for Private Preview and they met with ARM Review Board. Also, the bot adds the label to initiate the ARM review.
   
   

In Management-Plane Api Readiness App, we are trying to guide the user step by step of PR review.
If the PR is labelled with CI-FixRequiredOnFailure, we would expect the user to fix CI failures before ARM review.
And you are right that, we don't stop ARM Review Board from reviewing the PR right now.
@ladonnaq Do you think we should add ARMReview label after CI failures are fixed?

cc @josefree

[live1206]

8. Generate SDK screen/blade had no data.

The SDK generation data is based on OpenAPI hub result as well, it is not working for PR of forked repo.

[live1206]

9. There is no information to explain why something failed and where to go to fix it.

• The service partner told me that there would be new initial management libraries, so naming is required but in the app it status displays that it is not required.

Right now, if the user did not fill in the namespace review issue, the status of Library Namespace Review is Not Required.
It should be unavailable instead. @honghr

• The screen displays SDK generation is completed but on the Generate SDK screen it was blank (no SDKs seen).

For empty SDK generation, the status should be unavailable. @honghr

• Spec validation displays failed but there is no mention of the Spec anywhere else. Only Cadl validation.
  

The user needs to go back to previous step Clean up Validation Errors from the left navigation panel to get all the validation details.

[ladonnaq]

7. They found this page confusing. The engineers are working a new API version in the private repo for Public Preview. There is another API Version that was approved by ARM for Private Preview and they met with ARM Review Board. Also, the bot adds the label to initiate the ARM review.
   
   

In Management-Plane Api Readiness App, we are trying to guide the user step by step of PR review. If the PR is labelled with CI-FixRequiredOnFailure, we would expect the user to fix CI failures before ARM review. And you are right that, we don't stop ARM Review Board from reviewing the PR right now. @ladonnaq Do you think we should add ARMReview label after CI failures are fixed?

cc @josefree

I do think we should include the label after CI failures are fixed. Thanks!