Skip to content

Conversation

@fengzhou-msft
Copy link
Member

@fengzhou-msft fengzhou-msft commented Dec 10, 2020

Description

Make some tests record_only as they rely on existing resources in another subscription or require extra permissions such as reservation purchases.

Add retry for reimporting network private-dns zone as we are seeing more failures due to resource privioning not finished before calling the command.

Testing Guide

Test results: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=648963&view=artifacts&pathAsName=false&type=publishedArtifacts

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.


This checklist is used to make sure that common guidelines for a pull request are followed.

@qwordy
Copy link
Member

qwordy commented Dec 10, 2020

Skip does not fix them. Should we skip test that we can't fix to let data be beautiful?

Copy link
Contributor

@haroldrandom haroldrandom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_aro_delete Could you rerun the test? Or replace the network API version to fix CI?

@fengzhou-msft
Copy link
Member Author

Skip does not fix them. Should we skip test that we can't fix to let data be beautiful?

These tests rely on existing resources of another subscription and when the live test pipeline tests them with our subscription, it would have resource not found error or permission error.

The best way to fix it is to rewrite the tests to create the resources during the test if the resource does not require extra permissions or we apply for relative permissions (we have fixed some tests in this way before).

The second best way is to create resources in our subscription and lock them for usage in live tests if the creation of such resources is very time-consuming.

The least we want to do is to skip the test, but it's difficult to fix some tests due to permission issues or complexity. The failure of such tests does not tell us the real issues. I think it's OK to skip them now. When we or service teams have more resources, we can rewrite those tests. I think we will evaulate test coverage sometime in the future.

@fengzhou-msft
Copy link
Member Author

test_aro_delete Could you rerun the test? Or replace the network API version to fix CI?

submitted #16226

@fengzhou-msft fengzhou-msft removed the request for review from v-Ajnava December 10, 2020 10:33
@fengzhou-msft fengzhou-msft changed the title {Test} Fix or skip live tests for cdn, eventhubs, netappfiles, privatedns, reservations {Test} Fix or skip live tests for cdn, privatedns, reservations Dec 10, 2020
@fengzhou-msft fengzhou-msft merged commit 66f7bf2 into Azure:dev Dec 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants