Enable integration tests to run in parallel #388

avagraw · 2021-07-01T21:27:23Z

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Thank you for contributing to the IoT extension!

This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines

If introducing new functionality or modified behavior, are they backed by unit and integration tests?
In the same context as above are command names and their parameter definitions accurate? Do help docs have sufficient content?
Have all unit and integration tests passed locally? i.e. pytest <project root> -vv
Have static checks passed using the .pylintrc and .flake8 rules? Look at the CI scripts for example usage.
Have you made an entry in HISTORY.rst which concisely explains your feature or change?

…ure#341) * Introducing IoT Hub dataplane RBAC support. * IoT Hub rbac support for most commands. * Removed deprecated nested IoT edge artifacts. * Removed deprecated show-connection-string artifacts. * --auth-type supports configurable defaults. * Improved IoT Hub test infrastructure. * Significant refactor of IoT Hub tests. * Various additional improvements. * All warnings from tests are now shown. * Uamqp integration with jwt auth. * Modify HISTORY.rst

* add support for v1 and preview routes * update history file * address review comments * lint fixes

) * Add warning for qos deprecation and update contributing guide * removing version from deprecate_info * Integration tests command update in Contributing guide

* remove preview tags * history updates

…#344) * Fix for new identity parameter format * test updates * better differentiation of storage vars * updated to remove *all* user-identitites after storage tests until CLI core is patched

* pr comments * update dt reset to only support deleting both * Remove unused import

* Remove hiding of warnings from sample pytest.ini

* update to use wait * fix error status code * int testing * update help

* initial changes * update params, help * add missing module to test * Word change

* Pipeline updates Added nightly build pipeline Added template to run tests against minimum supported AZ CLI * Pipeline parameter refactoring * Moved deprecated ubuntu images to `ubuntu-latest` Co-authored-by: Paymaun <[email protected]>

… - update help to reflect the change (Azure#346) * Add warning for qos deprecation and update contributing guide * removing version from deprecate_info * Integration tests command update in Contributing guide * Update help message for d2c command and mqtt simulation to indicate only sas auth is supported * Checks and error messages when non SAS devices are used for MQTT operations * Indentation update in help file * fix styling * Adding enum and updating test * Addressed comments

IoTHub is not a MQTT Broker

* initial commit * add integration tests. * history file update. lint fixes * history updates * Update HISTORY.rst * address review comments Co-authored-by: Ryan K <[email protected]>

* Only system module content is validated by default. * The target schema used is with respect to "schemaVersion" in "properties.desired". * Any schema defintion processing issues will skip validation.

* generate sas token with all parsers * style fixes * update param, parser enum * updates with respect to pr * updated error message

* Initial DPS track 2 breaking change fixes * Test update to patch azure cli core Profile.get_subscription_id * revert change of unit test subscription id

Co-authored-by: Paymaun <[email protected]>

digimaun · 2021-07-22T18:26:07Z

We are looking for the tests to actually be run in parallel, rather than one at a time as your screenshots show. This would reveal naming or resource conflicts should they exist. Also please do not skip the storage identity tests. There are an assortment of updates to the tests and we need to ensure everything works.

azext_iot/tests/__init__.py

avagraw · 2021-07-22T18:33:04Z

We are looking for the tests to actually be run in parallel, rather than one at a time as your screenshots show. This would reveal naming or resource conflicts should they exist. Also please do not skip the storage identity tests. There are an assortment of updates to the tests and we need to ensure everything works.

The tests were run in parallel across multiple powershell windows.

avagraw · 2021-07-22T18:36:20Z

We are looking for the tests to actually be run in parallel, rather than one at a time as your screenshots show. This would reveal naming or resource conflicts should they exist. Also please do not skip the storage identity tests. There are an assortment of updates to the tests and we need to ensure everything works.

The tests were run in parallel across multiple powershell windows.

In the Azure DevOps pipeline this will be achieved by running a script which kicks off the pytest commands in parallel ( where each pytest command kicks of a different test suite)

digimaun · 2021-07-22T18:47:53Z

We are looking for the tests to actually be run in parallel, rather than one at a time as your screenshots show. This would reveal naming or resource conflicts should they exist. Also please do not skip the storage identity tests. There are an assortment of updates to the tests and we need to ensure everything works.

The tests were run in parallel across multiple powershell windows.

In the Azure DevOps pipeline this will be achieved by running a script which kicks off the pytest commands in parallel ( where each pytest command kicks of a different test suite)

Are you suggesting having to run a separate agent job per hub test type? This parallelization would primarily benefit the devops pipeline (at the cost of agents) as trying to execute pytest across local shell windows is not optimal devEx. Why not try to achieve parallelization which can also benefit local execution and not require numerous jobs?

azext_iot/tests/iothub/test_iothub_discovery_int.py

digimaun · 2021-07-22T19:42:46Z

We are looking for the tests to actually be run in parallel, rather than one at a time as your screenshots show. This would reveal naming or resource conflicts should they exist. Also please do not skip the storage identity tests. There are an assortment of updates to the tests and we need to ensure everything works.

The tests were run in parallel across multiple powershell windows.

In the Azure DevOps pipeline this will be achieved by running a script which kicks off the pytest commands in parallel ( where each pytest command kicks of a different test suite)

Are you suggesting having to run a separate agent job per hub test type? This parallelization would primarily benefit the devops pipeline (at the cost of agents) as trying to execute pytest across local shell windows is not optimal devEx. Why not try to achieve parallelization which can also benefit local execution and not require numerous jobs?

We talked about this and next steps offline.

avagraw · 2021-07-24T02:38:01Z

The following screenshots show tests running in parallel using 2 agents (poweshell processes running in parallel). There are flags we can setup to re-run failed tests and set time between re-runs ( the following commands were run with those flags). The re-run was not needed as all tests passed in first go here, but during my testing I was able to test the re-runs as well.

dev_requirements

azext_iot/tests/__init__.py

Merge from dev branch PR should be in different PR so we don't lose conversation / feedback history.

…o device_client_integration

azext_iot/tests/__init__.py

digimaun · 2021-07-27T18:36:43Z

azext_iot/tests/dps/test_iot_dps_int.py

        self.cmd_shell = Setting()
        setattr(self.cmd_shell, "cli_ctx", self.cli_ctx)

+        if hub != os.environ.get("azext_iot_testhub"):


IoT Hub tests uses DynamoSettings and the same could be used here to support uniform test plumbing across technology areas.

* Introducing IoT Hub dataplane RBAC support + various improvements (Azure#341) * Introducing IoT Hub dataplane RBAC support. * IoT Hub rbac support for most commands. * Removed deprecated nested IoT edge artifacts. * Removed deprecated show-connection-string artifacts. * --auth-type supports configurable defaults. * Improved IoT Hub test infrastructure. * Significant refactor of IoT Hub tests. * Various additional improvements. * All warnings from tests are now shown. * Uamqp integration with jwt auth. * Modify HISTORY.rst * Iotc command versioning (Azure#340) * add support for v1 and preview routes * update history file * address review comments * lint fixes * Add warning for qos deprecation and update contributing guide (Azure#342) * Add warning for qos deprecation and update contributing guide * removing version from deprecate_info * Integration tests command update in Contributing guide * Iotc command ga (Azure#348) * remove preview tags * history updates * Use enum value instead of literal str. (Azure#349) * Managed identity support for device-identity import and export (Azure#344) * Fix for new identity parameter format * test updates * better differentiation of storage vars * updated to remove *all* user-identitites after storage tests until CLI core is patched * Update azext_metadata.json (Azure#351) * Increment version to v0.10.13 * Update README.md * Update HISTORY.rst * Add dataplane reset (Azure#352) * pr comments * update dt reset to only support deleting both * Remove unused import * C2D messaging improvements. (Azure#354) * Remove hiding of warnings from sample pytest.ini * Digital Twin wait commands (Azure#345) * update to use wait * fix error status code * int testing * update help * Add Identity Storage Account ID param to sentinel values (Azure#355) * Module identity renew key (Azure#356) * initial changes * update params, help * add missing module to test * Word change * Update README.md * Pipeline updates (Azure#359) * Pipeline updates Added nightly build pipeline Added template to run tests against minimum supported AZ CLI * Pipeline parameter refactoring * Moved deprecated ubuntu images to `ubuntu-latest` Co-authored-by: Paymaun <[email protected]> * Check for conditionals before running test jobs (Azure#363) * Update d2c and simulate commands to return errors for non SaS devices - update help to reflect the change (Azure#346) * Add warning for qos deprecation and update contributing guide * removing version from deprecate_info * Integration tests command update in Contributing guide * Update help message for d2c command and mqtt simulation to indicate only sas auth is supported * Checks and error messages when non SAS devices are used for MQTT operations * Indentation update in help file * fix styling * Adding enum and updating test * Addressed comments * Update simulate message (Azure#371) IoTHub is not a MQTT Broker * Make gitHubConnection a variable. (Azure#373) * Iotc updates (Azure#368) * initial commit * add integration tests. * history file update. lint fixes * history updates * Update HISTORY.rst * address review comments Co-authored-by: Ryan K <[email protected]> * Edge deployment validation changes. (Azure#382) * Only system module content is validated by default. * The target schema used is with respect to "schemaVersion" in "properties.desired". * Any schema defintion processing issues will skip validation. * Couple linter fixes. (Azure#384) * Extend generate token (Azure#375) * generate sas token with all parsers * style fixes * update param, parser enum * updates with respect to pr * updated error message * Roll pipeline ubuntu versions back to Ubuntu-16.04 (Azure#386) * Support for upcoming changes in Track 2 SDKs (Azure#383) * Initial DPS track 2 breaking change fixes * Test update to patch azure cli core Profile.get_subscription_id * revert change of unit test subscription id * Remove six (Azure#387) Co-authored-by: Paymaun <[email protected]> * Separate IoT Hub per test suite to enable parallel test execution * Addressed comments * minor update * test environment pipeline updates * Update help for --login and --hub-name (Azure#389) * Update mock import. (Azure#391) * Refactoring and addressing comments * Update setup.py * fstrings fix * Formatting updates to tests * Create one hub per test suite and provide Data contributor role to user * Account refresh and test class parallel run * remove commented parts of test * lint updates * Merge with dev * Azure devops merge fix * Run integration tests in parallel * Update to tests Co-authored-by: Paymaun <[email protected]> Co-authored-by: valluriraj <[email protected]> Co-authored-by: Ryan K <[email protected]> Co-authored-by: Paymaun Heidari <[email protected]> Co-authored-by: vilit1 <[email protected]> Co-authored-by: Ricardo Minguez (Rido) <[email protected]>

digimaun and others added 29 commits May 17, 2021 14:51

Iotc command versioning (Azure#340)

624d849

* add support for v1 and preview routes * update history file * address review comments * lint fixes

Add warning for qos deprecation and update contributing guide (Azure#342

4bf999a

) * Add warning for qos deprecation and update contributing guide * removing version from deprecate_info * Integration tests command update in Contributing guide

Iotc command ga (Azure#348)

b268dbe

* remove preview tags * history updates

Use enum value instead of literal str. (Azure#349)

a4baacb

Managed identity support for device-identity import and export (Azure…

79bf6cb

…#344) * Fix for new identity parameter format * test updates * better differentiation of storage vars * updated to remove *all* user-identitites after storage tests until CLI core is patched

Update azext_metadata.json (Azure#351)

616e2c0

Increment version to v0.10.13

9ee1ddc

Update README.md

3f12f21

Update HISTORY.rst

058648a

Add dataplane reset (Azure#352)

759503a

* pr comments * update dt reset to only support deleting both * Remove unused import

C2D messaging improvements. (Azure#354)

f82ef12

* Remove hiding of warnings from sample pytest.ini

Digital Twin wait commands (Azure#345)

48af558

* update to use wait * fix error status code * int testing * update help

Add Identity Storage Account ID param to sentinel values (Azure#355)

354a98d

Module identity renew key (Azure#356)

b498a60

* initial changes * update params, help * add missing module to test * Word change

Update README.md

72fa744

Pipeline updates (Azure#359)

2ef0a41

* Pipeline updates Added nightly build pipeline Added template to run tests against minimum supported AZ CLI * Pipeline parameter refactoring * Moved deprecated ubuntu images to `ubuntu-latest` Co-authored-by: Paymaun <[email protected]>

Check for conditionals before running test jobs (Azure#363)

3170fd8

Update simulate message (Azure#371)

f36ca7b

IoTHub is not a MQTT Broker

Make gitHubConnection a variable. (Azure#373)

d88c94c

Iotc updates (Azure#368)

6738afb

* initial commit * add integration tests. * history file update. lint fixes * history updates * Update HISTORY.rst * address review comments Co-authored-by: Ryan K <[email protected]>

Edge deployment validation changes. (Azure#382)

a2adb11

* Only system module content is validated by default. * The target schema used is with respect to "schemaVersion" in "properties.desired". * Any schema defintion processing issues will skip validation.

Couple linter fixes. (Azure#384)

81c1153

Extend generate token (Azure#375)

5b30336

* generate sas token with all parsers * style fixes * update param, parser enum * updates with respect to pr * updated error message

Roll pipeline ubuntu versions back to Ubuntu-16.04 (Azure#386)

fdcdfbe

Support for upcoming changes in Track 2 SDKs (Azure#383)

0294711

* Initial DPS track 2 breaking change fixes * Test update to patch azure cli core Profile.get_subscription_id * revert change of unit test subscription id

Remove six (Azure#387)

e10bed4

Co-authored-by: Paymaun <[email protected]>

Separate IoT Hub per test suite to enable parallel test execution

8c396d2

avagraw requested a review from digimaun as a code owner July 1, 2021 21:27

digimaun reviewed Jul 22, 2021

View reviewed changes

azext_iot/tests/__init__.py Show resolved Hide resolved

digimaun reviewed Jul 22, 2021

View reviewed changes

azext_iot/tests/iothub/test_iothub_discovery_int.py Outdated Show resolved Hide resolved

digimaun reviewed Jul 22, 2021

View reviewed changes

azext_iot/tests/iothub/test_iothub_discovery_int.py Outdated Show resolved Hide resolved

avagraw added 3 commits July 23, 2021 19:44

Account refresh and test class parallel run

c9089d2

remove commented parts of test

828dfe6

lint updates

8c3a91c

digimaun reviewed Jul 26, 2021

View reviewed changes

dev_requirements Outdated Show resolved Hide resolved

digimaun reviewed Jul 26, 2021

View reviewed changes

azext_iot/tests/__init__.py Show resolved Hide resolved

Merge with dev

6081d3d

avagraw changed the title ~~Separate IoT Hub per test suite to enable parallel test execution~~ Merge with Dev branch Jul 27, 2021

digimaun previously approved these changes Jul 27, 2021

View reviewed changes

avagraw added 2 commits July 26, 2021 17:32

Azure devops merge fix

932e679

Run integration tests in parallel

51f0686

avagraw changed the title ~~Merge with Dev branch~~ Enable integration tests to run in parallel Jul 27, 2021

avagraw added 2 commits July 26, 2021 18:24

Merge remote-tracking branch 'upstream/device_client_integration' int…

2388e2b

…o device_client_integration

Update to tests

fc7ff94

digimaun reviewed Jul 27, 2021

View reviewed changes

azext_iot/tests/__init__.py Show resolved Hide resolved

digimaun reviewed Jul 27, 2021

View reviewed changes

digimaun approved these changes Jul 27, 2021

View reviewed changes

avagraw merged commit 599cf34 into Azure:device_client_integration Jul 27, 2021

c-ryan-k pushed a commit to c-ryan-k/azure-iot-cli-extension that referenced this pull request Apr 10, 2025

refactor: template refresh (Azure#388)

0b840f1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable integration tests to run in parallel #388

Enable integration tests to run in parallel #388

Uh oh!

avagraw commented Jul 1, 2021

Uh oh!

digimaun commented Jul 22, 2021 •

edited

Loading

Uh oh!

Uh oh!

avagraw commented Jul 22, 2021

Uh oh!

avagraw commented Jul 22, 2021

Uh oh!

digimaun commented Jul 22, 2021

Uh oh!

Uh oh!

Uh oh!

digimaun commented Jul 22, 2021

Uh oh!

avagraw commented Jul 24, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

digimaun Jul 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Enable integration tests to run in parallel #388

Enable integration tests to run in parallel #388

Uh oh!

Conversation

avagraw commented Jul 1, 2021

General Guidelines

Uh oh!

digimaun commented Jul 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

avagraw commented Jul 22, 2021

Uh oh!

avagraw commented Jul 22, 2021

Uh oh!

digimaun commented Jul 22, 2021

Uh oh!

Uh oh!

Uh oh!

digimaun commented Jul 22, 2021

Uh oh!

avagraw commented Jul 24, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

digimaun Jul 27, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

digimaun commented Jul 22, 2021 •

edited

Loading