az storage blob sync throws exception

[Bessonov]

Describe the bug

az storage blob sync throws exception below.

To Reproduce

az login --service-principal --username $AZ_BO_CLIENT_ID --password $AZ_BO_CLIENT_SECRET --tenant $AZ_BO_TENANT_ID

[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "...",
    "id": "...",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Azure-Abonnement",
    "state": "Enabled",
    "tenantId": "...",
    "user": {
      "name": "...",
      "type": "servicePrincipal"
    }
  }
]

Following command worked at Feb 5, 2021 12:48pm GMT+0000:

az storage blob sync --source ./frontend-sources/${RELEASE_VERSION} --account-name $AZ_BO_STORAGE_ACCOUNT_NAME --container '$web' --destination $RELEASE_VERSION
WARNING: This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
WARNING: There are no credentials provided in your command and environment, we will query for the account key inside your storage account. 
Please provide --connection-string, --account-key or --sas-token as credentials, or use `--auth-mode login` if you have required RBAC roles in your command. For more information about RBAC roles in storage, visit https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli. 
Setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information.
WARNING: Azcopy command: ['/root/bin/azcopy', 'sync', './frontend-sources/...', 'https://.....blob.core.windows.net/$web/...', '--delete-destination=true']
INFO: azcopy: A newer version 10.8.0 is available to download
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support
Job 651a476c-cce4-da4e-7b16-161b80e728d5 has started
Log file is located at: /root/.azcopy/651a476c-cce4-da4e-7b16-161b80e728d5.log
0 Files Scanned at Source, 0 Files Scanned at Destination
Job 651a476c-cce4-da4e-7b16-161b80e728d5 Summary
Files Scanned at Source: 21
Files Scanned at Destination: 0
Elapsed Time (Minutes): 0.0333
Number of Copy Transfers for Files: 21
Number of Copy Transfers for Folder Properties: 0 
Total Number Of Copy Transfers: 21
Number of Copy Transfers Completed: 21
Number of Copy Transfers Failed: 0
Number of Deletions at Destination: 0
Total Number of Bytes Transferred: 7089879
Total Number of Bytes Enumerated: 7089879
Final Job Status: Completed

But it stops working at Feb 10, 2021 9:46am GMT+0000:

az storage blob sync --source ./frontend-sources/${RELEASE_VERSION} --account-name $AZ_BO_STORAGE_ACCOUNT_NAME --container '$web' --destination $RELEASE_VERSION
WARNING: This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
WARNING: There are no credentials provided in your command and environment, we will query for the account key inside your storage account. 
Please provide --connection-string, --account-key or --sas-token as credentials, or use `--auth-mode login` if you have required RBAC roles in your command. For more information about RBAC roles in storage, visit https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli. 
Setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information.
ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: MSI auth not yet supported.
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 122, in client_auth_for_azcopy
    token_info = _unserialize_non_msi_token_payload(token_info)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 144, in _unserialize_non_msi_token_payload
    'refresh_token': token_info['refreshToken'],
KeyError: 'refreshToken'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 664, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 727, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 719, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/__init__.py", line 338, in new_handler
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 698, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 331, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 816, in default_command_handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/azcopy.py", line 65, in storage_blob_sync
    azcopy = _azcopy_blob_client(cmd, client)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/azcopy.py", line 92, in _azcopy_blob_client
    return AzCopy(creds=client_auth_for_azcopy(cmd, client))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 124, in client_auth_for_azcopy
    raise Exception('MSI auth not yet supported.')
Exception: MSI auth not yet supported.
To open an issue, please run: 'az feedback'

If I add --auth-mode login I get:

az storage blob sync --auth-mode login --source test --account-name $AZ_BO_STORAGE_ACCOUNT_NAME --container '$web' --destination test
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
The command failed with an unexpected error. Here is the traceback:
MSI auth not yet supported.
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 122, in client_auth_for_azcopy
    token_info = _unserialize_non_msi_token_payload(token_info)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 144, in _unserialize_non_msi_token_payload
    'refresh_token': token_info['refreshToken'],
KeyError: 'refreshToken'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 664, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 727, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 719, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/__init__.py", line 338, in new_handler
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 698, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 331, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 816, in default_command_handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/azcopy.py", line 65, in storage_blob_sync
    azcopy = _azcopy_blob_client(cmd, client)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/azcopy.py", line 92, in _azcopy_blob_client
    return AzCopy(creds=client_auth_for_azcopy(cmd, client))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 124, in client_auth_for_azcopy
    raise Exception('MSI auth not yet supported.')
Exception: MSI auth not yet supported.
To open an issue, please run: 'az feedback'

Expected behavior
Files are copied.

Environment Summary

Linux-4.15.0-65-generic-x86_64-with-debian-buster-sid
Python 3.6.10
Installer: DEB

azure-cli 2.19.1

Additional context
The above command is used on CI/CD pipeline.

From local testing:
Last worked version: 2.18.0-1~bionic
First don't working version: 2.19.0-1~bionic

[yonzhan]

storage

[miketimofeev]

Same here on version 2.19.1 when trying to execute az storage remove --account-name $StorageAccountSrc -c $containerName -n $blobName. Authorization was made by az login --service-principal --username $ClientId --password $ClientSecret --tenant $TenantId

2021-02-15T12:33:14.5652777Z WARNING: This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
2021-02-15T12:33:14.5655120Z WARNING: There are no credentials provided in your command and environment, we will query for the account key inside your storage account. 
2021-02-15T12:33:14.5656668Z Please provide --connection-string, --account-key or --sas-token as credentials, or use `--auth-mode login` if you have required RBAC roles in your command. For more information about RBAC roles in storage, visit https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli. 
2021-02-15T12:33:14.5658096Z Setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information.
2021-02-15T12:33:16.0344134Z ERROR: The command failed with an unexpected error. Here is the traceback:
2021-02-15T12:33:16.0375436Z ERROR: MSI auth not yet supported.
2021-02-15T12:33:16.0376587Z Traceback (most recent call last):
2021-02-15T12:33:16.0396320Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-7as_bwui\azure\cli\command_modules\storage\azcopy\util.py", line 122, in client_auth_for_azcopy
2021-02-15T12:33:16.0398160Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-7as_bwui\azure\cli\command_modules\storage\azcopy\util.py", line 144, in _unserialize_non_msi_token_payload
2021-02-15T12:33:16.0399183Z KeyError: 'refreshToken'
2021-02-15T12:33:16.0399411Z 
2021-02-15T12:33:16.0400246Z During handling of the above exception, another exception occurred:
2021-02-15T12:33:16.0400620Z 
2021-02-15T12:33:16.0401189Z Traceback (most recent call last):
2021-02-15T12:33:16.0401963Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-yccaebu4\knack\cli.py", line 233, in invoke
2021-02-15T12:33:16.0404675Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-8u6uejh7\azure\cli\core\commands\__init__.py", line 664, in execute
2021-02-15T12:33:16.0405784Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-8u6uejh7\azure\cli\core\commands\__init__.py", line 727, in _run_jobs_serially
2021-02-15T12:33:16.0406914Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-8u6uejh7\azure\cli\core\commands\__init__.py", line 720, in _run_job
2021-02-15T12:33:16.0407850Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-rg200gj8\six.py", line 703, in reraise
2021-02-15T12:33:16.0408850Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-8u6uejh7\azure\cli\core\commands\__init__.py", line 698, in _run_job
2021-02-15T12:33:16.0409956Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-8u6uejh7\azure\cli\core\commands\__init__.py", line 331, in __call__
2021-02-15T12:33:16.0411302Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-8u6uejh7\azure\cli\core\__init__.py", line 816, in default_command_handler
2021-02-15T12:33:16.0412456Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-7as_bwui\azure\cli\command_modules\storage\operations\azcopy.py", line 45, in storage_remove
2021-02-15T12:33:16.0415412Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-7as_bwui\azure\cli\command_modules\storage\operations\azcopy.py", line 92, in _azcopy_blob_client
2021-02-15T12:33:16.0416560Z   File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-7as_bwui\azure\cli\command_modules\storage\azcopy\util.py", line 124, in client_auth_for_azcopy
2021-02-15T12:33:16.0417498Z Exception: MSI auth not yet supported.
2021-02-15T12:33:16.0921995Z To open an issue, please run: 'az feedback'

[ChristianUlbrich]

We have the same issue; looks like MS hosted agents did an update to 2.19.1(today/yesterday's enterprise) and thus now all builds using azure-cli will break.

This effectively breaks all of our pipelines. :(

[miketimofeev]

@ChristianUlbrich unfortunately, yes, we have updated hosted win 2019 environments with 2.19.1 version. As a workaround, you can use the following steps to install version 2.18.0, but I'm afraid it takes about 12 minutes

choco uninstall azure-cli
choco install azure-cli --version 2.18.0

[SaulLoachamin]

I tried with azure cli 2.1.0 and is working in the MS vs2017-win2016.

[ChristianUlbrich]

@SaulLoachamin I think you are referring to the AzureDevOps task AzureCLI; but it is misleading; it does not include the azure-cli itself, but is just "glue" around it. The actual azure-cli is a prerequisite. All hosted build agents have it pre-installed. In the latest windows-latest image, this seems to have been updated quite recently (seemingly on 2021-02-15). Current ubuntu-latest still has the old version 2.18 that does not seem to have the regression from 2.19.1, at least for now. This is the work-around I am using right now.

@miketimofeev Deploying a broken azure-clito your build agents, will cause a lot of problems for a lot of users. I can only vouch for either a hotfix or a rollback. A workaround adding "12 minutes" to a build, does not seem to be a feasible solution to me. Thanks for confirmation though!

[viktoreinarsson]

This is breaking all of our ARM template deployments. Since we cannot specify a fixed Azure CLI/Task version this is broken for us until a rollback/hotfix is deployed.