Invoke-AzureRmVMRunCommand returns empty

[mcocak]

Description

Invoke-AzureRmVMRunCommand started returning empty after execution. Until last week, the command printed the stdout of the remotely executed command on the VM. However, it is now returning empty string.

As you see in the debug output below, I'm executing a basic "docker ps" command on the remote Linux VM. The command is for sure executed remotely and the output of the command is visible in the HTTP response from Azure, however, Invoke-AzureRmVMRunCommand output is empty.

Script/Steps for Reproduction

Invoke-AzureRmVMRunCommand -ResourceGroupName $resourceGroup -Name $vmName -CommandId 'RunShellScript' -ScriptPath $scriptFile

Module Version

Get-Module -Name AzureRM -ListAvailable
ModuleType Version    Name                                ExportedCommands                                                                      
---------- -------    ----                                ----------------                                                                      
Script     0.11.0     AzureRM.Aks.Netcore                 {Get-AzureRmAks, New-AzureRmAks, Remove-AzureRmAks, Import-AzureRmAksCredential...}   
Script     0.11.0     AzureRM.AnalysisServices.Netcore    {Resume-AzureRmAnalysisServicesServer, Suspend-AzureRmAnalysisServicesServer, Get-A...
Script     0.11.0     AzureRM.ApiManagement.Netcore       {Add-AzureRmApiManagementRegion, Get-AzureRmApiManagementSsoToken, New-AzureRmApiMa...
Script     0.11.0     AzureRM.ApplicationInsights.Netcore {Get-AzureRmApplicationInsights, New-AzureRmApplicationInsights, Remove-AzureRmAppl...
Script     0.11.0     AzureRM.Automation.Netcore          {Get-AzureRMAutomationHybridWorkerGroup, Get-AzureRmAutomationJobOutputRecord, Impo...
Script     0.11.0     AzureRM.Backup.Netcore              {Backup-AzureRmBackupItem, Enable-AzureRmBackupContainerReregistration, Get-AzureRm...
Script     0.11.0     AzureRM.Batch.Netcore               {Remove-AzureRmBatchAccount, Get-AzureRmBatchAccount, Get-AzureRmBatchAccountKeys, ...
Script     0.11.0     AzureRM.Billing.Netcore             {Get-AzureRmBillingInvoice, Get-AzureRmBillingPeriod, Get-AzureRmEnrollmentAccount}   
Script     0.11.0     AzureRM.Cdn.Netcore                 {Get-AzureRmCdnProfile, Get-AzureRmCdnProfileSsoUrl, New-AzureRmCdnProfile, Remove-...
Script     0.11.0     AzureRM.CognitiveServices.Netcore   {Get-AzureRmCognitiveServicesAccount, Get-AzureRmCognitiveServicesAccountKey, Get-A...
Script     0.11.0     AzureRM.Compute.Netcore             {Remove-AzureRmAvailabilitySet, Get-AzureRmAvailabilitySet, New-AzureRmAvailability...
Script     0.10.0     AzureRM.Compute.Netcore             {Remove-AzureRmAvailabilitySet, Get-AzureRmAvailabilitySet, New-AzureRmAvailability...
Script     0.11.0     AzureRM.Consumption.Netcore         Get-AzureRmConsumptionUsageDetail                                                     
Script     0.11.0     AzureRM.ContainerInstance.Netcore   {New-AzureRmContainerGroup, Get-AzureRmContainerGroup, Remove-AzureRmContainerGroup...
Script     0.11.0     AzureRM.ContainerRegistry.Netcore   {New-AzureRmContainerRegistry, Get-AzureRmContainerRegistry, Update-AzureRmContaine...
Script     0.11.0     AzureRM.DataFactories.Netcore       {Remove-AzureRmDataFactory, Get-AzureRmDataFactoryRun, Get-AzureRmDataFactorySlice,...
Script     0.11.0     AzureRM.DataFactoryV2.Netcore       {Set-AzureRmDataFactoryV2, Update-AzureRmDataFactoryV2, Get-AzureRmDataFactoryV2, R...
Script     0.11.0     AzureRM.DataLakeAnalytics.Netcore   {Get-AzureRmDataLakeAnalyticsDataSource, New-AzureRmDataLakeAnalyticsCatalogCredent...
Script     0.11.0     AzureRM.DataLakeStore.Netcore       {Get-AzureRmDataLakeStoreTrustedIdProvider, Remove-AzureRmDataLakeStoreTrustedIdPro...
Script     0.11.0     AzureRM.DevTestLabs.Netcore         {Get-AzureRmDtlAllowedVMSizesPolicy, Get-AzureRmDtlAutoShutdownPolicy, Get-AzureRmD...
Script     0.11.0     AzureRM.Dns.Netcore                 {Get-AzureRmDnsRecordSet, New-AzureRmDnsRecordConfig, Remove-AzureRmDnsRecordSet, S...
Script     0.11.0     AzureRM.EventGrid.Netcore           {New-AzureRmEventGridTopic, Get-AzureRmEventGridTopic, Set-AzureRmEventGridTopic, N...
Script     0.11.0     AzureRM.EventHub.Netcore            {New-AzureRmEventHubNamespace, Get-AzureRmEventHubNamespace, Set-AzureRmEventHubNam...
Script     0.11.0     AzureRM.Insights.Netcore            {Get-AzureRmMetricDefinition, Get-AzureRmMetric, Remove-AzureRmLogProfile, Get-Azur...
Script     0.11.0     AzureRM.IotHub.Netcore              {Add-AzureRmIotHubKey, Get-AzureRmIotHubEventHubConsumerGroup, Get-AzureRmIotHubCon...
Script     0.11.0     AzureRM.KeyVault.Netcore            {Add-AzureKeyVaultCertificate, Set-AzureKeyVaultCertificateAttribute, Stop-AzureKey...
Script     0.10.0     AzureRM.KeyVault.Netcore            {Add-AzureKeyVaultCertificate, Set-AzureKeyVaultCertificateAttribute, Stop-AzureKey...
Script     0.11.0     AzureRM.LogicApp.Netcore            {Get-AzureRmIntegrationAccountAgreement, Get-AzureRmIntegrationAccountCallbackUrl, ...
Script     0.11.0     AzureRM.MachineLearning.Netcore     {Move-AzureRmMlCommitmentAssociation, Get-AzureRmMlCommitmentAssociation, Get-Azure...
Script     0.11.0     AzureRM.MachineLearningCompute.N... {Get-AzureRmMlOpCluster, Get-AzureRmMlOpClusterKey, Test-AzureRmMlOpClusterSystemSe...
Script     0.11.0     AzureRM.MarketplaceOrdering.Netcore {Get-AzureRmMarketplaceTerms, Set-AzureRmMarketplaceTerms}                            
Script     0.11.0     AzureRM.Media.Netcore               {Sync-AzureRmMediaServiceStorageKeys, Set-AzureRmMediaServiceKey, Get-AzureRmMediaS...
Script     0.11.0     AzureRM.Netcore                                                                                                           
Manifest   0.10.0     AzureRM.Netcore                                                                                                           
Script     0.11.0     AzureRM.Network.Netcore             {Add-AzureRmApplicationGatewayAuthenticationCertificate, Get-AzureRmApplicationGate...
Script     0.10.0     AzureRM.Network.Netcore             {Add-AzureRmApplicationGatewayAuthenticationCertificate, Get-AzureRmApplicationGate...
Script     0.11.0     AzureRM.NotificationHubs.Netcore    {Get-AzureRmNotificationHub, Get-AzureRmNotificationHubAuthorizationRules, Get-Azur...
Script     0.11.0     AzureRM.OperationalInsights.Netcore {New-AzureRmOperationalInsightsAzureActivityLogDataSource, New-AzureRmOperationalIn...
Script     0.11.0     AzureRM.PowerBIEmbedded.Netcore     {Remove-AzureRmPowerBIWorkspaceCollection, Get-AzureRmPowerBIWorkspaceCollection, G...
Script     0.11.0     AzureRM.Profile.Netcore             {Disable-AzureRmDataCollection, Disable-AzureRmContextAutosave, Enable-AzureRmDataC...
Script     0.10.0     AzureRM.Profile.Netcore             {Disable-AzureRmDataCollection, Disable-AzureRmContextAutosave, Enable-AzureRmDataC...
Script     0.11.0     AzureRM.RedisCache.Netcore          {Remove-AzureRmRedisCachePatchSchedule, New-AzureRmRedisCacheScheduleEntry, Get-Azu...
Script     0.11.0     AzureRM.Relay.Netcore               {New-AzureRmRelayNamespace, Get-AzureRmRelayNamespace, Set-AzureRmRelayNamespace, R...
Script     0.11.0     AzureRM.Resources.Netcore           {Get-AzureRmProviderOperation, Remove-AzureRmRoleAssignment, Get-AzureRmRoleAssignm...
Script     0.10.0     AzureRM.Resources.Netcore           {Get-AzureRmProviderOperation, Remove-AzureRmRoleAssignment, Get-AzureRmRoleAssignm...
Script     0.11.0     AzureRM.Scheduler.Netcore           {Disable-AzureRmSchedulerJobCollection, Enable-AzureRmSchedulerJobCollection, Get-A...
Script     0.11.0     AzureRM.ServiceBus.Netcore          {New-AzureRmServiceBusNamespace, Get-AzureRmServiceBusNamespace, Set-AzureRmService...
Script     0.11.0     AzureRM.ServiceFabric.Netcore       {Add-AzureRmServiceFabricApplicationCertificate, Add-AzureRmServiceFabricClientCert...
Script     0.11.0     AzureRM.Sql.Netcore                 {Get-AzureRmSqlDatabaseTransparentDataEncryption, Get-AzureRmSqlDatabaseTransparent...
Script     0.11.0     AzureRM.Storage.Netcore             {Get-AzureRmStorageAccount, Get-AzureRmStorageAccountKey, New-AzureRmStorageAccount...
Script     0.10.0     AzureRM.Storage.Netcore             {Get-AzureRmStorageAccount, Get-AzureRmStorageAccountKey, New-AzureRmStorageAccount...
Script     0.11.0     AzureRM.StreamAnalytics.Netcore     {Get-AzureRmStreamAnalyticsFunction, Get-AzureRmStreamAnalyticsDefaultFunctionDefin...
Script     0.11.0     AzureRM.Tags.Netcore                {Remove-AzureRmTag, Get-AzureRmTag, New-AzureRmTag}                                   
Script     0.10.0     AzureRM.Tags.Netcore                {Remove-AzureRmTag, Get-AzureRmTag, New-AzureRmTag}                                   
Script     0.11.0     AzureRM.TrafficManager.Netcore      {Disable-AzureRmTrafficManagerEndpoint, Enable-AzureRmTrafficManagerEndpoint, Set-A...
Script     0.11.0     AzureRM.UsageAggregates.Netcore     Get-UsageAggregates                                                                   
Script     0.11.0     AzureRM.Websites.Netcore            {Get-AzureRmAppServicePlan, Set-AzureRmAppServicePlan, New-AzureRmAppServicePlan, R...
Script     0.10.0     AzureRM.Websites.Netcore            {Get-AzureRmAppServicePlan, Set-AzureRmAppServicePlan, New-AzureRmAppServicePlan, R...

Environment Data

$PSVersionTable
Name                           Value                                                                                                            
----                           -----                                                                                                            
PSVersion                      6.0.2                                                                                                            
PSEdition                      Core                                                                                                             
GitCommitId                    v6.0.2                                                                                                           
OS                             Darwin 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST 2018; root:xnu-4570.51.1~1/RELEASE_X86_64    
Platform                       Unix                                                                                                             
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                          
PSRemotingProtocolVersion      2.3                                                                                                              
SerializationVersion           1.1.0.1                                                                                                          
WSManStackVersion              3.0

Debug Output

DEBUG: 15:42:33 - InvokeAzureRmVMRunCommand begin processing with ParameterSet 'DefaultParameter'.
DEBUG: 15:42:33 - using account id 'REMOVED_AS_SENSITIVE'...
DEBUG: [Common.Authentication]: Authenticating using Account: 'REMOVED_AS_SENSITIVE', environment: 'AzureCloud', tenant: 'REMOVED_AS_SENSITIVE'
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
POST

Absolute Uri:
REMOVED_AS_SENSITIVE

Headers:
x-ms-client-request-id        : REMOVED_AS_SENSITIVE
accept-language               : en-US

Body:
{
  "commandId": "RunShellScript",
  "script": [
    "docker ps"
  ]
}


DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
Accepted

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
Location                      : REMOVED_AS_SENSITIVE
Azure-AsyncOperation          : REMOVED_AS_SENSITIVE
x-ms-ratelimit-remaining-resource: Microsoft.Compute/UpdateVM3Min;239,Microsoft.Compute/UpdateVM30Min;1192
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-served-by                : REMOVED_AS_SENSITIVE
x-ms-request-id               : REMOVED_AS_SENSITIVE
x-ms-ratelimit-remaining-subscription-writes: 1199
Server                        : Microsoft-HTTPAPI/2.0,Microsoft-HTTPAPI/2.0
x-ms-correlation-request-id   : REMOVED_AS_SENSITIVE
x-ms-routing-request-id       : REMOVED_AS_SENSITIVE
X-Content-Type-Options        : nosniff
Date                          : Wed, 18 Apr 2018 12:42:33 GMT

Body:



DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
REMOVED_AS_SENSITIVE

Headers:

Body:



DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-ratelimit-remaining-resource: Microsoft.Compute/GetOperation3Min;14998,Microsoft.Compute/GetOperation30Min;29976
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-served-by                : REMOVED_AS_SENSITIVE
x-ms-request-id               : REMOVED_AS_SENSITIVE
Server                        : Microsoft-HTTPAPI/2.0,Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-reads: 14998
x-ms-correlation-request-id   : REMOVED_AS_SENSITIVE
x-ms-routing-request-id       : REMOVED_AS_SENSITIVE
X-Content-Type-Options        : nosniff
Date                          : Wed, 18 Apr 2018 12:43:03 GMT

Body:
{
  "startTime": "2018-04-18T15:42:33.7502021+03:00",
  "endTime": "2018-04-18T15:42:48.6877604+03:00",
  "status": "Succeeded",
  "properties": {
    "output": [
      {
        "code": "ProvisioningState/succeeded",
        "level": "Info",
        "displayStatus": "Provisioning succeeded",
        "message": "Enable succeeded: \n[stdout]\nCONTAINER ID        IMAGE                                               COMMAND              CREATED             STATUS                 PORTS               NAMES\nREMOVED_AS_SENSITIVE   \"/something.sh\"   2 hours ago         Up 2 hours (healthy)                       something\n\n[stderr]\n"
      }
    ]
  },
  "name": "REMOVED_AS_SENSITIVE"
}


DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
REMOVED_AS_SENSITIVE

Headers:

Body:



DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-ratelimit-remaining-resource: Microsoft.Compute/GetOperation3Min;14997,Microsoft.Compute/GetOperation30Min;29975
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-served-by                : REMOVED_AS_SENSITIVE
x-ms-request-id               : REMOVED_AS_SENSITIVE
Server                        : Microsoft-HTTPAPI/2.0,Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-reads: 14997
x-ms-correlation-request-id   : REMOVED_AS_SENSITIVE
x-ms-routing-request-id       : REMOVED_AS_SENSITIVE
X-Content-Type-Options        : nosniff
Date                          : Wed, 18 Apr 2018 12:43:03 GMT

Body:
[{"code":"ProvisioningState/succeeded","level":"Info","displayStatus":"Provisioning succeeded","message":"Enable succeeded: \n[stdout]\nCONTAINER ID        IMAGE                                               COMMAND              CREATED             STATUS                 PORTS               NAMES\nREMOVED_AS_SENSITIVE   \"/something.sh\"   2 hours ago         Up 2 hours (healthy)                       something\n\n[stderr]\n"}]





DEBUG: AzureQoSEvent: CommandName - Invoke-AzureRmVMRunCommand; IsSuccess - True; Duration - 00:00:31.1142970; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 15:43:04 - InvokeAzureRmVMRunCommand end processing.
DEBUG: 15:43:04 - InvokeAzureRmVMRunCommand end processing.

[cormacpayne]

@mcocak Hey Mert, quick clarifying question: when you say that this was working last week, do you know if that was before or after you updated AzureRM.Netcore to the new 0.11.0 version?

[mcocak]

I didn't do an intended update to AzureRM.Netcore but I uninstalled my existing version and reinstalled the module recently. So if there was a new version released meanwhile, the newer version was installed I suppose, where I started having this issue.

[ljtill]

Also encountering this with the AzureRM modules (5.7.0 - April 2018).

Update:

After testing with -Debug it seems that the response does come back per below (message) however it doesn't output it on the Cmdlet.

Rolled back to the AzureRM modules (5.6.0 - March 2018), all working again.

Body:

{
  "startTime": "2018-04-22T12:57:03.0379341-05:00",
  "endTime": "2018-04-22T12:57:46.3975869-05:00",
  "status": "Succeeded",
  "properties": {
    "output": [
      {
        "code": "ComponentStatus/StdOut/succeeded",
        "level": "Info",
        "displayStatus": "Provisioning succeeded",
        "message": "5.1"
      },
      {
        "code": "ComponentStatus/StdErr/succeeded",
        "level": "Info",
        "displayStatus": "Provisioning succeeded",
        "message": ""
      }
    ]
  },
  "name": "**Removed**"
}

[bcreynolds]

This issue ended up breaking quite a few of our Azure runbooks after an automation account PowerShell module update.

Troubleshooting finding: When only AzureRM.profile v4.5.0 and AzureRM.Compute v4.5.0 modules are imported in an Azure runbook or interactive PowerShell v5.1 session, the issue does not occur until only the AzureRM.profile module version is updated to v4.6.0. At that point, all properties are always null for any Invoke-AzureRmVMRunCommand PSRunCommandResult return object.

[amoghnatu]

Were you running this Cmdlet from a runbook of type PowerShell or PowerShell Workflow? I've seen that this returns empty when executed within a Workflow. Discussed the same with one of the Azure Automation Product group employees and it is expected that way in Workflows.

If you want it to get proper output from it, run it within an InlineScript { } block in the Workflow runbook. And return, from the block, whatever it is that you want to use further.

[bcreynolds]

Returns output in an Azure runbook of type Powershell (not workflow) when the AzureRM.profile module is v4.5. Does not return output in the same PowerShell runbook after update of AzureRM.profile module to v4.6.