got HTTP 400 error when calling WebSiteManagementClient.web_apps.install_site_extension()

[APAC-DevOps]

I have a python function which does Azure Web App service deployment along with the extension AspNetCore. If I commented out the lines for installing site extension, my script ran happily; however, with the lines for AspNet Core extension installation, my script can not run successfully.

If I do the manual installation via Azure Portal, then, I can complete the extension installation successfully. I wrote ARM templates with site and extension provision, again, my code runs happily. But calling WebSiteManagementClient.web_apps.install_site_extension() like below doesn't work, and I got
msrestazure.azure_exceptions.CloudError: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/XXXXX(sensitive info)/siteextensions/AspNetCoreRuntime?api-version=2016-08-01

can you take a look and see if I missed anything here?

ARM Template
{
"apiVersion": "2015-08-01",
"name": "[parameters('siteName')]",
"type": "Microsoft.Web/sites",
"location": "[parameters('location')]",
"properties": {
"name": "[parameters('siteName')]",
"serverFarmId": "[parameters('appServicePlanId')]"
},
"resources": [
{
"apiVersion": "2015-08-01",
"name": "web",
"type": "config",
"dependsOn": [
"[resourceId('Microsoft.Web/Sites', parameters('siteName'))]"
],
"properties": {
"phpVersion": "off",
"netFrameworkVersion": "v4.0",
"webSocketsEnabled": true,
"requestTracingEnabled": true,
"httpLoggingEnabled": true,
"logsDirectorySizeLimit": 40,
"detailedErrorLoggingEnabled": true,
"scmType": "LocalGit"
}
},

      {
        "apiVersion": "2016-08-01",
        "name": "AspNetCoreRuntime",
        "type": "siteextensions",
        "dependsOn": [
          "[resourceId('Microsoft.Web/Sites', parameters('siteName'))]"
        ],
        "properties": {}
      }
    ]
  }

Python Code
def deploy_app_service():
print("\nDeploying App Service To Resource Group\n")
AzureServicePrincipalcredentials = ServicePrincipalCredentials(
client_id = my_azure_client_id,
secret = my_azure_secret_id,
tenant = my_azure_tenant_id
)
web_client = WebSiteManagementClient(AzureServicePrincipalcredentials, my_subscription_id)

site_async_operation = web_client.web_apps.create_or_update(
    my_rg_name,
    "jianhuafirstWebApp",
    {
        'location': my_location,
        'server_farm_id': my_service_plan_id,
        'reserved': False,
        'enabled': True,
        'kind': 'app',
        'site_config': {
            "kind": "web",
            "phpVersion": "off",
            "netFrameworkVersion": "v4.0",
            "webSocketsEnabled": True,
            "logsDirectorySizeLimit": 40,
            "requestTracingEnabled": True,
            "httpLoggingEnabled": True,
            "detailedErrorLoggingEnabled": True,
            "scmType": "LocalGit"
        }
    }
)

install site extension

my_site_extension = web_client.web_apps.install_site_extension(
    my_rg_name,
    "jianhuafirstWebApp",
    "AspNetCoreRuntime"
)

[lmazuel]

@yugangw-msft ?

[yugangw-msft]

got msrestazure.azure_exceptions.CloudError: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/XXXXX(sensitive info)/siteextensions/AspNetCoreRuntime?api-version=2016-08-01

Can you capture the error response on the wire and see whether there is any error detail inside?

[APAC-DevOps]

msrestazure.azure_exceptions.CloudError: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/_mysubsciptionId_/resourceGroups/agri-api-dev-appservice/providers/Microsoft.Web/sites/agriOUSDevAUE2/siteextensions/Microsoft.ApplicationInsights.AzureWebSites?api-version=2016-08-01

Below is a copy of my code. I can confirm that the Azure WebApp does get provisioned successfully. It is the provision of extension, which failed. I tried to capture error via AzureException, however, it doesn't seem logging anything which is useful for debugging.

import os.path
import json
import sys
from azure.mgmt.web import WebSiteManagementClient
from azure.common.credentials import ServicePrincipalCredentials
from azure.common import AzureException


parameters_common = "./azure/config/parameters-common.json"
with open(parameters_common, 'r') as parameters_common_fd:
    arm_parameters_common = json.load(parameters_common_fd)

parameters_appService_plan = "./azure/config/output-agri-api-dev-appservice.json"
with open(parameters_appService_plan, 'r') as parameters_appService_plan_fd:
    arm_parameters_appService_plan = json.load(parameters_appService_plan_fd)

my_subscription_id = arm_parameters_common['SubscriptionId']
my_azure_client_id = arm_parameters_common['ClientId']
my_azure_secret_id = arm_parameters_common['SecretId']
my_azure_tenant_id = arm_parameters_common['TenantId']

my_location = arm_parameters_appService_plan['location']
my_rg_id = arm_parameters_appService_plan['resource_group_id']
my_rg_name = my_rg_id.split('/')[-1]
my_service_plan_id = arm_parameters_appService_plan['appServicePlanName']

json_output = {} #Initialize a dictory container for JSON data processing

def deploy_app_service():
    print("\nDeploying App Service To Resource Group\n")
    AzureServicePrincipalcredentials = ServicePrincipalCredentials(
        client_id = my_azure_client_id,
        secret = my_azure_secret_id,
        tenant = my_azure_tenant_id
    )
    web_client = WebSiteManagementClient(AzureServicePrincipalcredentials, my_subscription_id)

    site_async_operation = web_client.web_apps.create_or_update(
        my_rg_name,
        "agriOUSDevAUE2",
        {
            'location': my_location,
            'server_farm_id': my_service_plan_id,
            'reserved': False,
            'enabled': True,
            'kind': 'app',
            'site_config': {
                "scmType": "LocalGit"
            }
        }
    )

    # my_site_extension = web_client.web_apps.list_site_extensions(
    #     my_rg_name,
    #     'agriOUSDevAUE'
    # )

    # for each_site in my_site_extension:
    #     print(each_site)
    try:
        my_site_extension = web_client.web_apps.install_site_extension(
            my_rg_name,
            "agriOUSDevAUE2",
            "Microsoft.ApplicationInsights.AzureWebSites"
        )
    except AzureException as e:
        print(e.response)

if __name__ == "__main__":
    deploy_app_service()

[lmazuel]

Hi @APAC-DevOps
To enable wiring log, you need to do this:

web_client.config.enable_http_logger=True

and then use "logging" to show DEBUG logs. You will the HTTP answer with all details.
Thanks,

[AaronTimms]

I'd try running this without "phpVersion": "off"
I've seen this setting triggering 400 since the latest php version update for Azure

[APAC-DevOps]

I added the line "web_client.config.enable_http_logger=True" to my code, and set my python code's logging level to "debug", below is what I got

(I commented out subscription ID from the output, but apart from that, all of the output from my Mac computer is displayed below)

WARNING:msrest.service_client:Accept header absent and forced to application/json
DEBUG:msrest.pipeline:Configuring request: timeout=100, verify=True, cert=None
DEBUG:msrest.pipeline:Configuring proxies: ''
DEBUG:msrest.pipeline:Evaluate proxies against ENV settings: True
DEBUG:msrest.pipeline:Configuring redirects: allow=True, max=30
DEBUG:msrest.pipeline:Configuring retry: max_retries=4, backoff_factor=0.8, max_backoff=90
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): management.azure.com:443
DEBUG:urllib3.connectionpool:https://management.azure.com:443 "PUT /subscriptions/subscriptionID/resourceGroups/agri-api-dev-appservice/providers/Microsoft.Web/sites/agriOUSDevAUE2/siteextensions/Microsoft.AspNetCore.AzureAppServices.SiteExtension?api-version=2016-08-01 HTTP/1.1" 400 0
DEBUG:msrest.http_logger:Request URL: 'https://management.azure.com/subscriptions/subscriptionID/resourceGroups/agri-api-dev-appservice/providers/Microsoft.Web/sites/agriOUSDevAUE2/siteextensions/Microsoft.AspNetCore.AzureAppServices.SiteExtension?api-version=2016-08-01'
DEBUG:msrest.http_logger:Request method: 'PUT'
DEBUG:msrest.http_logger:Request headers:
DEBUG:msrest.http_logger: 'User-Agent': 'python/3.6.5 (Darwin-17.6.0-x86_64-i386-64bit) requests/2.19.1 msrest/0.5.0 msrest_azure/0.4.32 websitemanagementclient/0.34.1 Azure-SDK-For-Python'
DEBUG:msrest.http_logger: 'Accept-Encoding': 'gzip, deflate'
DEBUG:msrest.http_logger: 'Accept': 'application/json'
DEBUG:msrest.http_logger: 'Connection': 'keep-alive'
DEBUG:msrest.http_logger: 'Content-Type': 'application/json; charset=utf-8'
DEBUG:msrest.http_logger: 'x-ms-client-request-id': 'ac7a4ff4-898f-11e8-8894-9801a7968dff'
DEBUG:msrest.http_logger: 'accept-language': 'en-US'
DEBUG:msrest.http_logger: 'Content-Length': '0'
DEBUG:msrest.http_logger: 'Authorization': '*****'
DEBUG:msrest.http_logger:Request body:
DEBUG:msrest.http_logger:None
DEBUG:msrest.http_logger:Response status: 400
DEBUG:msrest.http_logger:Response headers:
DEBUG:msrest.http_logger: 'Cache-Control': 'no-cache'
DEBUG:msrest.http_logger: 'Pragma': 'no-cache'
DEBUG:msrest.http_logger: 'Expires': '-1'
DEBUG:msrest.http_logger: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
DEBUG:msrest.http_logger: 'x-ms-request-id': '74be45e5-50ff-484b-b537-0960813123ee'
DEBUG:msrest.http_logger: 'Set-Cookie': 'ARRAffinity=60ecc41db6b19c150858a0271e02abb08bc4c432f0eeabbaa9ae45d1e3895350;Path=/;HttpOnly;Domain=agriousdevaue2.scm.azurewebsites.net'
DEBUG:msrest.http_logger: 'Server': 'Microsoft-IIS/10.0, Microsoft-IIS/10.0'
DEBUG:msrest.http_logger: 'X-AspNet-Version': '4.0.30319'
DEBUG:msrest.http_logger: 'X-Powered-By': 'ASP.NET, ASP.NET'
DEBUG:msrest.http_logger: 'x-ms-ratelimit-remaining-subscription-writes': '1199'
DEBUG:msrest.http_logger: 'x-ms-correlation-request-id': '99291ad1-ddd2-484f-b7fc-019edab5029f'
DEBUG:msrest.http_logger: 'x-ms-routing-request-id': 'AUSTRALIAEAST:20180717T070442Z:99291ad1-ddd2-484f-b7fc-019edab5029f'
DEBUG:msrest.http_logger: 'X-Content-Type-Options': 'nosniff'
DEBUG:msrest.http_logger: 'Date': 'Tue, 17 Jul 2018 07:04:41 GMT'
DEBUG:msrest.http_logger: 'Content-Length': '0'
DEBUG:msrest.http_logger:Response content:
DEBUG:msrest.http_logger:b''
DEBUG:msrest.exceptions:Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/14fcb15d-3af7-4499-8677-ae4e3497a99e/resourceGroups/agri-api-dev-appservice/providers/Microsoft.Web/sites/agriOUSDevAUE2/siteextensions/Microsoft.AspNetCore.AzureAppServices.SiteExtension?api-version=2016-08-01
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/msrestazure/azure_operation.py", line 357, in init
self._operation.set_initial_status(self._response)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/msrestazure/azure_operation.py", line 230, in set_initial_status
self._raise_if_bad_http_status_and_method(response)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/msrestazure/azure_operation.py", line 148, in _raise_if_bad_http_status_and_method
"Invalid return status for {!r} operation".format(self.method))
msrestazure.azure_operation.BadStatus: Invalid return status for 'PUT' operation

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "azure/cicd/devApi/appService/provision-web-app-service.py", line 78, in
deploy_app_service()
File "azure/cicd/devApi/appService/provision-web-app-service.py", line 69, in deploy_app_service
"Microsoft.AspNetCore.AzureAppServices.SiteExtension"
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/azure/mgmt/web/operations/web_apps_operations.py", line 9276, in install_site_extension
get_long_running_status, long_running_operation_timeout)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/msrestazure/azure_operation.py", line 360, in init
raise CloudError(self._response)
msrestazure.azure_exceptions.CloudError: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/subscriptionID/resourceGroups/agri-api-dev-appservice/providers/Microsoft.Web/sites/agriOUSDevAUE2/siteextensions/Microsoft.AspNetCore.AzureAppServices.SiteExtension?api-version=2016-08-01

[lmazuel]

Indeed, I don't see any troubling thing here :(. And the error from the server is 400 and has no body to explain the trouble :(

400 usually means the resource does not exist. @yugangw-msft I see that web_apps.create_or_update is not defined as LRO, is this possible that the resource is not yet fully provisioned and we should do a custom LRO?

@APAC-DevOps could you add after your web_apps.create_or_update this:

while True:
    try:
        web_client.web_apps.get(
            my_rg_name,
            "agriOUSDevAUE2"
        )
        break
    except Exception as err:
       print("Assuming WebApp is not ready")
       time.sleep(10)

And see:

• If this fails for a few amount of time (you see the message a few times), and then this passes, and then your site extension works
• If it never fails (you never see the message)

Thanks,

[yugangw-msft]

Maybe @panchagnula can help

[panchagnula]

Did not read entire thread completely but i see from the ARM template you have "phpVersion": "off",
we had a regression on our API end a while back that has since been fixed where off as a value for PhpVersion was being rejected & was throwing an error. "phpVersion": "", would work. Are you still seeing this issue will changing this line in the ARM template help, if not?

[APAC-DevOps]

@panchagnula
My ARM templates work fine (and I have set up my CI/CD pipeline with this set of code already, which works fine). It is my python version of Azure WebApp provision, which gets this issue.

With WebApp siteextensions, if I write define my ARM resource as embedded resource as below, the provision of the extension "Microsoft.ApplicationInsights.AzureWebSites" does fail occasionally.

"resources": [
{
"comment": "embedded resource definition"
"apiVersion": "2016-08-01",
"name": "[parameters('siteName')]",
"kind": "app",
"type": "Microsoft.Web/sites",
"location": "[parameters('location')]",
"properties": {
"reserved": false,
"serverFarmId": "[parameters('appServicePlanId')]",
"siteConfig": {
"appSettings": [
{
"name": "ASPNETCORE_ENVIRONMENT",
"value": "[parameters('coreEnvironment')]"
}
]
}
},
"resources": [
{
"apiVersion": "2016-08-01",
"name": "web",
"type": "config",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Web/Sites', parameters('siteName'))]"
],
"properties": {
"phpVersion": "off",
"netFrameworkVersion": "v4.0",
"webSocketsEnabled": false,
"always_on": true
}
},

      {
        "apiVersion": "2016-08-01",
        "name": "Microsoft.AspNetCore.AzureAppServices.SiteExtension",
        "type": "siteextensions",
        "dependsOn": [
          "[resourceId('Microsoft.Web/Sites', parameters('siteName'))]"
        ],
        "properties": {}
      },

      {
        "apiVersion": "2016-08-01",
        "name": "Microsoft.ApplicationInsights.AzureWebSites",
        "type": "siteextensions",
        "dependsOn": [
          "[resourceId('Microsoft.Web/Sites', parameters('siteName'))]"
        ],
        "properties": {}
      }
    ]
  }
]

however, since I define resources by using "dependson" as below, my WebApp provision works fine. To me, it is something bizarre, and should not have happened, but it did.
{
"apiVersion": "2016-08-01",
"name": "[parameters('siteName')]",
"kind": "app",
"type": "Microsoft.Web/sites",
"location": "[parameters('location')]",
"properties": {
"reserved": false,
"serverFarmId": "[parameters('appServicePlanId')]",
"siteConfig": {
"appSettings": [
{
"name": "ASPNETCORE_ENVIRONMENT",
"value": "[parameters('coreEnvironment')]"
}
]
}
}

  },

  {
    "comments": "no comments",
    "type": "Microsoft.Web/sites/hostNameBindings",
    "name": "[concat(parameters('siteName'), '/', parameters('customDomain'))]",
    "apiVersion": "2016-08-01",
    "location": "[parameters('location')]",
    "properties": {
        "siteName": "[parameters('siteName')]",
        "hostNameType": "Verified",
        "sslState": "SniEnabled",
        "thumbprint": "[parameters('thumbPrint')]"
    },
    "dependsOn": [
        "[resourceId('Microsoft.Web/sites', parameters('siteName'))]"
    ]
    },

  {
        "name": "[concat(parameters('siteName'), '/', 'Microsoft.AspNetCore.AzureAppServices.SiteExtension')]",
        "type": "Microsoft.Web/sites/siteextensions",
        "apiVersion": "2016-08-01",
        "dependsOn": [
            "[resourceId('Microsoft.Web/sites', parameters('siteName'))]"
        ]
    },

    {
        "name": "[concat(parameters('siteName'), '/', 'Microsoft.ApplicationInsights.AzureWebSites')]",
        "type": "Microsoft.Web/sites/siteextensions",
        "apiVersion": "2016-08-01",
        "dependsOn": [
            "[resourceId('Microsoft.Web/sites', parameters('siteName'))]"
        ]
    }
]

@lmazuel
I thought of that when the WebApp extension resource provision executes, the "WebApp" resource might not be ready, so I tried to re-run my Python code a few minutes after the WebApp has provisioned successfully, that failed. I manually added WebApp, and use python code to provision extension only, that failed as well. Lastly, I copied the HTTP URL and pasted it into my browser, this threw out authentication error as well, on the occasion that I had logged into my Azure with an account with Owner permission.

As I have got my pipeline working fine with ARM templates, and the issue with python code seems too difficult to be debugged, so I am OK to let it pass for the time being.

[APAC-DevOps]

Up to this point, I would say this seems likely to be a bug in Azure Rest Api.

Apart from Python SDK, I also tried the rest api call at: https://docs.microsoft.com/en-us/rest/api/appservice/webapps/webapps_installsiteextension
neither appears to be working.

Eventually, I found a sample on Internet which used kudu api in powershell code excerpt for enabling extension on Azure WebApp. I did reading, and probing, and here comes my python code for enabling extensions of Azure WebApp.

add asp net core extension to OUS WebApp

web_app_profile = web_client.web_apps.list_publishing_profile_xml_with_secrets(
    resource_group,
    resource_group + '-ous',
    format = 'WebDeploy'
)

for web_app_profile_element in web_app_profile:
    my_web_app_profile = web_app_profile_element

web_app_profile_xml = ET.fromstring(my_web_app_profile)
web_app_site_name = web_app_profile_xml[0].attrib['msdeploySite']
web_app_url = web_app_profile_xml[0].attrib['publishUrl']
web_app_user_name = web_app_profile_xml[0].attrib['userName']
web_app_user_pwd = web_app_profile_xml[0].attrib['userPWD']

asp_net_core_id = "https://" + resource_group + '-ous' + ".scm.azurewebsites.net/api/siteextensions/" + "Microsoft.AspNetCore.AzureAppServices.SiteExtension"
requests.put(url = asp_net_core_id, auth = (web_app_user_name, web_app_user_pwd))

I found it is very difficult to find python code sample for AWS SDK on Internet. Can you tell me if there is any good website which can give me some guide when I went into deadend (apart from github).