[storage] Add async APIs for Files SDK (#6405)

* Refactored blob shared utils * Refactored file shared utils * Refactored queue shared utils * Refactored downloads * Refactored file downloads * Started async downloads * Async Files API * Flatten copy polling * Renamed uploads * Fixes samples based on vendor feedback (#6357) * fixed import statements for sync samples * fixed secrets readme samples * fixed keys readme samples * deduped secret names across samples * Upload refactor * Release approval docs (#6361) * updating dev documentation with approval details * Updated async pipeline * Avoid surprising aiohttp with unexpected kwargs (#6355) * Add challenge authentication to azure-keyvault-keys (#6244) * Add decorator (#6299) * add opencensus impl * dont need to check for noop tracer * get rid of span and trace id * fix span instance * added documentation * added documentation and refactor names of a few variables * write test for opencensus wrapper * put opencensus in the dev requirements * initial common * only import the wrapper when necessary * add check for the exporter * rework logic and fix some settings * added initial decorator * some mroe documentation * added decorators * small change * some minor fixes * test patch happening * fix space * share a function * clearer logic for setting span context * better logic * better logic * fix environ variable * test the way opencensus does it * middle of tests * only load if opencensus has already been imported * fix spelling mistake * temp * finish writing tests for common * charles fixes * fix tests * fix test settings * to header should not take a dict * from header should be class method * initial tests * dont create trace and get rid of end_tracer * dont need to save the trace * more little fixes * some intermediatary changes * fix type annotations * rst fix types * add :class:annotations * fix line wrapping * added tests for decorator * rename opencensus wrapper * intermediate changes * use spans the right way * some formatting * some grammar * restructure settings and make tests pass * rename get_parent * fix typings * use protocol and from_headers becomes links * ramifications of opencensus wrapper being a protocol * add tests for link * added async tests * delete the unit test thing * added add_attribute * added add_attribute * added tests for add attributes * remove unused import * added docstrings * minor docstring formatting * fix pylint errors * don't rely on opencensus children to check * use exporter to not rely on parent.children * test should only propagate * made test tracing helper * decrease flakiness of test * simplify get parent * calling a decorator decorator is redundant * fix settings * more elegant code * add await for async * should only have to wait a 1/1000 of a second * fix tests spans too short * accidentally deleted setup.cfg * Added async file tests * Consolidate Key Vault shared code (#6384) * Download tests * Service property tests * No recordings * Add credential wrapping MSAL ConfidentialClientApplication (#6358) * Add policy (#6379) * add opencensus impl * dont need to check for noop tracer * get rid of span and trace id * fix span instance * added documentation * added documentation and refactor names of a few variables * write test for opencensus wrapper * put opencensus in the dev requirements * initial common * only import the wrapper when necessary * add check for the exporter * rework logic and fix some settings * added initial decorator * some mroe documentation * added decorators * small change * some minor fixes * test patch happening * fix space * share a function * clearer logic for setting span context * better logic * better logic * fix environ variable * test the way opencensus does it * middle of tests * only load if opencensus has already been imported * fix spelling mistake * temp * finish writing tests for common * charles fixes * fix tests * fix test settings * to header should not take a dict * from header should be class method * initial tests * dont create trace and get rid of end_tracer * dont need to save the trace * more little fixes * some intermediatary changes * fix type annotations * rst fix types * add :class:annotations * fix line wrapping * added tests for decorator * rename opencensus wrapper * intermediate changes * use spans the right way * some formatting * some grammar * restructure settings and make tests pass * rename get_parent * fix typings * use protocol and from_headers becomes links * ramifications of opencensus wrapper being a protocol * add tests for link * added async tests * delete the unit test thing * added add_attribute * added add_attribute * added tests for add attributes * add initial policy * remove unused import * added docstrings * minor docstring formatting * fix pylint errors * don't rely on opencensus children to check * use exporter to not rely on parent.children * added documentation and span attributes * added test tracing policy * test should only propagate * made test tracing helper * decrease flakiness of test * simplify get parent * calling a decorator decorator is redundant * middle of writing tests * fix settings * add tests * test propogation also happens * more elegant code * await async stuff * add await for async * should only have to wait a 1/1000 of a second * fix tests spans too short * accidentally deleted setup.cfg * add component * add set_http_attributes * fix span network name * fix http request types * fix more types * bryan fixes * more efficient tests * non flakey tests * make tracing only use my context * test user agent on exception * pylint formatting * delete unused import * fix spelling * pylint * adding dockerfile (#6393) * Update cheatsheet.md * Async share tests * Async directory tests * Fixed some tests * aiohttp socket timeout * Patch azure core * CI fixes * Fix async tests for py35 * Python 3.5 support * Clean pylint
Azure · annatisch · Aug 6, 2019 · Jul 18, 2019 · Jul 22, 2019 · Jul 22, 2019
commit 5eca2bb4429c9297d3b88186992f8b2e39a8dc03
diff --git a/.gitignore b/.gitignore
@@ -78,3 +78,4 @@ code_reports
 sdk/storage/azure-storage-blob/tests/settings_real.py
 sdk/storage/azure-storage-queue/tests/settings_real.py
 sdk/storage/azure-storage-file/tests/settings_real.py
+*.code-workspace
diff --git a/doc/dev/mgmt/cheatsheet.md b/doc/dev/mgmt/cheatsheet.md
@@ -1,5 +1,11 @@
 # Mgmt helper commands cheat sheet
 
+## Dockerized environment
+
+Following Dockerfile can be used to build entire enviornment:
+
+https://github.com/Azure/azure-sdk-for-python/blob/master/tools/Dockerfile
+
 ## Create a venv
 
 Windows:<br/>

diff --git a/doc/dev/release.md b/doc/dev/release.md
@@ -16,13 +16,17 @@ Python packages are uploaded to [PyPI](https://pypi.org/). Once you've uploaded
 
 ### Production - Deploy with Azure Dev Ops
 
-Go to this Url: https://aka.ms/pysdkrelease
+To avoid "accidental" pushes to our target repositories, [approval](https://docs.microsoft.com/en-us/azure/devops/pipelines/release/approvals/approvals?view=azure-devops) will be requested directly prior to the final PyPI publish. Reference this [wiki page](https://aka.ms/python-approval-groups) and click on `Release to PyPI Approvers` to add yourself to the group for PyPI publishing. 
+
+After taking care of the above, go to this Url: https://aka.ms/pysdkrelease
 
 - Click on "Run pipeline"
 - Change "BuildTargetingString" to the name of your package. Example: azure-mgmt-compute
 - Change "CandidateForRelease" value to `True` (it should be defaulted to `False`)
 
-Et voila :). Azure Dev Ops will tests one last time the package, dependencies, and the distributions (sdist/wheel) and push to PyPI using the user [azure-sdk](https://pypi.org/user/azure-sdk/).
+Et voila :). Azure Dev Ops will tests one last time the package, dependencies, and the distributions (sdist/wheel) and ask for approval prior to pushing to PyPI using the user [azure-sdk](https://pypi.org/user/azure-sdk/).
+
+Validate artifacts prior to clicking `approve` on the pre-deployment confirmation mail waiting in your inbox. Please allow ~5 minutes for the email to be sent by Azure DevOps
 
 ### Production - Deploy manually
 

diff --git a/sdk/core/azure-core/azure/core/pipeline/policies/distributed_tracing.py b/sdk/core/azure-core/azure/core/pipeline/policies/distributed_tracing.py
@@ -0,0 +1,106 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+#
+# --------------------------------------------------------------------------
+"""Traces network calls using the implementation library from the settings."""
+
+from azure.core.pipeline import PipelineRequest, PipelineResponse
+from azure.core.tracing.context import tracing_context
+from azure.core.tracing.abstract_span import AbstractSpan
+from azure.core.tracing.common import set_span_contexts
+from azure.core.pipeline.policies import SansIOHTTPPolicy
+from azure.core.settings import settings
+
+try:
+    from urllib.parse import urlparse
+except ImportError:
+    from urlparse import urlparse
+
+try:
+    from typing import TYPE_CHECKING
+except ImportError:
+    TYPE_CHECKING = False
+
+if TYPE_CHECKING:
+    from typing import Any, Optional
+    from azure.core.pipeline.transport import HttpRequest, HttpResponse
+
+
+class DistributedTracingPolicy(SansIOHTTPPolicy):
+    """The policy to create spans for Azure Calls"""
+
+    def __init__(self):
+        # type: (str, str, str) -> None
+        self.parent_span_dict = {}
+        self._request_id = "x-ms-request-id"
+
+    def set_header(self, request, span):
+        # type: (PipelineRequest[HttpRequest], Any) -> None
+        """
+        Sets the header information on the span.
+        """
+        headers = span.to_header()
+        request.http_request.headers.update(headers)
+
+    def on_request(self, request, **kwargs):
+        # type: (PipelineRequest[HttpRequest], Any) -> None
+        parent_span = tracing_context.current_span.get()  # type: AbstractSpan
+
+        if parent_span is None:
+            return
+
+        only_propagate = settings.tracing_should_only_propagate()
+        if only_propagate:
+            self.set_header(request, parent_span)
+            return
+
+        path = urlparse(request.http_request.url).path
+        if not path:
+            path = "/"
+        child = parent_span.span(name=path)
+        child.start()
+
+        set_span_contexts(child)
+        self.parent_span_dict[child] = parent_span
+        self.set_header(request, child)
+
+    def end_span(self, request, response=None):
+        # type: (HttpRequest, Optional[HttpResponse]) -> None
+        """Ends the span that is tracing the network and updates its status."""
+        span = tracing_context.current_span.get()   # type: AbstractSpan
+        only_propagate = settings.tracing_should_only_propagate()
+        if span and not only_propagate:
+            span.set_http_attributes(request, response=response)
+            if response and self._request_id in response.headers:
+                span.add_attribute(self._request_id, response.headers[self._request_id])
+            span.finish()
+            set_span_contexts(self.parent_span_dict[span])
+
+    def on_response(self, request, response, **kwargs):
+        # type: (PipelineRequest[HttpRequest], PipelineResponse[HttpRequest, HttpResponse], Any) -> None
+        self.end_span(request.http_request, response=response.http_response)
+
+    def on_exception(self, _request, **kwargs):  # pylint: disable=unused-argument
+        # type: (PipelineRequest[HttpRequest], Any) -> bool
+        self.end_span(_request.http_request)
diff --git a/sdk/core/azure-core/azure/core/pipeline/policies/universal.py b/sdk/core/azure-core/azure/core/pipeline/policies/universal.py
@@ -350,7 +350,10 @@ def deserialize_from_http_generics(cls, response):
         # Try to use content-type from headers if available
         content_type = None
         if response.content_type: # type: ignore
-            content_type = response.content_type[0].strip().lower() # type: ignore
+            try:
+                content_type = response.content_type.strip().lower() # type: ignore
+            except AttributeError:
+                content_type = response.content_type[0].strip().lower() # type: ignore
 
         # Ouch, this server did not declare what it sent...
         # Let's guess it's JSON...

diff --git a/sdk/core/azure-core/azure/core/pipeline/transport/aiohttp.py b/sdk/core/azure-core/azure/core/pipeline/transport/aiohttp.py
@@ -97,9 +97,7 @@ async def close(self):
             self._session_owner = False
             self.session = None
 
-    def _build_ssl_config(self, **config):
-        verify = config.get('connection_verify', self.config.connection.verify)
-        cert = config.get('connection_cert', self.config.connection.cert)
+    def _build_ssl_config(self, cert, verify):
         ssl_ctx = None
 
         if cert or verify not in (True, False):
@@ -144,15 +142,18 @@ async def send(self, request: HttpRequest, **config: Any) -> Optional[AsyncHttpR
         await self.open()
         error = None
         response = None
-        config['ssl'] = self._build_ssl_config(**config)
+        config['ssl'] = self._build_ssl_config(
+            cert=config.pop('connection_cert', self.config.connection.cert),
+            verify=config.pop('connection_verify', self.config.connection.verify)
+        )
         try:
             stream_response = config.pop("stream", False)
             result = await self.session.request(
                 request.method,
                 request.url,
                 headers=request.headers,
                 data=self._get_request_data(request),
-                timeout=config.get('connection_timeout', self.config.connection.timeout),
+                timeout=config.pop('connection_timeout', self.config.connection.timeout),
                 allow_redirects=False,
                 **config
             )

diff --git a/sdk/core/azure-core/azure/core/settings.py b/sdk/core/azure-core/azure/core/settings.py
@@ -31,7 +31,19 @@
 from collections import namedtuple
 import logging
 import os
-from typing import Any, Union
+import six
+import sys
+
+try:
+    from typing import TYPE_CHECKING
+except ImportError:
+    TYPE_CHECKING = False
+
+if TYPE_CHECKING:
+    from typing import Any, Union
+
+
+from azure.core.tracing import AbstractSpan
 
 
 __all__ = ("settings",)
@@ -61,9 +73,9 @@ def convert_bool(value):
         return value  # type: ignore
 
     val = value.lower()  # type: ignore
-    if val in ["yes", "1", "on"]:
+    if val in ["yes", "1", "on", "true", "True"]:
         return True
-    if val in ["no", "0", "off"]:
+    if val in ["no", "0", "off", "false", "False"]:
         return False
     raise ValueError("Cannot convert {} to boolean value".format(value))
 
@@ -102,14 +114,63 @@ def convert_logging(value):
     val = value.upper()  # type: ignore
     level = _levels.get(val)
     if not level:
-        raise ValueError(
-            "Cannot convert {} to log level, valid values are: {}".format(
-                value, ", ".join(_levels)
-            )
-        )
+        raise ValueError("Cannot convert {} to log level, valid values are: {}".format(value, ", ".join(_levels)))
     return level
 
 
+def get_opencensus_span():
+    # type: () -> OpenCensusSpan
+    """Returns the OpenCensusSpan if opencensus is installed else returns None"""
+    try:
+        from azure.core.tracing.ext.opencensus_span import OpenCensusSpan
+
+        return OpenCensusSpan
+    except ImportError:
+        return None
+
+
+def get_opencensus_span_if_opencensus_is_imported():
+    if "opencensus" not in sys.modules:
+        return None
+    return get_opencensus_span()
+
+
+_tracing_implementation_dict = {"opencensus": get_opencensus_span}
+
+
+def convert_tracing_impl(value):
+    # type: (Union[str, AbstractSpan]) -> AbstractSpan
+    """Convert a string to AbstractSpan
+
+    If a AbstractSpan is passed in, it is returned as-is. Otherwise the function
+    understands the following strings, ignoring case:
+
+    * "opencensus"
+
+    :param value: the value to convert
+    :type value: string
+    :returns: AbstractSpan
+    :raises ValueError: If conversion to AbstractSpan fails
+
+    """
+    if value is None:
+        return get_opencensus_span_if_opencensus_is_imported()
+
+    wrapper_class = value
+    if isinstance(value, six.string_types):
+        value = value.lower()
+        get_wrapper_class = _tracing_implementation_dict.get(value, lambda: _Unset)
+        wrapper_class = get_wrapper_class()
+        if wrapper_class is _Unset:
+            raise ValueError(
+                "Cannot convert {} to AbstractSpan, valid values are: {}".format(
+                    value, ", ".join(_tracing_implementation_dict)
+                )
+            )
+
+    return wrapper_class
+
+
 class PrioritizedSetting(object):
     """Return a value for a global setting according to configuration precedence.
 
@@ -138,9 +199,7 @@ class PrioritizedSetting(object):
 
     """
 
-    def __init__(
-        self, name, env_var=None, system_hook=None, default=_Unset, convert=None
-    ):
+    def __init__(self, name, env_var=None, system_hook=None, default=_Unset, convert=None):
 
         self._name = name
         self._env_var = env_var
@@ -311,11 +370,7 @@ def defaults(self):
         """ Return implicit default values for all settings, ignoring environment and system.
 
         """
-        props = {
-            k: v.default
-            for (k, v) in self.__class__.__dict__.items()
-            if isinstance(v, PrioritizedSetting)
-        }
+        props = {k: v.default for (k, v) in self.__class__.__dict__.items() if isinstance(v, PrioritizedSetting)}
         return self._config(props)
 
     @property
@@ -335,11 +390,7 @@ def config(self, **kwargs):
         settings.config(log_level=logging.DEBUG)
 
         """
-        props = {
-            k: v()
-            for (k, v) in self.__class__.__dict__.items()
-            if isinstance(v, PrioritizedSetting)
-        }
+        props = {k: v() for (k, v) in self.__class__.__dict__.items() if isinstance(v, PrioritizedSetting)}
         props.update(kwargs)
         return self._config(props)
 
@@ -348,17 +399,19 @@ def _config(self, props):  # pylint: disable=no-self-use
         return Config(**props)
 
     log_level = PrioritizedSetting(
-        "log_level",
-        env_var="AZURE_LOG_LEVEL",
-        convert=convert_logging,
-        default=logging.INFO,
+        "log_level", env_var="AZURE_LOG_LEVEL", convert=convert_logging, default=logging.INFO
     )
 
     tracing_enabled = PrioritizedSetting(
-        "tracing_enbled",
-        env_var="AZURE_TRACING_ENABLED",
-        convert=convert_bool,
-        default=False,
+        "tracing_enbled", env_var="AZURE_TRACING_ENABLED", convert=convert_bool, default=False
+    )
+
+    tracing_implementation = PrioritizedSetting(
+        "tracing_implementation", env_var="AZURE_SDK_TRACING_IMPLEMENTATION", convert=convert_tracing_impl, default=None
+    )
+
+    tracing_should_only_propagate = PrioritizedSetting(
+        "tracing_should_only_propagate", env_var="AZURE_TRACING_ONLY_PROPAGATE", convert=convert_bool, default=False
     )
 
 

diff --git a/sdk/core/azure-core/azure/core/tracing/__init__.py b/sdk/core/azure-core/azure/core/tracing/__init__.py
@@ -3,7 +3,7 @@
 # Licensed under the MIT License.
 # ------------------------------------
 from azure.core.tracing.abstract_span import AbstractSpan
-from azure.core.tracing.context import tracing_context
-
-__all__ = ["tracing_context", "AbstractSpan"]
 
+__all__ = [
+    "AbstractSpan",
+]