Storage Preview 3 Release (#7154)

* Storage swagger (#6777)

* Python swagger update

This only changes to the official swagger version and regenerates

* Upgrade blobs swagger to 2019-02-02

* Fixed namespace dir and header

* Enable XML for blobs

* Generate vanilla blobs

* Fixed global parameters in swagger spec

* Regenerated blobs

* Revert "Fixed global parameters in swagger spec"

This reverts commit 75bd21b.

* Moved parameter fix to readme

* Fixed blob list serialization

* Fix for new metadata model

* Fixed datetime formatting

* Fixed required access policy

* Fixed readme analysis

* Regenerated queues

* Regenerated files

* Fixed files datetime

* Skip tests pending msrest fix

* Fixed for queue XML

* Fix for docsettings

* [storage] Fix for Files upload return type (#6772)

* Fix missing import

* Fix for Files inconsistent return types

* Blob async trace decorators

* [SnapshotSAS]Add Snapshot SAS (#6819)

* [SnapshotSAS]Add Snapshot SAS

* [SnapshotSAS]Extract SharedAccessSignature for Blob File and Queue

* [SnapshotSAS]Fix Typo

* [SnapshotSAS]Delete Redundant code

The removed code is for another feature.

* [SnapshotSAS]Stylistic Tweak

* Updating Storage Files swagger to 2019-02-02 (#6872)

* Updating Storage Files swagger to 2019-02-02

This includes a few transform changes to set default values for new required
parameters and marked certain 8601 dates as strings when they use more precision
that Python's dates.

I'm seeing a few test failures that I'm hoping someone on the Python side can
investigate (mostly "The specified share is being deleted. Try operation later."
that I was seeing before my changes).  There are a few other failures related to
new service features that we can fix if easy or disable and file a work item to
get these changes in.  The recorded tests will also have to be updated, but we
don't need to block on that right now.

* skip file permission related tests

* [Storage] File swagger updates (#6926)

* Updating Storage Files swagger to 2019-02-02

This includes a few transform changes to set default values for new required
parameters and marked certain 8601 dates as strings when they use more precision
that Python's dates.

I'm seeing a few test failures that I'm hoping someone on the Python side can
investigate (mostly "The specified share is being deleted. Try operation later."
that I was seeing before my changes).  There are a few other failures related to
new service features that we can fix if easy or disable and file a work item to
get these changes in.  The recorded tests will also have to be updated, but we
don't need to block on that right now.

* swagger changes

* [storage] make storage pylint-clean (#6929)

* make storage pylint-clean

* feedback

* Remove Storage swagger files and reference azure-rest-api-specs (#6943)

Also includes some changes to the generated code updated to the latest version
of blobs

* Append block from url sync and async (#6925)

* [Sync][AppendBlock]

* [Async][AppendBlock]Add append_block_from_url

* [Blob][AppendBlock]add recordings for append_block_from_url

* [Blob][AppendBlock]fix failing CI

* [Blob][AppendBlock]fix Pylint

* [Blob][AppendBlock]Adjust Stylistic Things

* [UpdatePage]add upload_pages_from_url sync and async (#6960)

* [UpdatePage]add upload_pages_from_url sync and async

* [UploadPage]Adjust Stylistic Things

* Fix Pylint

* Fix Pylint

* [Queues] Migrate to devtools test framework (#6882)

* initial migration

* samples

* comments

* queues test fixes

* recodings update

* update

* comment address

* pylint fix

* comments address

* Put range from url (#7081)

* [File]Upload Range From Url Sync

* [File]Upload Range From Url ASync

* Fix Pylint

* Fix Pylint

* CPK--Client Provided Encryption Key (#7104)

* [Blob]CPK without test

* Add Generated Code

* Fix Pylint

* [Blob][CPK]Test

* Add Missing Recordings

* [Blob][CPK]use existing field self.scheme

* File rest parity (#7001)

* [File][RestParity]Rest Parity Sync

Add Rest Parity Sync part(except create permission)
tweak _shared package a bit

* [File][RestParity]Rest Parity Async

Add Rest Parity Async part(except create permission)

* [File][RestParity]Add Rest Parity Async Recording Files

Add recording files for both sync and async

* [File][RestParity]Fix CI

* [File][RestParity]Recording again to fix CI

* Add Generated Code

* Stylistic Things and Record

* [Swagger][BugFix]workaround to fix swagger generated code

* [File][RestParity]Add Create_Permission API and Test

* Fix Test

* Fix Pylint

* Revert the workaround

* [File][RestParity]Tweak Documentation and Tests

* delete .dat file

* [Blob][SAS] Added support for identity SAS (#7020)

* [Storage] Incremented version for preview 3 and updated change log (#7156)

* Update Azure Core and msrest

* Add Workaround for SharePermissions,revert msrest

* Update version and history.md

* updates on history.md

* minor change

* more changes

* [Storage] Fix flaky tests (#7179)

* Update HISTORY.md

* Fix some typo in error message

* Fix Some Tests

* some more fixes

* Skip OAUTH Test

.docsettings.yml

@@ -48,6 +48,9 @@ known_content_issues:
   - ['sdk/loganalytics/azure-loganalytics/README.rst', '#4554']
   - ['sdk/servicefabric/azure-servicefabric/README.rst', '#4554']
   - ['sdk/core/azure-servicemanagement-legacy/README.rst', '#4554']
+  - ['sdk/storage/azure-storage-file/swagger/README.md', '#4554']
+  - ['sdk/storage/azure-storage-queue/swagger/README.md', '#4554']
+  - ['sdk/storage/azure-storage-blob/swagger/README.md', '#4554']
 
   # nspckg and common.
   - ['sdk/cognitiveservices/azure-cognitiveservices-nspkg/README.rst', 'nspkg and common']

sdk/storage/azure-storage-blob/HISTORY.md

@@ -1,6 +1,27 @@
 # Change Log azure-storage-blob
 
 
+## Version 12.0.0b3:
+
+**New features**
+- Added SAS support for snapshot and identity.
+- Distributed tracing framework OpenCensus is now supported.
+- Added support for append_block_from_url API for append blobs.
+- Added support for upload_pages_from_url API for page blobs.
+- Added support for client provided encryption key to numerous APIs.
+
+**Dependency updates**
+- Adopted [azure-core](https://pypi.org/project/azure-core/) 1.0.0b3
+  - If you later want to revert to previous versions of azure-storage-blob, or another Azure SDK
+  library requiring azure-core 1.0.0b1 or azure-core 1.0.0b2, you must explicitly install
+  the specific version of azure-core as well. For example:
+
+  `pip install azure-core==1.0.0b2 azure-storage-blob==12.0.0b2`
+
+**Fixes and improvements**
+- Fix where content-type was being added in the request when not mentioned explicitly.
+
+
 ## Version 12.0.0b2:
 
 **Breaking changes**

sdk/storage/azure-storage-blob/azure/storage/blob/_generated/__init__.py

@@ -8,11 +8,9 @@
 # Changes may cause incorrect behavior and will be lost if the code is
 # regenerated.
 # --------------------------------------------------------------------------
-# pylint: skip-file
 
-from ._configuration import AzureBlobStorageConfiguration
 from ._azure_blob_storage import AzureBlobStorage
-__all__ = ['AzureBlobStorage', 'AzureBlobStorageConfiguration']
+__all__ = ['AzureBlobStorage']
 
 from .version import VERSION
 

sdk/storage/azure-storage-blob/azure/storage/blob/_generated/_azure_blob_storage.py

@@ -8,7 +8,6 @@
 # Changes may cause incorrect behavior and will be lost if the code is
 # regenerated.
 # --------------------------------------------------------------------------
-# pylint: skip-file
 
 from azure.core import PipelineClient
 from msrest import Serializer, Deserializer
@@ -17,6 +16,7 @@
 from azure.core.exceptions import map_error
 from .operations import ServiceOperations
 from .operations import ContainerOperations
+from .operations import DirectoryOperations
 from .operations import BlobOperations
 from .operations import PageBlobOperations
 from .operations import AppendBlobOperations
@@ -29,38 +29,42 @@ class AzureBlobStorage(object):
 
 
     :ivar service: Service operations
-    :vartype service: blob.operations.ServiceOperations
+    :vartype service: azure.storage.blob.operations.ServiceOperations
     :ivar container: Container operations
-    :vartype container: blob.operations.ContainerOperations
+    :vartype container: azure.storage.blob.operations.ContainerOperations
+    :ivar directory: Directory operations
+    :vartype directory: azure.storage.blob.operations.DirectoryOperations
     :ivar blob: Blob operations
-    :vartype blob: blob.operations.BlobOperations
+    :vartype blob: azure.storage.blob.operations.BlobOperations
     :ivar page_blob: PageBlob operations
-    :vartype page_blob: blob.operations.PageBlobOperations
+    :vartype page_blob: azure.storage.blob.operations.PageBlobOperations
     :ivar append_blob: AppendBlob operations
-    :vartype append_blob: blob.operations.AppendBlobOperations
+    :vartype append_blob: azure.storage.blob.operations.AppendBlobOperations
     :ivar block_blob: BlockBlob operations
-    :vartype block_blob: blob.operations.BlockBlobOperations
+    :vartype block_blob: azure.storage.blob.operations.BlockBlobOperations
 
     :param url: The URL of the service account, container, or blob that is the
      targe of the desired operation.
     :type url: str
     """
 
-    def __init__(self, url, config=None, **kwargs):
+    def __init__(self, url, **kwargs):
 
         base_url = '{url}'
-        self._config = config or AzureBlobStorageConfiguration(url, **kwargs)
+        self._config = AzureBlobStorageConfiguration(url, **kwargs)
         self._client = PipelineClient(base_url=base_url, config=self._config, **kwargs)
 
         client_models = {k: v for k, v in models.__dict__.items() if isinstance(v, type)}
-        self.api_version = '2018-03-28'
+        self.api_version = '2019-02-02'
         self._serialize = Serializer(client_models)
         self._deserialize = Deserializer(client_models)
 
         self.service = ServiceOperations(
             self._client, self._config, self._serialize, self._deserialize)
         self.container = ContainerOperations(
             self._client, self._config, self._serialize, self._deserialize)
+        self.directory = DirectoryOperations(
+            self._client, self._config, self._serialize, self._deserialize)
         self.blob = BlobOperations(
             self._client, self._config, self._serialize, self._deserialize)
         self.page_blob = PageBlobOperations(

sdk/storage/azure-storage-blob/azure/storage/blob/_generated/_configuration.py

@@ -8,9 +8,8 @@
 # Changes may cause incorrect behavior and will be lost if the code is
 # regenerated.
 # --------------------------------------------------------------------------
-# pylint: skip-file
 
-from azure.core.configuration import Configuration, ConnectionConfiguration
+from azure.core.configuration import Configuration
 from azure.core.pipeline import policies
 
 from .version import VERSION
@@ -37,18 +36,17 @@ def __init__(self, url, **kwargs):
         super(AzureBlobStorageConfiguration, self).__init__(**kwargs)
         self._configure(**kwargs)
 
-        self.user_agent_policy.add_user_agent('azureblobstorage/{}'.format(VERSION))
+        self.user_agent_policy.add_user_agent('azsdk-python-azureblobstorage/{}'.format(VERSION))
         self.generate_client_request_id = True
-        self.accept_language = None
 
         self.url = url
-        self.version = "2018-03-28"
+        self.version = "2019-02-02"
 
     def _configure(self, **kwargs):
-        self.connection = ConnectionConfiguration(**kwargs)
-        self.user_agent_policy = policies.UserAgentPolicy(**kwargs)
-        self.headers_policy = policies.HeadersPolicy(**kwargs)
-        self.proxy_policy = policies.ProxyPolicy(**kwargs)
-        self.logging_policy = policies.NetworkTraceLoggingPolicy(**kwargs)
-        self.retry_policy = policies.RetryPolicy(**kwargs)
-        self.redirect_policy = policies.RedirectPolicy(**kwargs)
+        self.user_agent_policy = kwargs.get('user_agent_policy') or policies.UserAgentPolicy(**kwargs)
+        self.headers_policy = kwargs.get('headers_policy') or policies.HeadersPolicy(**kwargs)
+        self.proxy_policy = kwargs.get('proxy_policy') or policies.ProxyPolicy(**kwargs)
+        self.logging_policy = kwargs.get('logging_policy') or policies.NetworkTraceLoggingPolicy(**kwargs)
+        self.retry_policy = kwargs.get('retry_policy') or policies.RetryPolicy(**kwargs)
+        self.custom_hook_policy = kwargs.get('custom_hook_policy') or policies.CustomHookPolicy(**kwargs)
+        self.redirect_policy = kwargs.get('redirect_policy') or policies.RedirectPolicy(**kwargs)

sdk/storage/azure-storage-blob/azure/storage/blob/_generated/aio/__init__.py

@@ -8,8 +8,6 @@
 # Changes may cause incorrect behavior and will be lost if the code is
 # regenerated.
 # --------------------------------------------------------------------------
-# pylint: skip-file
 
 from ._azure_blob_storage_async import AzureBlobStorage
-from ._configuration_async import AzureBlobStorageConfiguration
-__all__ = ['AzureBlobStorage', 'AzureBlobStorageConfiguration']
+__all__ = ['AzureBlobStorage']

sdk/storage/azure-storage-blob/azure/storage/blob/_generated/aio/_azure_blob_storage_async.py

@@ -8,7 +8,6 @@
 # Changes may cause incorrect behavior and will be lost if the code is
 # regenerated.
 # --------------------------------------------------------------------------
-# pylint: skip-file
 
 from azure.core import AsyncPipelineClient
 from msrest import Serializer, Deserializer
@@ -17,6 +16,7 @@
 from azure.core.exceptions import map_error
 from .operations_async import ServiceOperations
 from .operations_async import ContainerOperations
+from .operations_async import DirectoryOperations
 from .operations_async import BlobOperations
 from .operations_async import PageBlobOperations
 from .operations_async import AppendBlobOperations
@@ -29,39 +29,43 @@ class AzureBlobStorage(object):
 
 
     :ivar service: Service operations
-    :vartype service: blob.aio.operations_async.ServiceOperations
+    :vartype service: azure.storage.blob.aio.operations_async.ServiceOperations
     :ivar container: Container operations
-    :vartype container: blob.aio.operations_async.ContainerOperations
+    :vartype container: azure.storage.blob.aio.operations_async.ContainerOperations
+    :ivar directory: Directory operations
+    :vartype directory: azure.storage.blob.aio.operations_async.DirectoryOperations
     :ivar blob: Blob operations
-    :vartype blob: blob.aio.operations_async.BlobOperations
+    :vartype blob: azure.storage.blob.aio.operations_async.BlobOperations
     :ivar page_blob: PageBlob operations
-    :vartype page_blob: blob.aio.operations_async.PageBlobOperations
+    :vartype page_blob: azure.storage.blob.aio.operations_async.PageBlobOperations
     :ivar append_blob: AppendBlob operations
-    :vartype append_blob: blob.aio.operations_async.AppendBlobOperations
+    :vartype append_blob: azure.storage.blob.aio.operations_async.AppendBlobOperations
     :ivar block_blob: BlockBlob operations
-    :vartype block_blob: blob.aio.operations_async.BlockBlobOperations
+    :vartype block_blob: azure.storage.blob.aio.operations_async.BlockBlobOperations
 
     :param url: The URL of the service account, container, or blob that is the
      targe of the desired operation.
     :type url: str
     """
 
     def __init__(
-            self, url, config=None, **kwargs):
+            self, url, **kwargs):
 
         base_url = '{url}'
-        self._config = config or AzureBlobStorageConfiguration(url, **kwargs)
+        self._config = AzureBlobStorageConfiguration(url, **kwargs)
         self._client = AsyncPipelineClient(base_url=base_url, config=self._config, **kwargs)
 
         client_models = {k: v for k, v in models.__dict__.items() if isinstance(v, type)}
-        self.api_version = '2018-03-28'
+        self.api_version = '2019-02-02'
         self._serialize = Serializer(client_models)
         self._deserialize = Deserializer(client_models)
 
         self.service = ServiceOperations(
             self._client, self._config, self._serialize, self._deserialize)
         self.container = ContainerOperations(
             self._client, self._config, self._serialize, self._deserialize)
+        self.directory = DirectoryOperations(
+            self._client, self._config, self._serialize, self._deserialize)
         self.blob = BlobOperations(
             self._client, self._config, self._serialize, self._deserialize)
         self.page_blob = PageBlobOperations(

sdk/storage/azure-storage-blob/azure/storage/blob/_generated/aio/_configuration_async.py

@@ -8,9 +8,8 @@
 # Changes may cause incorrect behavior and will be lost if the code is
 # regenerated.
 # --------------------------------------------------------------------------
-# pylint: skip-file
 
-from azure.core.configuration import Configuration, ConnectionConfiguration
+from azure.core.configuration import Configuration
 from azure.core.pipeline import policies
 
 from ..version import VERSION
@@ -37,18 +36,18 @@ def __init__(self, url, **kwargs):
         super(AzureBlobStorageConfiguration, self).__init__(**kwargs)
         self._configure(**kwargs)
 
-        self.user_agent_policy.add_user_agent('azureblobstorage/{}'.format(VERSION))
+        self.user_agent_policy.add_user_agent('azsdk-python-azureblobstorage/{}'.format(VERSION))
         self.generate_client_request_id = True
         self.accept_language = None
 
         self.url = url
-        self.version = "2018-03-28"
+        self.version = "2019-02-02"
 
     def _configure(self, **kwargs):
-        self.connection = ConnectionConfiguration(**kwargs)
-        self.user_agent_policy = policies.UserAgentPolicy(**kwargs)
-        self.headers_policy = policies.HeadersPolicy(**kwargs)
-        self.proxy_policy = policies.ProxyPolicy(**kwargs)
-        self.logging_policy = policies.NetworkTraceLoggingPolicy(**kwargs)
-        self.retry_policy = policies.AsyncRetryPolicy(**kwargs)
-        self.redirect_policy = policies.AsyncRedirectPolicy(**kwargs)
+        self.user_agent_policy = kwargs.get('user_agent_policy') or policies.UserAgentPolicy(**kwargs)
+        self.headers_policy = kwargs.get('headers_policy') or policies.HeadersPolicy(**kwargs)
+        self.proxy_policy = kwargs.get('proxy_policy') or policies.ProxyPolicy(**kwargs)
+        self.logging_policy = kwargs.get('logging_policy') or policies.NetworkTraceLoggingPolicy(**kwargs)
+        self.retry_policy = kwargs.get('retry_policy') or policies.AsyncRetryPolicy(**kwargs)
+        self.custom_hook_policy = kwargs.get('custom_hook_policy') or policies.CustomHookPolicy(**kwargs)
+        self.redirect_policy = kwargs.get('redirect_policy') or policies.AsyncRedirectPolicy(**kwargs)

sdk/storage/azure-storage-blob/azure/storage/blob/_generated/aio/operations_async/__init__.py

@@ -8,10 +8,10 @@
 # Changes may cause incorrect behavior and will be lost if the code is
 # regenerated.
 # --------------------------------------------------------------------------
-# pylint: skip-file
 
 from ._service_operations_async import ServiceOperations
 from ._container_operations_async import ContainerOperations
+from ._directory_operations_async import DirectoryOperations
 from ._blob_operations_async import BlobOperations
 from ._page_blob_operations_async import PageBlobOperations
 from ._append_blob_operations_async import AppendBlobOperations
@@ -20,6 +20,7 @@
 __all__ = [
     'ServiceOperations',
     'ContainerOperations',
+    'DirectoryOperations',
     'BlobOperations',
     'PageBlobOperations',
     'AppendBlobOperations',