Skip to content

Append block from url sync and async #6925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
annatisch merged 6 commits into from
Sep 6, 2019
Merged

Append block from url sync and async #6925

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f4b3bc9
[Sync][AppendBlock]
xiafu-msft Aug 22, 2019
c821e1a
[Async][AppendBlock]Add append_block_from_url
xiafu-msft Aug 22, 2019
bbb03e5
[Blob][AppendBlock]add recordings for append_block_from_url
xiafu-msft Aug 31, 2019
81735d5
[Blob][AppendBlock]fix failing CI
xiafu-msft Sep 4, 2019
d354dde
[Blob][AppendBlock]fix Pylint
xiafu-msft Sep 5, 2019
11fc289
[Blob][AppendBlock]Adjust Stylistic Things
xiafu-msft Sep 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sdk/storage/azure-storage-blob/azure/storage/blob/_shared/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import sys


X_MS_VERSION = '2018-06-17'
X_MS_VERSION = '2019-02-02'

# Socket timeout in seconds
DEFAULT_SOCKET_TIMEOUT = 20
Expand Down
105 changes: 105 additions & 0 deletions sdk/storage/azure-storage-blob/azure/storage/blob/aio/blob_client_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1631,3 +1631,108 @@ async def append_block( # type: ignore
return await self._client.append_blob.append_block(**options) # type: ignore
except StorageErrorException as error:
process_storage_error(error)

@distributed_trace_async()
async def append_block_from_url(self, copy_source_url, # type: str
source_range_start=None, # type Optional[int]
source_range_end=None, # type Optional[int]
source_content_md5=None, # type: Optional[bytearray]
maxsize_condition=None, # type: Optional[int]
appendpos_condition=None, # type: Optional[int]
lease=None, if_modified_since=None,
if_unmodified_since=None, if_match=None,
if_none_match=None, source_if_modified_since=None,
source_if_unmodified_since=None, source_if_match=None,
source_if_none_match=None, timeout=None):
# type: (...) -> Dict[str, Union[str, datetime, int]]
"""
Creates a new block to be committed as part of a blob, where the contents are read from a source url.

:param str copy_source_url:
The URL of the source data. It can point to any Azure Blob or File, that is either public or has a
shared access signature attached.
:param int source_range_start:
This indicates the start of the range of bytes(inclusive) that has to be taken from the copy source.
:param int source_range_end:
This indicates the end of the range of bytes(inclusive) that has to be taken from the copy source.
:param bytearray source_content_md5:
If given, the service will calculate the MD5 hash of the block content and compare against this value.
:param int maxsize_condition:
Optional conditional header. The max length in bytes permitted for
the append blob. If the Append Block operation would cause the blob
to exceed that limit or if the blob size is already greater than the
value specified in this header, the request will fail with
MaxBlobSizeConditionNotMet error (HTTP status code 412 - Precondition Failed).
:param int appendpos_condition:
Optional conditional header, used only for the Append Block operation.
A number indicating the byte offset to compare. Append Block will
succeed only if the append position is equal to this number. If it
is not, the request will fail with the
AppendPositionConditionNotMet error
(HTTP status code 412 - Precondition Failed).
:param ~azure.storage.blob.lease.LeaseClient or str lease:
Required if the blob has an active lease. Value can be a LeaseClient object
or the lease ID as a string.
:param datetime if_modified_since:
A DateTime value. Azure expects the date value passed in to be UTC.
If timezone is included, any non-UTC datetimes will be converted to UTC.
If a date is passed in without timezone info, it is assumed to be UTC.
Specify this header to perform the operation only
if the resource has been modified since the specified time.
:param datetime if_unmodified_since:
A DateTime value. Azure expects the date value passed in to be UTC.
If timezone is included, any non-UTC datetimes will be converted to UTC.
If a date is passed in without timezone info, it is assumed to be UTC.
Specify this header to perform the operation only if
the resource has not been modified since the specified date/time.
:param str if_match:
An ETag value, or the wildcard character (*). Specify this header to perform
the operation only if the resource's ETag matches the value specified.
:param str if_none_match:
An ETag value, or the wildcard character (*). Specify this header
to perform the operation only if the resource's ETag does not match
the value specified. Specify the wildcard character (*) to perform
the operation only if the resource does not exist, and fail the
operation if it does exist.
:param datetime source_if_modified_since:
A DateTime value. Azure expects the date value passed in to be UTC.
If timezone is included, any non-UTC datetimes will be converted to UTC.
If a date is passed in without timezone info, it is assumed to be UTC.
Specify this header to perform the operation only
if the source resource has been modified since the specified time.
:param datetime source_if_unmodified_since:
A DateTime value. Azure expects the date value passed in to be UTC.
If timezone is included, any non-UTC datetimes will be converted to UTC.
If a date is passed in without timezone info, it is assumed to be UTC.
Specify this header to perform the operation only if
the source resource has not been modified since the specified date/time.
:param str source_if_match:
An ETag value, or the wildcard character (*). Specify this header to perform
the operation only if the source resource's ETag matches the value specified.
:param str source_if_none_match:
An ETag value, or the wildcard character (*). Specify this header
to perform the operation only if the source resource's ETag does not match
the value specified. Specify the wildcard character (*) to perform
the operation only if the source resource does not exist, and fail the
operation if it does exist.
:param int timeout:
The timeout parameter is expressed in seconds.
"""
options = self._append_block_from_url_options(
copy_source_url,
source_range_start=source_range_start,
source_range_end=source_range_end,
source_content_md5=source_content_md5,
maxsize_condition=maxsize_condition,
appendpos_condition=appendpos_condition,
lease=lease,
if_modified_since=if_modified_since,
if_unmodified_since=if_unmodified_since, if_match=if_match,
if_none_match=if_none_match, source_if_modified_since=source_if_modified_since,
source_if_unmodified_since=source_if_unmodified_since, source_if_match=source_if_match,
source_if_none_match=source_if_none_match, timeout=timeout
)
try:
return await self._client.append_blob.append_block_from_url(**options) # type: ignore
except StorageErrorException as error:
process_storage_error(error)
165 changes: 163 additions & 2 deletions sdk/storage/azure-storage-blob/azure/storage/blob/blob_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1627,7 +1627,7 @@ def _stage_block_from_url_options(
):
# type: (...) -> Dict[str, Any]
if source_length is not None and source_offset is None:
raise ValueError("Source offset value must not be None is length is set.")
raise ValueError("Source offset value must not be None if length is set.")
block_id = encode_base64(str(block_id))
access_conditions = get_access_conditions(kwargs.pop('lease', None))
range_header = None
Expand Down Expand Up @@ -2384,7 +2384,7 @@ def _append_block_options( # type: ignore
data = data[:length]

append_conditions = None
if maxsize_condition or appendpos_condition:
if maxsize_condition or appendpos_condition is not None:
append_conditions = AppendPositionAccessConditions(
max_size=maxsize_condition,
append_position=appendpos_condition
Expand Down Expand Up @@ -2487,3 +2487,164 @@ def append_block( # type: ignore
return self._client.append_blob.append_block(**options) # type: ignore
except StorageErrorException as error:
process_storage_error(error)

def _append_block_from_url_options( # type: ignore
self, copy_source_url, # type: str
source_range_start=None, # type Optional[int]
source_range_end=None, # type Optional[int]
source_content_md5=None, # type: Optional[bytearray]
maxsize_condition=None, # type: Optional[int]
appendpos_condition=None, # type: Optional[int]
**kwargs
):
# type: (...) -> Dict[str, Any]
if self.require_encryption or (self.key_encryption_key is not None):
raise ValueError(_ERROR_UNSUPPORTED_METHOD_FOR_ENCRYPTION)

# If end range is provided, start range must be provided
if source_range_end is not None and source_range_start is None:
raise ValueError("source_range_start should also be specified if source_range_end is specified")
# Format based on whether end_range is present
source_range = None
if source_range_end is not None:
source_range = 'bytes={0}-{1}'.format(source_range_start, source_range_end)
elif source_range_start is not None:
source_range = "bytes={0}-".format(source_range_start)

append_conditions = None
if maxsize_condition or appendpos_condition is not None:
append_conditions = AppendPositionAccessConditions(
max_size=maxsize_condition,
append_position=appendpos_condition
)
access_conditions = get_access_conditions(kwargs.pop('lease', None))
mod_conditions = ModifiedAccessConditions(
if_modified_since=kwargs.pop('if_modified_since', None),
if_unmodified_since=kwargs.pop('if_unmodified_since', None),
if_match=kwargs.pop('if_match', None),
if_none_match=kwargs.pop('if_none_match', None))
source_mod_conditions = SourceModifiedAccessConditions(
source_if_modified_since=kwargs.pop('source_if_modified_since', None),
source_if_unmodified_since=kwargs.pop('source_if_unmodified_since', None),
source_if_match=kwargs.pop('source_if_match', None),
source_if_none_match=kwargs.pop('source_if_none_match', None))

options = {
'source_url': copy_source_url,
'content_length': 0,
'source_range': source_range,
'source_content_md5': source_content_md5,
'timeout': kwargs.pop('timeout', None),
'transactional_content_md5': None,
'lease_access_conditions': access_conditions,
'append_position_access_conditions': append_conditions,
'modified_access_conditions': mod_conditions,
'source_modified_access_conditions': source_mod_conditions,
'cls': return_response_headers}
options.update(kwargs)
return options

@distributed_trace
def append_block_from_url(self, copy_source_url, # type: str
source_range_start=None, # type Optional[int]
source_range_end=None, # type Optional[int]
source_content_md5=None, # type: Optional[bytearray]
maxsize_condition=None, # type: Optional[int]
appendpos_condition=None, # type: Optional[int]
lease=None, if_modified_since=None,
if_unmodified_since=None, if_match=None,
if_none_match=None, source_if_modified_since=None,
source_if_unmodified_since=None, source_if_match=None,
source_if_none_match=None, timeout=None):
# type: (...) -> Dict[str, Union[str, datetime, int]]
"""
Creates a new block to be committed as part of a blob, where the contents are read from a source url.

:param str copy_source_url:
The URL of the source data. It can point to any Azure Blob or File, that is either public or has a
shared access signature attached.
Copy link
Contributor

@zezha-msft zezha-msft Sep 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe external sources (not inside Azure) are supported too, could you please verify?

Copy link
Contributor Author

@xiafu-msft xiafu-msft Sep 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't work for external url, got this error in response The requested URI does not represent any resource on the server.

Copy link
Contributor

@zezha-msft zezha-msft Sep 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, sounds good

:param int source_range_start:
This indicates the start of the range of bytes(inclusive) that has to be taken from the copy source.
:param int source_range_end:
This indicates the end of the range of bytes(inclusive) that has to be taken from the copy source.
:param bytearray source_content_md5:
If given, the service will calculate the MD5 hash of the block content and compare against this value.
:param int maxsize_condition:
Optional conditional header. The max length in bytes permitted for
the append blob. If the Append Block operation would cause the blob
to exceed that limit or if the blob size is already greater than the
value specified in this header, the request will fail with
MaxBlobSizeConditionNotMet error (HTTP status code 412 - Precondition Failed).
:param int appendpos_condition:
Optional conditional header, used only for the Append Block operation.
A number indicating the byte offset to compare. Append Block will
succeed only if the append position is equal to this number. If it
is not, the request will fail with the
AppendPositionConditionNotMet error
(HTTP status code 412 - Precondition Failed).
:param ~azure.storage.blob.lease.LeaseClient or str lease:
Required if the blob has an active lease. Value can be a LeaseClient object
or the lease ID as a string.
:param datetime if_modified_since:
A DateTime value. Azure expects the date value passed in to be UTC.
If timezone is included, any non-UTC datetimes will be converted to UTC.
If a date is passed in without timezone info, it is assumed to be UTC.
Specify this header to perform the operation only
if the resource has been modified since the specified time.
:param datetime if_unmodified_since:
A DateTime value. Azure expects the date value passed in to be UTC.
If timezone is included, any non-UTC datetimes will be converted to UTC.
If a date is passed in without timezone info, it is assumed to be UTC.
Specify this header to perform the operation only if
the resource has not been modified since the specified date/time.
:param str if_match:
An ETag value, or the wildcard character (*). Specify this header to perform
the operation only if the resource's ETag matches the value specified.
:param str if_none_match:
An ETag value, or the wildcard character (*). Specify this header
to perform the operation only if the resource's ETag does not match
the value specified. Specify the wildcard character (*) to perform
the operation only if the resource does not exist, and fail the
operation if it does exist.
:param datetime source_if_modified_since:
A DateTime value. Azure expects the date value passed in to be UTC.
If timezone is included, any non-UTC datetimes will be converted to UTC.
If a date is passed in without timezone info, it is assumed to be UTC.
Specify this header to perform the operation only
if the source resource has been modified since the specified time.
:param datetime source_if_unmodified_since:
A DateTime value. Azure expects the date value passed in to be UTC.
If timezone is included, any non-UTC datetimes will be converted to UTC.
If a date is passed in without timezone info, it is assumed to be UTC.
Specify this header to perform the operation only if
the source resource has not been modified since the specified date/time.
:param str source_if_match:
An ETag value, or the wildcard character (*). Specify this header to perform
the operation only if the source resource's ETag matches the value specified.
:param str source_if_none_match:
An ETag value, or the wildcard character (*). Specify this header
to perform the operation only if the source resource's ETag does not match
the value specified. Specify the wildcard character (*) to perform
the operation only if the source resource does not exist, and fail the
operation if it does exist.
:param int timeout:
The timeout parameter is expressed in seconds.
"""
options = self._append_block_from_url_options(
copy_source_url,
source_range_start=source_range_start,
source_range_end=source_range_end,
source_content_md5=source_content_md5,
maxsize_condition=maxsize_condition,
appendpos_condition=appendpos_condition,
lease=lease,
if_modified_since=if_modified_since,
if_unmodified_since=if_unmodified_since, if_match=if_match,
if_none_match=if_none_match, source_if_modified_since=source_if_modified_since,
source_if_unmodified_since=source_if_unmodified_since, source_if_match=source_if_match,
source_if_none_match=source_if_none_match, timeout=timeout
)
try:
return self._client.append_blob.append_block_from_url(**options) # type: ignore
except StorageErrorException as error:
process_storage_error(error)
Loading