From 7fa8c2e464b446c969e957c02efe000c8385a45d Mon Sep 17 00:00:00 2001 From: Amanda Nguyen Date: Sun, 5 Jan 2025 20:15:55 -0800 Subject: [PATCH 1/7] Update readme/samples --- .../README.md | 171 +++++++++--------- .../samples/Sample01b_HelloWorldAsync.cs | 6 +- .../README.md | 11 +- .../samples/Sample01b_HelloWorldAsync.cs | 39 ++-- sdk/storage/Azure.Storage.DataMovement.sln | 12 ++ .../Azure.Storage.DataMovement/README.md | 41 ++++- 6 files changed, 157 insertions(+), 123 deletions(-) diff --git a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md index 677fc4f8cee8..b739d0caca89 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md +++ b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md @@ -1,7 +1,5 @@ # Azure Storage Data Movement Blobs client library for .NET -> Server Version: 2020-04-08, 2020-02-10, 2019-12-12, 2019-07-07, and 2020-02-02 - ## Project Status: Beta This product is in beta. Some features will be missing or have significant bugs. Please see [Known Issues](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.DataMovement/KnownIssues.md) for detailed information. @@ -9,12 +7,10 @@ This product is in beta. Some features will be missing or have significant bugs. --- Azure Storage is a Microsoft-managed service providing cloud storage that is -highly available, secure, durable, scalable, and redundant. Azure Storage -includes Azure Blobs (objects), Azure Data Lake Storage Gen2, Azure Files, -and Azure Queues. +highly available, secure, durable, scalable, and redundant. -The Azure Storage Data Movement library is optimized for uploading, downloading and -copying customer data. +The Azure Storage Data Movement Blobs library is optimized for uploading, downloading and +copying blobs. The Azure.Storage.DataMovement.Blobs library provides infrastructure shared by the other Azure Storage client libraries. @@ -72,82 +68,10 @@ We guarantee that all client instance methods are thread-safe and independent of This section demonstrates usage of Data Movement for interacting with blob storage. -### Extensions on `BlobContainerClient` - -For applications with preexisting code using Azure.Storage.Blobs, this package provides extension methods for `BlobContainerClient` to get some of the benefits of the `TransferManager` with minimal extra code. - -Instantiate the BlobContainerClient -```C# Snippet:ExtensionMethodCreateContainerClient -BlobServiceClient service = new BlobServiceClient(serviceUri, credential); - -BlobContainerClient container = service.GetBlobContainerClient(containerName); -``` - -Upload a local directory to the root of the container -```C# Snippet:ExtensionMethodSimpleUploadToRoot -DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath); +### Using the TransferManager for Blob Transfers -await transfer.WaitForCompletionAsync(); -``` +The `TransferManager` is the primary class for managing data transfers between storage resources. See [Setup the TransferManager sample](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement#setup-the-transfermanager). -Upload a local directory to a virtual directory in the container by specifying a directory prefix -```C# Snippet:ExtensionMethodSimpleUploadToDirectoryPrefix -DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath, blobDirectoryPrefix); - -await transfer.WaitForCompletionAsync(); -``` - -Upload a local directory to a virtual directory in the container specifying more advanced options -```C# Snippet:ExtensionMethodSimpleUploadWithOptions -BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions -{ - BlobContainerOptions = new BlobStorageResourceContainerOptions - { - BlobDirectoryPrefix = blobDirectoryPrefix - }, - TransferOptions = new DataTransferOptions() - { - CreationPreference = StorageResourceCreationPreference.OverwriteIfExists, - } -}; - -DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath, options); - -await transfer.WaitForCompletionAsync(); -``` - -Download the entire container to a local directory -```C# Snippet:ExtensionMethodSimpleDownloadContainer -DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath); - -await transfer.WaitForCompletionAsync(); -``` - -Download a directory in the container by specifying a directory prefix -```C# Snippet:ExtensionMethodSimpleDownloadContainerDirectory -DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix); - -await tranfer.WaitForCompletionAsync(); -``` - -Download from the container specifying more advanced options -```C# Snippet:ExtensionMethodSimpleDownloadContainerDirectoryWithOptions -BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions -{ - BlobContainerOptions = new BlobStorageResourceContainerOptions - { - BlobDirectoryPrefix = blobDirectoryPrefix - }, - TransferOptions = new DataTransferOptions() - { - CreationPreference = StorageResourceCreationPreference.OverwriteIfExists, - } -}; - -DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options); - -await tranfer.WaitForCompletionAsync(); -``` ### Initializing Blob Storage `StorageResource` @@ -302,13 +226,92 @@ destinationResource: blobs.FromContainer( await dataTransfer.WaitForCompletionAsync(); ``` +### Extensions on `BlobContainerClient` + +For applications with preexisting code using Azure.Storage.Blobs, this package provides extension methods for `BlobContainerClient` to get some of the benefits of the `TransferManager` with minimal extra code. + +Instantiate the BlobContainerClient +```C# Snippet:ExtensionMethodCreateContainerClient +BlobServiceClient service = new BlobServiceClient(serviceUri, credential); + +BlobContainerClient container = service.GetBlobContainerClient(containerName); +``` + +Upload a local directory to the root of the container +```C# Snippet:ExtensionMethodSimpleUploadToRoot +DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath); + +await transfer.WaitForCompletionAsync(); +``` + +Upload a local directory to a virtual directory in the container by specifying a directory prefix +```C# Snippet:ExtensionMethodSimpleUploadToDirectoryPrefix +DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath, blobDirectoryPrefix); + +await transfer.WaitForCompletionAsync(); +``` + +Upload a local directory to a virtual directory in the container specifying more advanced options +```C# Snippet:ExtensionMethodSimpleUploadWithOptions +BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions +{ + BlobContainerOptions = new BlobStorageResourceContainerOptions + { + BlobDirectoryPrefix = blobDirectoryPrefix + }, + TransferOptions = new DataTransferOptions() + { + CreationPreference = StorageResourceCreationPreference.OverwriteIfExists, + } +}; + +DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath, options); + +await transfer.WaitForCompletionAsync(); +``` + +Download the entire container to a local directory +```C# Snippet:ExtensionMethodSimpleDownloadContainer +DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath); + +await transfer.WaitForCompletionAsync(); +``` + +Download a directory in the container by specifying a directory prefix +```C# Snippet:ExtensionMethodSimpleDownloadContainerDirectory +DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix); + +await tranfer.WaitForCompletionAsync(); +``` + +Download from the container specifying more advanced options +```C# Snippet:ExtensionMethodSimpleDownloadContainerDirectoryWithOptions +BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions +{ + BlobContainerOptions = new BlobStorageResourceContainerOptions + { + BlobDirectoryPrefix = blobDirectoryPrefix + }, + TransferOptions = new DataTransferOptions() + { + CreationPreference = StorageResourceCreationPreference.OverwriteIfExists, + } +}; + +DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options); + +await tranfer.WaitForCompletionAsync(); +``` + ## Troubleshooting -***TODO*** +See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Logging](https://learn.microsoft.com/en-us/dotnet/azure/sdk/logging) to assist with any troubleshooting. ## Next steps -***TODO*** +Get started with our [Share Files Samples][share_samples]. + +For more base Transfer Manager scenarios see [DataMovement samples][samples]. ## Contributing @@ -342,7 +345,7 @@ additional questions or comments. [azure_sub]: https://azure.microsoft.com/free/dotnet/ [RequestFailedException]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/core/Azure.Core/src/RequestFailedException.cs [error_codes]: https://learn.microsoft.com/rest/api/storageservices/common-rest-api-error-codes -[samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Blobs/samples +[share_samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Files.Shares/samples [storage_contrib]: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/CONTRIBUTING.md [cla]: https://cla.microsoft.com [coc]: https://opensource.microsoft.com/codeofconduct/ diff --git a/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs b/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs index b3607bc683a3..198dcf58e4ab 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs +++ b/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs @@ -9,11 +9,9 @@ using NUnit.Framework; using Azure.Core; using Azure.Identity; -using System.Threading; using Azure.Storage.Blobs.Models; using Azure.Storage.Blobs; using System.Collections.Generic; -using System.Linq; namespace Azure.Storage.DataMovement.Blobs.Samples { @@ -968,7 +966,7 @@ public async Task PauseAndResumeAsync_DataTransferPause() } /// - /// Use the extention method to upload an entire directory. + /// Use the extension method to upload an entire directory. /// [Test] public async Task UploadDirectory() @@ -1037,7 +1035,7 @@ public async Task UploadDirectory() } /// - /// Use the extention method to upload an entire directory. + /// Use the extension method to upload an entire directory. /// [Test] public async Task DownloadDirectory() diff --git a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md index fa5833660665..cd22c340f919 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md +++ b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md @@ -1,7 +1,5 @@ # Azure Storage Data Movement File Shares client library for .NET -> Server Version: 2020-04-08, 2020-02-10, 2019-12-12, 2019-07-07, and 2020-02-02 - ## Project Status: Beta This product is in beta. Some features will be missing or have significant bugs. Please see [Known Issues](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.DataMovement/KnownIssues.md) for detailed information. @@ -167,11 +165,13 @@ await directoryTransfer.WaitForCompletionAsync(); ## Troubleshooting -***TODO*** +See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Logging](https://learn.microsoft.com/en-us/dotnet/azure/sdk/logging) to assist with any troubleshooting. ## Next steps -***TODO*** +Get started with our [Blob DataMovement samples][blob_samples]. + +For more base Transfer Manager scenarios see [DataMovement samples][samples]. ## Contributing @@ -205,7 +205,8 @@ additional questions or comments. [azure_sub]: https://azure.microsoft.com/free/dotnet/ [RequestFailedException]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/core/Azure.Core/src/RequestFailedException.cs [error_codes]: https://learn.microsoft.com/rest/api/storageservices/common-rest-api-error-codes -[samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Files.Shares/samples +[samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement/samples +[blob_samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Blobs/samples [storage_contrib]: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/CONTRIBUTING.md [cla]: https://cla.microsoft.com [coc]: https://opensource.microsoft.com/codeofconduct/ diff --git a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/samples/Sample01b_HelloWorldAsync.cs b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/samples/Sample01b_HelloWorldAsync.cs index 600ee05a4638..50ad06515ad9 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/samples/Sample01b_HelloWorldAsync.cs +++ b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/samples/Sample01b_HelloWorldAsync.cs @@ -7,14 +7,10 @@ using NUnit.Framework; using Azure.Core; using Azure.Identity; -using System.Collections.Generic; -using System.Linq; using Azure.Storage.Files.Shares; -using Azure.Storage.DataMovement.Files.Shares; using Azure.Storage.Sas; -using BenchmarkDotNet.Disassemblers; -namespace Azure.Storage.DataMovement.Blobs.Samples +namespace Azure.Storage.DataMovement.Files.Shares.Samples { /// /// Basic Azure File Share Storage samples. @@ -44,12 +40,12 @@ public async Task ResourceConstructionDemonstration() // Get local filesystem provider LocalFilesStorageResourceProvider files = new(); - // Get blobs provider with credential + // Get shares provider with credential #region Snippet:MakeProvider_TokenCredential_Shares ShareFilesStorageResourceProvider shares = new(tokenCredential); #endregion - // Construct simple blob resources for data movement + // Construct simple share file resources for data movement #region Snippet:ResourceConstruction_Shares StorageResource directory = shares.FromDirectory( new Uri("http://myaccount.files.core.windows.net/share/path/to/directory")); @@ -70,7 +66,7 @@ public async Task ResourceConstructionDemonstration() } { StorageSharedKeyCredential sharedKeyCredential = new(StorageAccountName, StorageAccountKey); - // Get blobs provider with credential + // Get shares provider with credential AzureSasCredential GenerateSas(Uri uri, bool readOnly) { // Quick sample demonstrating minimal steps @@ -99,22 +95,22 @@ public async Task Upload() string sourceLocalFile = CreateTempFile(SampleFileContent); string sourceLocalDirectory = CreateTempDirectoryPath(); string connectionString = ConnectionString; - string containerName = Randomize("sample-container"); + string shareName = Randomize("sample-share"); // Create a client that can authenticate with a connection string - ShareClient share = new(connectionString, containerName); + ShareClient share = new(connectionString, shareName); await share.CreateIfNotExistsAsync(); try { ShareFilesStorageResourceProvider shares = new(new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey)); LocalFilesStorageResourceProvider files = new(); - // Get a reference to a destination blobs + // Get a reference to a destination share file/directory Uri destinationFolderUri = share.GetDirectoryClient("sample-directory").Uri; Uri destinationFileUri = share.GetRootDirectoryClient().GetFileClient("sample-file").Uri; TransferManager transferManager = new TransferManager(new TransferManagerOptions()); - // Create simple transfer single blob upload job + // Create simple transfer single share file upload job #region Snippet:SimplefileUpload_Shares DataTransfer fileTransfer = await transferManager.StartTransferAsync( sourceResource: files.FromFile(sourceLocalFile), @@ -122,6 +118,7 @@ public async Task Upload() await fileTransfer.WaitForCompletionAsync(); #endregion + // Create simple transfer single share directory upload job #region Snippet:SimpleDirectoryUpload_Shares DataTransfer folderTransfer = await transferManager.StartTransferAsync( sourceResource: files.FromDirectory(sourceLocalDirectory), @@ -142,22 +139,22 @@ public async Task Download() string destinationLocalFile = CreateTempFile(SampleFileContent); string destinationLocalDirectory = CreateTempDirectoryPath(); string connectionString = ConnectionString; - string containerName = Randomize("sample-container"); + string shareName = Randomize("sample-share"); // Create a client that can authenticate with a connection string - ShareClient share = new(connectionString, containerName); + ShareClient share = new(connectionString, shareName); await share.CreateIfNotExistsAsync(); try { ShareFilesStorageResourceProvider shares = new(new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey)); LocalFilesStorageResourceProvider files = new(); - // Get a reference to a destination blobs + // Get a reference to a destination share file/directory Uri sourceDirectoryUri = share.GetDirectoryClient("sample-directory").Uri; Uri sourceFileUri = share.GetRootDirectoryClient().GetFileClient("sample-file").Uri; TransferManager transferManager = new TransferManager(new TransferManagerOptions()); - // Create simple transfer single blob upload job + // Create simple transfer single share file upload job #region Snippet:SimpleFileDownload_Shares DataTransfer fileTransfer = await transferManager.StartTransferAsync( sourceResource: shares.FromFile(sourceFileUri), @@ -165,6 +162,7 @@ public async Task Download() await fileTransfer.WaitForCompletionAsync(); #endregion + // Create simple transfer single share directory upload job #region Snippet:SimpleDirectoryDownload_Shares DataTransfer directoryTransfer = await transferManager.StartTransferAsync( sourceResource: shares.FromDirectory(sourceDirectoryUri), @@ -182,23 +180,23 @@ public async Task Download() public async Task Copy() { string connectionString = ConnectionString; - string containerName = Randomize("sample-container"); + string shareName = Randomize("sample-share"); // Create a client that can authenticate with a connection string - ShareClient share = new(connectionString, containerName); + ShareClient share = new(connectionString, shareName); await share.CreateIfNotExistsAsync(); try { ShareFilesStorageResourceProvider shares = new(new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey)); - // Get a reference to a destination blobs + // Get a reference to a destination share files/directories Uri sourceDirectoryUri = share.GetDirectoryClient("sample-directory-1").Uri; Uri destinationDirectoryUri = share.GetDirectoryClient("sample-directory-2").Uri; Uri sourceFileUri = share.GetRootDirectoryClient().GetFileClient("sample-file-1").Uri; Uri destinationFileUri = share.GetRootDirectoryClient().GetFileClient("sample-file-2").Uri; TransferManager transferManager = new TransferManager(new TransferManagerOptions()); - // Create simple transfer single blob upload job + // Create simple transfer single share file upload job #region Snippet:s2sCopyFile_Shares DataTransfer fileTransfer = await transferManager.StartTransferAsync( sourceResource: shares.FromFile(sourceFileUri), @@ -206,6 +204,7 @@ public async Task Copy() await fileTransfer.WaitForCompletionAsync(); #endregion + // Create simple transfer single share directory upload job #region Snippet:s2sCopyDirectory_Shares DataTransfer directoryTransfer = await transferManager.StartTransferAsync( sourceResource: shares.FromDirectory(sourceDirectoryUri), diff --git a/sdk/storage/Azure.Storage.DataMovement.sln b/sdk/storage/Azure.Storage.DataMovement.sln index b12ee0be03be..fcc85276888d 100644 --- a/sdk/storage/Azure.Storage.DataMovement.sln +++ b/sdk/storage/Azure.Storage.DataMovement.sln @@ -35,6 +35,10 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Azure.Storage.Dat EndProject Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Azure.Storage.DataMovement.Blobs.Files.Shares.Tests", "Azure.Storage.DataMovement.Files.Shares\BlobToFileSharesTests\Azure.Storage.DataMovement.Blobs.Files.Shares.Tests.csproj", "{4603F1BF-EDCF-44B7-8A85-E13EC4267A35}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Azure.Storage.DataMovement.Blobs.Samples.Tests", "Azure.Storage.DataMovement.Blobs\samples\Azure.Storage.DataMovement.Blobs.Samples.Tests.csproj", "{B8A57DA3-768F-43F3-B876-19BE685C15AF}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Azure.Storage.DataMovement.Files.Shares.Samples.Tests", "Azure.Storage.DataMovement.Files.Shares\samples\Azure.Storage.DataMovement.Files.Shares.Samples.Tests.csproj", "{76FB3889-81CD-4AD2-8372-13E2CA98B226}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -105,6 +109,14 @@ Global {4603F1BF-EDCF-44B7-8A85-E13EC4267A35}.Debug|Any CPU.Build.0 = Debug|Any CPU {4603F1BF-EDCF-44B7-8A85-E13EC4267A35}.Release|Any CPU.ActiveCfg = Release|Any CPU {4603F1BF-EDCF-44B7-8A85-E13EC4267A35}.Release|Any CPU.Build.0 = Release|Any CPU + {B8A57DA3-768F-43F3-B876-19BE685C15AF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {B8A57DA3-768F-43F3-B876-19BE685C15AF}.Debug|Any CPU.Build.0 = Debug|Any CPU + {B8A57DA3-768F-43F3-B876-19BE685C15AF}.Release|Any CPU.ActiveCfg = Release|Any CPU + {B8A57DA3-768F-43F3-B876-19BE685C15AF}.Release|Any CPU.Build.0 = Release|Any CPU + {76FB3889-81CD-4AD2-8372-13E2CA98B226}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {76FB3889-81CD-4AD2-8372-13E2CA98B226}.Debug|Any CPU.Build.0 = Debug|Any CPU + {76FB3889-81CD-4AD2-8372-13E2CA98B226}.Release|Any CPU.ActiveCfg = Release|Any CPU + {76FB3889-81CD-4AD2-8372-13E2CA98B226}.Release|Any CPU.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE diff --git a/sdk/storage/Azure.Storage.DataMovement/README.md b/sdk/storage/Azure.Storage.DataMovement/README.md index 9d2cf65115c0..913ff4250481 100644 --- a/sdk/storage/Azure.Storage.DataMovement/README.md +++ b/sdk/storage/Azure.Storage.DataMovement/README.md @@ -1,7 +1,5 @@ # Azure Storage Data Movement Common client library for .NET -> Server Version: 2021-02-12, 2020-12-06, 2020-10-02, 2020-08-04, 2020-06-12, 2020-04-08, 2020-02-10, 2019-12-12, 2019-07-07, and 2020-02-02 - ## Project Status: Beta This product is in beta. Some features will be missing or have significant bugs. Please see [Known Issues](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.DataMovement/KnownIssues.md) for detailed information. @@ -14,7 +12,7 @@ highly available, secure, durable, scalable, and redundant. The Azure Storage Data Movement library is optimized for uploading, downloading and copying customer data. -Currently this version of the Data Movement library only supports Blobs. +Currently this version of the Data Movement library only supports Blobs and File Shares. [Source code][source] | [API reference documentation][docs] | [REST API documentation][rest_docs] | [Product documentation][product_docs] @@ -48,8 +46,8 @@ Authentication is specific to the targeted storage service. Please see documenta ## Key concepts -The Azure Storage Common client library contains shared infrastructure like -[authentication credentials][auth_credentials] and [RequestFailedException][RequestFailedException]. +The Azure Storage DataMovement client library contains shared infrastructure like +[TokenCredential](https://learn.microsoft.com/en-us/dotnet/api/azure.core.tokencredential?view=azure-dotnet), [TransferManager](#setup-the-transfermanager) and [RequestFailedException][RequestFailedException]. ### Thread safety We guarantee that all client instance methods are thread-safe and independent of each other ([guideline](https://azure.github.io/azure-sdk/dotnet_introduction.html#dotnet-service-methods-thread-safety)). This ensures that the recommendation of reusing client instances is always safe, even across threads. @@ -99,6 +97,8 @@ await dataTransfer.WaitForCompletionAsync(cancellationToken); By persisting transfer progress to disk, DataMovement allows resuming of transfers that failed partway through, or were otherwise paused. To resume a transfer, the transfer manager needs to be setup in the first place with `StorageResourceProvider` instances (the same ones used above in [Starting New Transfers](#starting-new-transfers)) which are capable of reassembling the transfer components from persisted data. +To pause see [Pausing Transfers](#pausing-transfers). + The below sample initializes the `TransferManager` such that it's capable of resuming transfers between the local filesystem and Azure Blob Storage, using the Azure.Storage.DataMovement.Blobs package. **Important:** Credentials to storage providers are not persisted. Storage access which requires credentials will need its appropriate `StorageResourceProvider` to be configured with those credentials. Below uses an `Azure.Core` token credential with permission to the appropriate resources. @@ -118,7 +118,25 @@ To resume a transfer, provide the transfer's ID, as shown below. In the case whe DataTransfer resumedTransfer = await transferManager.ResumeTransferAsync(transferId); ``` -Note: the location of persisted transfer data will be different than the default location if `TransferCheckpointStoreOptions` were set in `TransferManagerOptions`. To resume transfers recorded in a non-default location, the transfer manager resuming the transfer will also need the appropriate checkpoint store options. +#### Pause and Resume Checkpointing + +The location of persisted transfer data will be different than the default location if `TransferCheckpointStoreOptions` were set in `TransferManagerOptions`. To resume transfers recorded in a non-default location, the transfer manager resuming the transfer will also need the appropriate checkpoint store options. + +To specify the checkpoint folder directory: +```csharp +TransferManagerOptions options = new TransferManagerOptions() +{ + CheckpointerOptions = TransferCheckpointStoreOptions.Local() +}; +``` + +To disable checkpointing: +```csharp +TransferManagerOptions options = new TransferManagerOptions() +{ + CheckpointerOptions = TransferCheckpointStoreOptions.Disabled() +}; +``` ### Monitoring Transfers @@ -242,7 +260,7 @@ transferOptions.TransferFailed += (TransferFailedEventArgs args) => }; ``` -### Initializing Local File `StorageResource` +### Initializing Local File or Directory `StorageResource` Local filesystem resources are provided by `LocalFilesStorageResourceProvider`. This provider requires no setup to produce storage resources. @@ -254,11 +272,13 @@ StorageResource directoryResource = files.FromDirectory("C:/path/to/dir"); ## Troubleshooting -***TODO*** +See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Logging](https://learn.microsoft.com/en-us/dotnet/azure/sdk/logging) to assist with any troubleshooting. ## Next steps -Get started with our [Blob DataMovement samples][samples]. +Get started with our [Blob DataMovement samples][blob_samples]. + +Get started with our [Share File DataMovement samples][share_samples]. ## Contributing @@ -293,7 +313,8 @@ additional questions or comments. [blobs_examples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Blobs#examples [RequestFailedException]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/core/Azure.Core/src/RequestFailedException.cs [error_codes]: https://learn.microsoft.com/rest/api/storageservices/common-rest-api-error-codes -[samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Blobs/samples +[blob_samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Blobs/samples +[share_samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Files.Shares/samples [storage_contrib]: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/CONTRIBUTING.md [cla]: https://cla.microsoft.com [coc]: https://opensource.microsoft.com/codeofconduct/ From 691b11403ddae5659b3d7aa88f1415b7829434a3 Mon Sep 17 00:00:00 2001 From: Amanda Nguyen Date: Mon, 6 Jan 2025 15:03:13 -0800 Subject: [PATCH 2/7] Removed base samples, rely on blob and share samples --- ....Storage.DataMovement.Samples.Tests.csproj | 40 -- .../samples/Sample01b_HelloWorldAsync.cs | 384 ------------------ 2 files changed, 424 deletions(-) delete mode 100644 sdk/storage/Azure.Storage.DataMovement/samples/Azure.Storage.DataMovement.Samples.Tests.csproj delete mode 100644 sdk/storage/Azure.Storage.DataMovement/samples/Sample01b_HelloWorldAsync.cs diff --git a/sdk/storage/Azure.Storage.DataMovement/samples/Azure.Storage.DataMovement.Samples.Tests.csproj b/sdk/storage/Azure.Storage.DataMovement/samples/Azure.Storage.DataMovement.Samples.Tests.csproj deleted file mode 100644 index 1470c83f1602..000000000000 --- a/sdk/storage/Azure.Storage.DataMovement/samples/Azure.Storage.DataMovement.Samples.Tests.csproj +++ /dev/null @@ -1,40 +0,0 @@ - - - $(RequiredTargetFrameworks) - Microsoft Azure.Storage.DataMovement client library samples - false - - - - - - - - - - - - PreserveNewest - - - - - - - - - - - - - - - - - - - - - - - diff --git a/sdk/storage/Azure.Storage.DataMovement/samples/Sample01b_HelloWorldAsync.cs b/sdk/storage/Azure.Storage.DataMovement/samples/Sample01b_HelloWorldAsync.cs deleted file mode 100644 index f6372e550e4c..000000000000 --- a/sdk/storage/Azure.Storage.DataMovement/samples/Sample01b_HelloWorldAsync.cs +++ /dev/null @@ -1,384 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT License. - -using System; -using System.IO; -using System.Threading.Tasks; -using Azure.Storage.Blobs.Specialized; -using Azure.Storage.Sas; -using NUnit.Framework; -using Azure.Core; -using Azure.Identity; -using System.Threading; -using Azure.Storage.Blobs.Models; -using Azure.Storage.Blobs; -using System.Collections.Generic; -using System.Linq; - -namespace Azure.Storage.DataMovement.Blobs.Samples -{ - /// - /// Basic Azure Blob Storage samples. - /// - public class Sample01b_HelloWorldAsync : SampleTest - { - public Random _rand = new Random(); - - /// - /// Use a connection string to connect to a Storage account and upload two single blobs. - /// - [Test] - public async Task UploadSingle_ConnectionStringAsync() - { - // Create a temporary Lorem Ipsum file on disk that we can upload - string sourceLocalPath = CreateTempFile(SampleFileContent); - - // Get a connection string to our Azure Storage account. You can - // obtain your connection string from the Azure Portal (click - // Access Keys under Settings in the Portal Storage account blade) - // or using the Azure CLI with: - // - // az storage account show-connection-string --name --resource-group - // - // And you can provide the connection string to your application - // using an environment variable. - - string connectionString = ConnectionString; - string containerName = Randomize("sample-container"); - - // Create a client that can authenticate with a connection string - BlobContainerClient container = new BlobContainerClient(connectionString, containerName); - await container.CreateIfNotExistsAsync(); - try - { - // Get a reference to a source local file - StorageResource sourceResource = new LocalFileStorageResource(sourceLocalPath); - - // Get a reference to a destination blobs - BlockBlobClient destinationBlob = container.GetBlockBlobClient("sample-blob"); - StorageResource destinationResource = new BlockBlobStorageResource(destinationBlob); - - // Create the transfer manager - #region Snippet:CreateTransferManagerSimple_BasePackage - TransferManager transferManager = new TransferManager(new TransferManagerOptions()); - #endregion - - // Create simple transfer single blob upload job - #region Snippet:SimpleBlobUpload_BasePackage - LocalFilesStorageResourceProvider files = new(); - BlobsStorageResourceProvider blobs = new(tokenCredential); - DataTransfer dataTransfer = await transferManager.StartTransferAsync( - files.FromFile("C:/path/to/file.txt"), - blobs.FromBlob("https://myaccount.blob.core.windows.net/mycontainer/myblob"), - cancellationToken: cancellationToken); - await dataTransfer.WaitForCompletionAsync(cancellationToken); - #endregion - - await TransferAsync( - new LocalFileStorageResource(sourceLocalPath), - new BlockBlobStorageResource(destinationBlob)); - } - finally - { - await container.DeleteIfExistsAsync(); - } - } - - [Test] - public async Task MonitorUploadAsync() - { - string sourceLocalPath = CreateTempFile(SampleFileContent); - BlobContainerClient container = new BlobContainerClient(ConnectionString, Randomize("sample-container")); - await container.CreateIfNotExistsAsync(); - - try - { - // Get a reference to a source local file - StorageResource sourceResource = new LocalFileStorageResource(sourceLocalPath); - - // Get a reference to a destination blob - TransferManager transferManager = new TransferManager(); - - string logFile = CreateTempPath(); - - #region Snippet:EnumerateTransfers - async Task CheckTransfersAsync(TransferManager transferManager) - { - await foreach (DataTransfer transfer in transferManager.GetTransfersAsync()) - { - using StreamWriter logStream = File.AppendText(logFile); - logStream.WriteLine(Enum.GetName(typeof(StorageTransferStatus), transfer.TransferStatus)); - } - } - #endregion - - #region Snippet:ListenToTransferEvents - async Task ListenToTransfersAsync(TransferManager transferManager, - StorageResource source, StorageResource destination) - { - TransferOptions transferOptions = new(); - transferOptions.SingleTransferCompleted += (SingleTransferCompletedEventArgs args) => - { - using StreamWriter logStream = File.AppendText(logFile); - logStream.WriteLine($"File Completed Transfer: {args.SourceResource.Path}"); - return Task.CompletedTask; - }; - return await transferManager.StartTransferAsync( - source, - destination, - transferOptions); - } - #endregion - - #region Snippet:ListenToProgress - async Task ListenToProgressAsync(TransferManager transferManager, IProgress progress, - StorageResource source, StorageResource destination) - { - TransferOptions transferOptions = new() - { - ProgressHandler = progress, - // optionally include the below if progress updates on bytes transferred are desired - ProgressHandlerOptions = new() - { - TrackBytesTransferred = true - } - }; - return await transferManager.StartTransferAsync( - source, - destination, - transferOptions); - } - #endregion - - StorageResource destinationResource1 = new BlockBlobStorageResource(container.GetBlockBlobClient("sample-blob-1")); - StorageResource destinationResource2 = new BlockBlobStorageResource(container.GetBlockBlobClient("sample-blob-2")); - DataTransfer t1 = await ListenToTransfersAsync(transferManager, sourceResource, destinationResource1); - DataTransfer t2 = await ListenToProgressAsync(transferManager, new Progress(p => {}), sourceResource, destinationResource2); - await CheckTransfersAsync(transferManager); - await t1.WaitForCompletionAsync(); - await t2.WaitForCompletionAsync(); - } - finally - { - await container.DeleteIfExistsAsync(); - } - } - - [Test] - public async Task PauseTransferAsync() - { - string sourceLocalPath = CreateTempFile(SampleFileContent); - BlobContainerClient container = new BlobContainerClient(ConnectionString, Randomize("sample-container")); - await container.CreateIfNotExistsAsync(); - - try - { - StorageResource sourceResource = new LocalFileStorageResource(sourceLocalPath); - StorageResource destinationResource = new BlockBlobStorageResource(container.GetBlockBlobClient("sample-blob")); - - // Get a reference to a destination blob - TransferManager transferManager = new TransferManager(); - DataTransfer dataTransfer = await transferManager.StartTransferAsync( - sourceResource, destinationResource); - - CancellationToken cancellationToken = CancellationToken.None; - #region Snippet:PauseFromTransfer - await dataTransfer.PauseIfRunningAsync(cancellationToken); - #endregion - - string transferId = dataTransfer.Id; - #region Snippet:PauseFromManager - await transferManager.PauseTransferIfRunningAsync(transferId, cancellationToken); - #endregion - } - finally - { - await container.DeleteIfExistsAsync(); - } - } - - [Test] - public async Task ResumeTransferAsync() - { - string sourceLocalPath = CreateTempFile(SampleFileContent); - BlobContainerClient container = new BlobContainerClient(ConnectionString, Randomize("sample-container")); - await container.CreateIfNotExistsAsync(); - - try - { - StorageResource sourceResource = new LocalFileStorageResource(sourceLocalPath); - StorageResource destinationResource = new BlockBlobStorageResource(container.GetBlockBlobClient("sample-blob")); - - StorageSharedKeyCredential credential = new(StorageAccountName, StorageAccountKey); - async Task<(StorageResource Source, StorageResource Destination)> MakeResourcesAsync(DataTransferProperties info) - { - StorageResource sourceResource = null, destinationResource = null; - // ask DataMovement.Blobs if it can recreate source or destination resources to Blob Storage - if (BlobStorageResources.TryGetResourceProviders( - info, - out BlobStorageResourceProvider blobSrcProvider, - out BlobStorageResourceProvider blobDstProvider)) - { - sourceResource ??= await blobSrcProvider?.MakeResourceAsync(credential); - destinationResource ??= await blobSrcProvider?.MakeResourceAsync(credential); - } - // ask DataMovement if it can recreate source or destination resources to local storage - if (LocalStorageResources.TryGetResourceProviders( - info, - out LocalStorageResourceProvider localSrcProvider, - out LocalStorageResourceProvider localDstProvider)) - { - sourceResource ??= localSrcProvider?.MakeResource(); - destinationResource ??= localDstProvider?.MakeResource(); - } - return (sourceResource, destinationResource); - } - - // Get a reference to a destination blob - TransferManager transferManager = new TransferManager(); - DataTransfer dataTransfer = await transferManager.StartTransferAsync( - sourceResource, destinationResource); - - List resumedTransfers = new(); - await foreach (DataTransferProperties transferProperties in transferManager.GetResumableTransfersAsync()) - { - (StorageResource resumeSource, StorageResource resumeDestination) = await MakeResourcesAsync(transferProperties); - resumedTransfers.Add(await transferManager.ResumeTransferAsync(transferProperties.TransferId, resumeSource, resumeDestination)); - } - - await Task.WhenAll(resumedTransfers.Select(transfer => transfer.WaitForCompletionAsync())); - } - finally - { - await container.DeleteIfExistsAsync(); - } - } - - [Test] - public async Task HandleFailedTransfer() - { - string sourceLocalPath = CreateTempFile(SampleFileContent); - BlobContainerClient container = new BlobContainerClient(ConnectionString, Randomize("sample-container")); - await container.CreateIfNotExistsAsync(); - string logFile = CreateTempPath(); - - try - { - StorageResource sourceResource = new LocalFileStorageResource(sourceLocalPath); - StorageResource destinationResource = new BlockBlobStorageResource(container.GetBlockBlobClient("sample-blob")); - - TransferOptions transferOptions = new(); - #region Snippet:LogIndividualTransferFailures - transferOptions.TransferFailed += (TransferFailedEventArgs args) => - { - using (StreamWriter logStream = File.AppendText(logFile)) - { - // Specifying specific resources that failed, since its a directory transfer - // maybe only one file failed out of many - logStream.WriteLine($"Exception occured with TransferId: {args.TransferId}," + - $"Source Resource: {args.SourceResource.Path}, +" + - $"Destination Resource: {args.DestinationResource.Path}," + - $"Exception Message: {args.Exception.Message}"); - } - return Task.CompletedTask; - }; - #endregion - - TransferManager transferManager = new TransferManager(); - DataTransfer dataTransfer = await transferManager.StartTransferAsync( - sourceResource, destinationResource); - - #region Snippet:LogTotalTransferFailure - await dataTransfer.WaitForCompletionAsync(); - if (dataTransfer.TransferStatus == StorageTransferStatus.CompletedWithFailedTransfers) - { - using (StreamWriter logStream = File.AppendText(logFile)) - { - logStream.WriteLine($"Failure for TransferId: {dataTransfer.Id}"); - } - } - #endregion - } - finally - { - await container.DeleteIfExistsAsync(); - } - } - - public async Task CreateBlobContainerTestDirectory(BlobContainerClient client, int depth = 0, string basePath = default) - { - basePath = basePath ?? Path.GetTempFileName(); - - var dirPath = string.IsNullOrEmpty(basePath) ? Path.GetTempFileName() : $"{basePath}/{Path.GetTempFileName()}"; - - await CreateBlobTestFiles(client, dirPath, 5); - - if (depth > 0) - { - await CreateBlobContainerTestDirectory(client, --depth, dirPath); - } - - return dirPath; - } - - public async Task CreateBlobTestFiles(BlobContainerClient client, string dirPath = default, int count = 1) - { - var buff = new byte[1000]; - - for (int i = 0; i < count; i++) - { - var blobPath = string.IsNullOrEmpty(dirPath) ? $"{Path.GetTempFileName()}.txt" : $"{dirPath}/{Path.GetTempFileName()}.txt"; - - _rand.NextBytes(buff); - - await client.UploadBlobAsync(blobPath, new MemoryStream(buff)); - } - } - - public string CreateLocalTestDirectory(int depth = 0, string basePath = default) - { - basePath = basePath ?? Path.GetTempPath(); - - var dirPath = Path.Combine(basePath, Path.GetTempFileName()); - - Directory.CreateDirectory(dirPath); - - CreateLocalTestFiles(dirPath, 5); - - if (depth > 0) - { - CreateLocalTestDirectory(--depth, dirPath); - } - - return dirPath; - } - - public void CreateLocalTestFiles(string dirPath, int count = 1) - { - var buff = new byte[1000]; - - for (int i = 0; i < count; i++) - { - var filePath = Path.Combine(dirPath, Path.GetTempFileName() + ".txt"); - - _rand.NextBytes(buff); - - File.WriteAllText(filePath, Convert.ToBase64String(buff)); - } - } - - public struct StoredCredentials - { - public StorageResourceContainer SourceContainer { get; set; } - public StorageResourceContainer DestinationContainer { get; set; } - - public StoredCredentials( - StorageResourceContainer sourceContainer, - StorageResourceContainer destinationContainer) - { - SourceContainer = sourceContainer; - DestinationContainer = destinationContainer; - } - } - } -} From 6054d754784c29e4177c205e03263faf31e8460e Mon Sep 17 00:00:00 2001 From: Amanda Nguyen Date: Mon, 6 Jan 2025 15:19:52 -0800 Subject: [PATCH 3/7] Update blob readme --- sdk/storage/Azure.Storage.Blobs/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sdk/storage/Azure.Storage.Blobs/README.md b/sdk/storage/Azure.Storage.Blobs/README.md index 3ac35d02eb56..36557dfa1415 100644 --- a/sdk/storage/Azure.Storage.Blobs/README.md +++ b/sdk/storage/Azure.Storage.Blobs/README.md @@ -188,6 +188,8 @@ Get started with our [Blob samples][samples]: 1. [Hello World](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/samples/Sample01a_HelloWorld.cs): Upload, download, and list blobs (or [asynchronously](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/samples/Sample01b_HelloWorldAsync.cs)) 2. [Auth](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/samples/Sample02_Auth.cs): Authenticate with connection strings, public access, shared keys, shared access signatures, and Azure Active Directory. + +For more advanced scenarios like transferring blob virtual directories, we recommend looking into our [Azure.Storage.DataMovement](https://www.nuget.org/packages/Azure.Storage.DataMovement) and [Azure.Storage.DataMovement.Blob](https://www.nuget.org/packages/Azure.Storage.DataMovement.Blobs) packages. Get started with our [DataMovement Blob Samples](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs). ## Contributing See the [Storage CONTRIBUTING.md][storage_contrib] for details on building, From fddd830b1e7b06cc50600dcc5b6b4ede93728060 Mon Sep 17 00:00:00 2001 From: Amanda Nguyen Date: Tue, 7 Jan 2025 12:55:42 -0800 Subject: [PATCH 4/7] Fix sample snippet references to use blobs; added extension samples to base blob README --- sdk/storage/Azure.Storage.Blobs/README.md | 68 +++++++++++ .../README.md | 1 + .../samples/Sample01b_HelloWorldAsync.cs | 112 +++++++++++++++++- .../Azure.Storage.DataMovement/README.md | 44 ++++--- 4 files changed, 196 insertions(+), 29 deletions(-) diff --git a/sdk/storage/Azure.Storage.Blobs/README.md b/sdk/storage/Azure.Storage.Blobs/README.md index 36557dfa1415..b421393e753e 100644 --- a/sdk/storage/Azure.Storage.Blobs/README.md +++ b/sdk/storage/Azure.Storage.Blobs/README.md @@ -188,8 +188,76 @@ Get started with our [Blob samples][samples]: 1. [Hello World](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/samples/Sample01a_HelloWorld.cs): Upload, download, and list blobs (or [asynchronously](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/samples/Sample01b_HelloWorldAsync.cs)) 2. [Auth](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/samples/Sample02_Auth.cs): Authenticate with connection strings, public access, shared keys, shared access signatures, and Azure Active Directory. +## Advanced Scenarios using Azure.DataMovement.Blobs For more advanced scenarios like transferring blob virtual directories, we recommend looking into our [Azure.Storage.DataMovement](https://www.nuget.org/packages/Azure.Storage.DataMovement) and [Azure.Storage.DataMovement.Blob](https://www.nuget.org/packages/Azure.Storage.DataMovement.Blobs) packages. Get started with our [DataMovement Blob Samples](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs). + +Upload a local directory to the root of the `BlobContainerClient`. +```C# Snippet:ExtensionMethodSimpleUploadToRoot +DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath); + +await transfer.WaitForCompletionAsync(); +``` + +Upload a local directory to a virtual blob directory in the `BlobContainerClient` by specifying a directory prefix +```C# Snippet:ExtensionMethodSimpleUploadToDirectoryPrefix +DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath, blobDirectoryPrefix); + +await transfer.WaitForCompletionAsync(); +``` + +Upload a local directory to a virtual blob directory in the `BlobContainerClient` specifying more advanced options +```C# Snippet:ExtensionMethodSimpleUploadWithOptions +BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions +{ + BlobContainerOptions = new BlobStorageResourceContainerOptions + { + BlobDirectoryPrefix = blobDirectoryPrefix + }, + TransferOptions = new DataTransferOptions() + { + CreationPreference = StorageResourceCreationPreference.OverwriteIfExists, + } +}; + +DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath, options); + +await transfer.WaitForCompletionAsync(); +``` + +Download the entire `BlobContainerClient` to a local directory +```C# Snippet:ExtensionMethodSimpleDownloadContainer +DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath); + +await transfer.WaitForCompletionAsync(); +``` + +Download a virtual blob directory in the `BlobContainerClient` by specifying a directory prefix +```C# Snippet:ExtensionMethodSimpleDownloadContainerDirectory +DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix); + +await tranfer.WaitForCompletionAsync(); +``` + +Download from the `BlobContainerClient` specifying more advanced options +```C# Snippet:ExtensionMethodSimpleDownloadContainerDirectoryWithOptions +BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions +{ + BlobContainerOptions = new BlobStorageResourceContainerOptions + { + BlobDirectoryPrefix = blobDirectoryPrefix + }, + TransferOptions = new DataTransferOptions() + { + CreationPreference = StorageResourceCreationPreference.OverwriteIfExists, + } +}; + +DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options); + +await tranfer.WaitForCompletionAsync(); +``` + ## Contributing See the [Storage CONTRIBUTING.md][storage_contrib] for details on building, diff --git a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md index b739d0caca89..7fd57a94b47e 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md +++ b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md @@ -345,6 +345,7 @@ additional questions or comments. [azure_sub]: https://azure.microsoft.com/free/dotnet/ [RequestFailedException]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/core/Azure.Core/src/RequestFailedException.cs [error_codes]: https://learn.microsoft.com/rest/api/storageservices/common-rest-api-error-codes +[samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement/samples [share_samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Files.Shares/samples [storage_contrib]: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/CONTRIBUTING.md [cla]: https://cla.microsoft.com diff --git a/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs b/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs index 198dcf58e4ab..446236cc0576 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs +++ b/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs @@ -184,20 +184,24 @@ public async Task UploadSingle_ConnectionStringAsync() // And you can provide the connection string to your application // using an environment variable. - string connectionString = ConnectionString; + TokenCredential tokenCredential = new DefaultAzureCredential(); string containerName = Randomize("sample-container"); - // Create a client that can authenticate with a connection string - BlobContainerClient container = new BlobContainerClient(connectionString, containerName); + BlobServiceClient serviceClient = new BlobServiceClient(ActiveDirectoryBlobUri, tokenCredential); + BlobContainerClient container = serviceClient.GetBlobContainerClient(containerName); await container.CreateIfNotExistsAsync(); try { - BlobsStorageResourceProvider blobs = new(new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey)); - LocalFilesStorageResourceProvider files = new(); + #region Snippet:CreateTransferManagerSimple_BasePackage + TransferManager transferManager = new TransferManager(new TransferManagerOptions()); + #endregion // Get a reference to a destination blobs Uri destinationBlobUri = container.GetBlockBlobClient("sample-blob").Uri; - TransferManager transferManager = new TransferManager(new TransferManagerOptions()); + + #region Snippet:SimpleBlobUpload_BasePackage + LocalFilesStorageResourceProvider files = new(); + BlobsStorageResourceProvider blobs = new(tokenCredential); // Create simple transfer single blob upload job #region Snippet:SimpleBlobUpload @@ -206,6 +210,7 @@ public async Task UploadSingle_ConnectionStringAsync() destinationResource: blobs.FromBlob(destinationBlobUri)); await dataTransfer.WaitForCompletionAsync(); #endregion + #endregion } finally { @@ -552,6 +557,7 @@ public async Task UploadDirectory_EventHandler_SasAsync() } return Task.CompletedTask; }; + #region Snippet:LogIndividualTransferFailures transferOptions.ItemTransferFailed += (TransferItemFailedEventArgs args) => { using (StreamWriter logStream = File.AppendText(logFile)) @@ -565,6 +571,7 @@ public async Task UploadDirectory_EventHandler_SasAsync() } return Task.CompletedTask; }; + #endregion TransferManager transferManager = new TransferManager(options); BlobsStorageResourceProvider blobs = new(); LocalFilesStorageResourceProvider files = new(); @@ -893,7 +900,9 @@ public async Task PauseAndResumeAsync_ManagerId() string transferId = dataTransfer.Id; // Pause from the Transfer Manager using the Transfer Id + #region Snippet:PauseFromManager await transferManager.PauseTransferIfRunningAsync(transferId); + #endregion // Resume all transfers List transfers = await transferManager.ResumeAllTransfersAsync(); @@ -951,7 +960,9 @@ public async Task PauseAndResumeAsync_DataTransferPause() destinationResource: destinationResource); // Pause from the DataTransfer object + #region Snippet:PauseFromTransfer await dataTransfer.PauseAsync(); + #endregion DataTransfer resumedTransfer = await transferManager.ResumeTransferAsync( transferId: dataTransfer.Id); @@ -1104,6 +1115,95 @@ public async Task DownloadDirectory() } } + [Test] + public async Task MonitorUploadAsync() + { + string sourceLocalPath = CreateTempFile(SampleFileContent); + BlobContainerClient container = new BlobContainerClient(ConnectionString, Randomize("sample-container")); + await container.CreateIfNotExistsAsync(); + + try + { + // Get a reference to a source local file + LocalFilesStorageResourceProvider files = new(); + StorageResource sourceResource = files.FromFile(sourceLocalPath); + + // Get a reference to a destination blob + TransferManager transferManager = new TransferManager(); + + string logFile = CreateTempPath(); + + #region Snippet:EnumerateTransfers + async Task CheckTransfersAsync(TransferManager transferManager) + { + await foreach (DataTransfer transfer in transferManager.GetTransfersAsync()) + { + using StreamWriter logStream = File.AppendText(logFile); + logStream.WriteLine(Enum.GetName(typeof(DataTransferStatus), transfer.TransferStatus)); + } + } + #endregion + + #region Snippet:ListenToTransferEvents + async Task ListenToTransfersAsync(TransferManager transferManager, + StorageResource source, StorageResource destination) + { + DataTransferOptions transferOptions = new(); + transferOptions.ItemTransferCompleted += (TransferItemCompletedEventArgs args) => + { + using StreamWriter logStream = File.AppendText(logFile); + logStream.WriteLine($"File Completed Transfer: {args.SourceResource.Uri.LocalPath}"); + return Task.CompletedTask; + }; + return await transferManager.StartTransferAsync( + source, + destination, + transferOptions); + } + #endregion + + #region Snippet:ListenToProgress + async Task ListenToProgressAsync(TransferManager transferManager, IProgress progress, + StorageResource source, StorageResource destination) + { + DataTransferOptions transferOptions = new() + { + // optionally include the below if progress updates on bytes transferred are desired + ProgressHandlerOptions = new(progress, trackBytesTransferred: true) + }; + return await transferManager.StartTransferAsync( + source, + destination, + transferOptions); + } + #endregion + + BlobsStorageResourceProvider blobs = new(); + StorageResource destinationResource1 = blobs.FromClient(container.GetBlockBlobClient("sample-blob-1")); + StorageResource destinationResource2 = blobs.FromClient(container.GetBlockBlobClient("sample-blob-2")); + DataTransfer dataTransfer1 = await ListenToTransfersAsync(transferManager, sourceResource, destinationResource1); + DataTransfer dataTransfer2 = await ListenToProgressAsync(transferManager, new Progress(p => { }), sourceResource, destinationResource2); + await CheckTransfersAsync(transferManager); + await dataTransfer1.WaitForCompletionAsync(); + + #region Snippet:LogTotalTransferFailure + await dataTransfer2.WaitForCompletionAsync(); + if (dataTransfer2.TransferStatus.State == DataTransferState.Completed + && dataTransfer2.TransferStatus.HasFailedItems) + { + using (StreamWriter logStream = File.AppendText(logFile)) + { + logStream.WriteLine($"Failure for TransferId: {dataTransfer2.Id}"); + } + } + #endregion + } + finally + { + await container.DeleteIfExistsAsync(); + } + } + public async Task CreateBlobContainerTestDirectory(BlobContainerClient client, int depth = 0, string basePath = default) { basePath = basePath ?? Path.GetTempFileName(); diff --git a/sdk/storage/Azure.Storage.DataMovement/README.md b/sdk/storage/Azure.Storage.DataMovement/README.md index 913ff4250481..36d555d4fdbd 100644 --- a/sdk/storage/Azure.Storage.DataMovement/README.md +++ b/sdk/storage/Azure.Storage.DataMovement/README.md @@ -86,11 +86,12 @@ The below sample demonstrates `StorageResourceProvider` use to start transfers b ```C# Snippet:SimpleBlobUpload_BasePackage LocalFilesStorageResourceProvider files = new(); BlobsStorageResourceProvider blobs = new(tokenCredential); + +// Create simple transfer single blob upload job DataTransfer dataTransfer = await transferManager.StartTransferAsync( - files.FromFile("C:/path/to/file.txt"), - blobs.FromBlob("https://myaccount.blob.core.windows.net/mycontainer/myblob"), - cancellationToken: cancellationToken); -await dataTransfer.WaitForCompletionAsync(cancellationToken); + sourceResource: files.FromFile(sourceLocalPath), + destinationResource: blobs.FromBlob(destinationBlobUri)); +await dataTransfer.WaitForCompletionAsync(); ``` ### Resuming Existing Transfers @@ -154,7 +155,7 @@ async Task CheckTransfersAsync(TransferManager transferManager) await foreach (DataTransfer transfer in transferManager.GetTransfersAsync()) { using StreamWriter logStream = File.AppendText(logFile); - logStream.WriteLine(Enum.GetName(typeof(StorageTransferStatus), transfer.TransferStatus)); + logStream.WriteLine(Enum.GetName(typeof(DataTransferStatus), transfer.TransferStatus)); } } ``` @@ -173,11 +174,11 @@ A function that listens to status events for a given transfer: async Task ListenToTransfersAsync(TransferManager transferManager, StorageResource source, StorageResource destination) { - TransferOptions transferOptions = new(); - transferOptions.SingleTransferCompleted += (SingleTransferCompletedEventArgs args) => + DataTransferOptions transferOptions = new(); + transferOptions.ItemTransferCompleted += (TransferItemCompletedEventArgs args) => { using StreamWriter logStream = File.AppendText(logFile); - logStream.WriteLine($"File Completed Transfer: {args.SourceResource.Path}"); + logStream.WriteLine($"File Completed Transfer: {args.SourceResource.Uri.LocalPath}"); return Task.CompletedTask; }; return await transferManager.StartTransferAsync( @@ -194,17 +195,13 @@ When starting a transfer, `TransferOptions` allows setting a progress handler th A function that listens to progress updates for a given transfer with a supplied `IProgress`: ```C# Snippet:ListenToProgress -async Task ListenToProgressAsync(TransferManager transferManager, IProgress progress, +async Task ListenToProgressAsync(TransferManager transferManager, IProgress progress, StorageResource source, StorageResource destination) { - TransferOptions transferOptions = new() + DataTransferOptions transferOptions = new() { - ProgressHandler = progress, // optionally include the below if progress updates on bytes transferred are desired - ProgressHandlerOptions = new() - { - TrackBytesTransferred = true - } + ProgressHandlerOptions = new(progress, trackBytesTransferred: true) }; return await transferManager.StartTransferAsync( source, @@ -218,11 +215,11 @@ async Task ListenToProgressAsync(TransferManager transferManager, Transfers can be paused either by a given `DataTransfer` or through the `TransferManager` handling the transfer by referencing the transfer ID. The ID can be found on the `DataTransfer` object you received upon transfer start. ```C# Snippet:PauseFromTransfer -await dataTransfer.PauseIfRunningAsync(cancellationToken); +await dataTransfer.PauseAsync(); ``` ```C# Snippet:PauseFromManager -await transferManager.PauseTransferIfRunningAsync(transferId, cancellationToken); +await transferManager.PauseTransferIfRunningAsync(transferId); ``` ### Handling Failed Transfers @@ -232,12 +229,13 @@ Transfer failure can be observed by checking the `DataTransfer` status upon comp Below logs failure for a single transfer by checking its status after completion. ```C# Snippet:LogTotalTransferFailure -await dataTransfer.WaitForCompletionAsync(); -if (dataTransfer.TransferStatus == StorageTransferStatus.CompletedWithFailedTransfers) +await dataTransfer2.WaitForCompletionAsync(); +if (dataTransfer2.TransferStatus.State == DataTransferState.Completed + && dataTransfer2.TransferStatus.HasFailedItems) { using (StreamWriter logStream = File.AppendText(logFile)) { - logStream.WriteLine($"Failure for TransferId: {dataTransfer.Id}"); + logStream.WriteLine($"Failure for TransferId: {dataTransfer2.Id}"); } } ``` @@ -245,15 +243,15 @@ if (dataTransfer.TransferStatus == StorageTransferStatus.CompletedWithFailedTran Below logs individual failures in a container transfer via `TransferOptions` events. ```C# Snippet:LogIndividualTransferFailures -transferOptions.TransferFailed += (TransferFailedEventArgs args) => +transferOptions.ItemTransferFailed += (TransferItemFailedEventArgs args) => { using (StreamWriter logStream = File.AppendText(logFile)) { // Specifying specific resources that failed, since its a directory transfer // maybe only one file failed out of many logStream.WriteLine($"Exception occured with TransferId: {args.TransferId}," + - $"Source Resource: {args.SourceResource.Path}, +" + - $"Destination Resource: {args.DestinationResource.Path}," + + $"Source Resource: {args.SourceResource.Uri.AbsoluteUri}, +" + + $"Destination Resource: {args.DestinationResource.Uri.AbsoluteUri}," + $"Exception Message: {args.Exception.Message}"); } return Task.CompletedTask; From 4e5e4d734463ea6b1280720d0c6a00494dc01484 Mon Sep 17 00:00:00 2001 From: Amanda Nguyen Date: Tue, 7 Jan 2025 13:01:26 -0800 Subject: [PATCH 5/7] Mispellings --- sdk/storage/Azure.Storage.Blobs/README.md | 8 ++++---- sdk/storage/Azure.Storage.DataMovement.Blobs/README.md | 8 ++++---- .../samples/Sample01b_HelloWorldAsync.cs | 10 +++++----- .../src/BlobContainerClientTransferOptions.cs | 2 +- .../src/ShareDirectoryClientTransferOptions.cs | 2 +- sdk/storage/Azure.Storage.DataMovement/README.md | 2 +- .../src/QueueStageChunkArgs.cs | 4 ++-- 7 files changed, 18 insertions(+), 18 deletions(-) diff --git a/sdk/storage/Azure.Storage.Blobs/README.md b/sdk/storage/Azure.Storage.Blobs/README.md index b421393e753e..ec2b627451e1 100644 --- a/sdk/storage/Azure.Storage.Blobs/README.md +++ b/sdk/storage/Azure.Storage.Blobs/README.md @@ -234,9 +234,9 @@ await transfer.WaitForCompletionAsync(); Download a virtual blob directory in the `BlobContainerClient` by specifying a directory prefix ```C# Snippet:ExtensionMethodSimpleDownloadContainerDirectory -DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix); +DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix); -await tranfer.WaitForCompletionAsync(); +await transfer.WaitForCompletionAsync(); ``` Download from the `BlobContainerClient` specifying more advanced options @@ -253,9 +253,9 @@ BlobContainerClientTransferOptions options = new BlobContainerClientTransferOpti } }; -DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options); +DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options); -await tranfer.WaitForCompletionAsync(); +await transfer.WaitForCompletionAsync(); ``` ## Contributing diff --git a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md index 7fd57a94b47e..3174b83d83b5 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md +++ b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md @@ -279,9 +279,9 @@ await transfer.WaitForCompletionAsync(); Download a directory in the container by specifying a directory prefix ```C# Snippet:ExtensionMethodSimpleDownloadContainerDirectory -DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix); +DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix); -await tranfer.WaitForCompletionAsync(); +await transfer.WaitForCompletionAsync(); ``` Download from the container specifying more advanced options @@ -298,9 +298,9 @@ BlobContainerClientTransferOptions options = new BlobContainerClientTransferOpti } }; -DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options); +DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options); -await tranfer.WaitForCompletionAsync(); +await transfer.WaitForCompletionAsync(); ``` ## Troubleshooting diff --git a/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs b/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs index 446236cc0576..bb383db9a584 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs +++ b/sdk/storage/Azure.Storage.DataMovement.Blobs/samples/Sample01b_HelloWorldAsync.cs @@ -564,7 +564,7 @@ public async Task UploadDirectory_EventHandler_SasAsync() { // Specifying specific resources that failed, since its a directory transfer // maybe only one file failed out of many - logStream.WriteLine($"Exception occured with TransferId: {args.TransferId}," + + logStream.WriteLine($"Exception occurred with TransferId: {args.TransferId}," + $"Source Resource: {args.SourceResource.Uri.AbsoluteUri}, +" + $"Destination Resource: {args.DestinationResource.Uri.AbsoluteUri}," + $"Exception Message: {args.Exception.Message}"); @@ -1084,9 +1084,9 @@ public async Task DownloadDirectory() { // download a virtual directory, with a specific prefix, within the container #region Snippet:ExtensionMethodSimpleDownloadContainerDirectory - DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix); + DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix); - await tranfer.WaitForCompletionAsync(); + await transfer.WaitForCompletionAsync(); #endregion } { @@ -1103,9 +1103,9 @@ public async Task DownloadDirectory() } }; - DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options); + DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options); - await tranfer.WaitForCompletionAsync(); + await transfer.WaitForCompletionAsync(); #endregion } } diff --git a/sdk/storage/Azure.Storage.DataMovement.Blobs/src/BlobContainerClientTransferOptions.cs b/sdk/storage/Azure.Storage.DataMovement.Blobs/src/BlobContainerClientTransferOptions.cs index 5bba76575758..884f82c52401 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Blobs/src/BlobContainerClientTransferOptions.cs +++ b/sdk/storage/Azure.Storage.DataMovement.Blobs/src/BlobContainerClientTransferOptions.cs @@ -18,7 +18,7 @@ public class BlobContainerClientTransferOptions public BlobStorageResourceContainerOptions BlobContainerOptions { get; set; } /// - /// Options pertaining to the data tranfer. + /// Options pertaining to the data transfer. /// public DataTransferOptions TransferOptions { get; set; } } diff --git a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/src/ShareDirectoryClientTransferOptions.cs b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/src/ShareDirectoryClientTransferOptions.cs index 79a5f36d19df..116ecf89a6fe 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/src/ShareDirectoryClientTransferOptions.cs +++ b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/src/ShareDirectoryClientTransferOptions.cs @@ -18,7 +18,7 @@ public class ShareDirectoryClientTransferOptions public ShareFileStorageResourceOptions ShareDirectoryOptions { get; set; } /// - /// Options pertaining to the data tranfer. + /// Options pertaining to the data transfer. /// public DataTransferOptions TransferOptions { get; set; } } diff --git a/sdk/storage/Azure.Storage.DataMovement/README.md b/sdk/storage/Azure.Storage.DataMovement/README.md index 36d555d4fdbd..f27dd2622fb2 100644 --- a/sdk/storage/Azure.Storage.DataMovement/README.md +++ b/sdk/storage/Azure.Storage.DataMovement/README.md @@ -249,7 +249,7 @@ transferOptions.ItemTransferFailed += (TransferItemFailedEventArgs args) => { // Specifying specific resources that failed, since its a directory transfer // maybe only one file failed out of many - logStream.WriteLine($"Exception occured with TransferId: {args.TransferId}," + + logStream.WriteLine($"Exception occurred with TransferId: {args.TransferId}," + $"Source Resource: {args.SourceResource.Uri.AbsoluteUri}, +" + $"Destination Resource: {args.DestinationResource.Uri.AbsoluteUri}," + $"Exception Message: {args.Exception.Message}"); diff --git a/sdk/storage/Azure.Storage.DataMovement/src/QueueStageChunkArgs.cs b/sdk/storage/Azure.Storage.DataMovement/src/QueueStageChunkArgs.cs index 7bb59cc7f329..4a4d7026bb56 100644 --- a/sdk/storage/Azure.Storage.DataMovement/src/QueueStageChunkArgs.cs +++ b/sdk/storage/Azure.Storage.DataMovement/src/QueueStageChunkArgs.cs @@ -4,14 +4,14 @@ namespace Azure.Storage.DataMovement { /// - /// This class is interchangable for + /// This class is interchangeable for /// Stage Block (Put Block), Stage Block From Uri (Put Block From URL), /// Append Block (Append Block), Append Block From Uri (Append Block From URL), /// Upload Page (Put Page), Upload Pages From Uri (Put Pages From URL) /// /// Basically any transfer operation that must end in a Commit Block List /// will end up using this internal event argument to track the success - /// and the bytes transferred to ensure the correct amount of bytes are tranferred. + /// and the bytes transferred to ensure the correct amount of bytes are transferred. /// internal class QueueStageChunkArgs { From f8f78bce13348d69811e0c39d242f3fbd520df98 Mon Sep 17 00:00:00 2001 From: Amanda Nguyen Date: Tue, 7 Jan 2025 13:38:58 -0800 Subject: [PATCH 6/7] Fix links --- sdk/storage/Azure.Storage.DataMovement.Blobs/README.md | 4 ++-- sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md index 3174b83d83b5..9a7d51e1adfd 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md +++ b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md @@ -311,7 +311,7 @@ See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Loggin Get started with our [Share Files Samples][share_samples]. -For more base Transfer Manager scenarios see [DataMovement samples][samples]. +For more base Transfer Manager scenarios see [DataMovement samples][datamovement_base]. ## Contributing @@ -345,7 +345,7 @@ additional questions or comments. [azure_sub]: https://azure.microsoft.com/free/dotnet/ [RequestFailedException]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/core/Azure.Core/src/RequestFailedException.cs [error_codes]: https://learn.microsoft.com/rest/api/storageservices/common-rest-api-error-codes -[samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement/samples +[datamovement_base]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement [share_samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Files.Shares/samples [storage_contrib]: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/CONTRIBUTING.md [cla]: https://cla.microsoft.com diff --git a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md index cd22c340f919..12469332d7f7 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md +++ b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md @@ -171,7 +171,7 @@ See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Loggin Get started with our [Blob DataMovement samples][blob_samples]. -For more base Transfer Manager scenarios see [DataMovement samples][samples]. +For more base Transfer Manager scenarios see [DataMovement samples][datamovement_base]. ## Contributing @@ -205,7 +205,7 @@ additional questions or comments. [azure_sub]: https://azure.microsoft.com/free/dotnet/ [RequestFailedException]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/core/Azure.Core/src/RequestFailedException.cs [error_codes]: https://learn.microsoft.com/rest/api/storageservices/common-rest-api-error-codes -[samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement/samples +[datamovement_base]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement [blob_samples]: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/storage/Azure.Storage.DataMovement.Blobs/samples [storage_contrib]: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/CONTRIBUTING.md [cla]: https://cla.microsoft.com From 861d17794f5f900c29fd7a75ba37d5597082fd80 Mon Sep 17 00:00:00 2001 From: Amanda Nguyen Date: Wed, 8 Jan 2025 12:52:07 -0800 Subject: [PATCH 7/7] Fix links --- sdk/storage/Azure.Storage.DataMovement.Blobs/README.md | 2 +- sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md | 2 +- sdk/storage/Azure.Storage.DataMovement/README.md | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md index 9a7d51e1adfd..2fcda93010f6 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md +++ b/sdk/storage/Azure.Storage.DataMovement.Blobs/README.md @@ -305,7 +305,7 @@ await transfer.WaitForCompletionAsync(); ## Troubleshooting -See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Logging](https://learn.microsoft.com/en-us/dotnet/azure/sdk/logging) to assist with any troubleshooting. +See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Logging](https://learn.microsoft.com/dotnet/azure/sdk/logging) to assist with any troubleshooting. ## Next steps diff --git a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md index 12469332d7f7..097781d1c395 100644 --- a/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md +++ b/sdk/storage/Azure.Storage.DataMovement.Files.Shares/README.md @@ -165,7 +165,7 @@ await directoryTransfer.WaitForCompletionAsync(); ## Troubleshooting -See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Logging](https://learn.microsoft.com/en-us/dotnet/azure/sdk/logging) to assist with any troubleshooting. +See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Logging](https://learn.microsoft.com/dotnet/azure/sdk/logging) to assist with any troubleshooting. ## Next steps diff --git a/sdk/storage/Azure.Storage.DataMovement/README.md b/sdk/storage/Azure.Storage.DataMovement/README.md index f27dd2622fb2..d7c77d61a778 100644 --- a/sdk/storage/Azure.Storage.DataMovement/README.md +++ b/sdk/storage/Azure.Storage.DataMovement/README.md @@ -47,7 +47,7 @@ Authentication is specific to the targeted storage service. Please see documenta ## Key concepts The Azure Storage DataMovement client library contains shared infrastructure like -[TokenCredential](https://learn.microsoft.com/en-us/dotnet/api/azure.core.tokencredential?view=azure-dotnet), [TransferManager](#setup-the-transfermanager) and [RequestFailedException][RequestFailedException]. +[TokenCredential](https://learn.microsoft.com/dotnet/api/azure.core.tokencredential?view=azure-dotnet), [TransferManager](#setup-the-transfermanager) and [RequestFailedException][RequestFailedException]. ### Thread safety We guarantee that all client instance methods are thread-safe and independent of each other ([guideline](https://azure.github.io/azure-sdk/dotnet_introduction.html#dotnet-service-methods-thread-safety)). This ensures that the recommendation of reusing client instances is always safe, even across threads. @@ -270,7 +270,7 @@ StorageResource directoryResource = files.FromDirectory("C:/path/to/dir"); ## Troubleshooting -See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Logging](https://learn.microsoft.com/en-us/dotnet/azure/sdk/logging) to assist with any troubleshooting. +See [Handling Failed Transfers](#handling-failed-transfers) and [Enabling Logging](https://learn.microsoft.com/dotnet/azure/sdk/logging) to assist with any troubleshooting. ## Next steps