Skip to content

Updated Default Concurrency count #54428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nickliu-msft wants to merge 5 commits into
base: feature/storage/stg101base
Choose a base branch
from
Open

Updated Default Concurrency count #54428

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
837e65c
Updated default concurrency to processor count + compat switch logic
nickliu-msft Dec 10, 2025
449d18e
updated default concurrency
nickliu-msft Dec 11, 2025
374bc66
updated changelogs
nickliu-msft Dec 11, 2025
330d533
slight revision
nickliu-msft Dec 11, 2025
14889a1
More descriptive changelogs
nickliu-msft Dec 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions sdk/storage/Azure.Storage.Blobs/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@
- Added cross-tenant support for Principal-Bound User Delegation SAS.
- Added support for Dynamic User Delegation SAS.

### Other Changes
- Changed the default concurrency download count from 5 to Math.Clamp(Environment.ProcessorCount * 2, 8, 32). This controls the maximum number of concurrent tasks that will be used during large downloads, and this change should result in higher throughput for these operations by default in most environments. This can be reverted by enabling "Azure.Storage.UseLegacyDefaultConcurrency" in the AppContext switch or "AZURE_STORAGE_USE_LEGACY_DEFAULT_CONCURRENCY" in the environment variable.

### Bugs Fixed
- Added BlobErrorCode.IncrementalCopyOfEarlierSnapshotNotAllowed, deprecated BlobErrorCode.IncrementalCopyOfEarlierVersionSnapshotNotAllowed.
- Added support for missing SkuName values.
Expand Down
6 changes: 5 additions & 1 deletion sdk/storage/Azure.Storage.Blobs/src/PartitionedDownloader.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ private DownloadTransferValidationOptions ValidationOptions

private readonly ArrayPool<byte> _arrayPool;

private readonly int DefaultConcurrentTransfersCount = Math.Min(Math.Max(Environment.ProcessorCount * 2, 8), 32);
Copy link
Member Author

@nickliu-msft nickliu-msft Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentionally had to do this since Math.Clamp() is not supported for netstandard2.0 and earlier


public PartitionedDownloader(
BlobBaseClient client,
StorageTransferOptions transferOptions = default,
Expand All @@ -89,7 +91,9 @@ public PartitionedDownloader(
}
else
{
_maxWorkerCount = Constants.Blob.Block.DefaultConcurrentTransfersCount;
_maxWorkerCount = CompatSwitches.UseLegacyDefaultConcurrency
? Constants.Blob.Block.LegacyDefaultConcurrentTransfersCount
: DefaultConcurrentTransfersCount;
}

// Set _rangeSize
Expand Down
3 changes: 3 additions & 0 deletions sdk/storage/Azure.Storage.Common/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
### Features Added
- This release contains bug fixes to improve quality.

### Other Changes
- Changed the default concurrency upload count from 5 to Math.Clamp(Environment.ProcessorCount * 2, 8, 32). This controls the maximum number of concurrent tasks that will be used during large uploads, and this change should result in higher throughput for these operations by default in most environments. This can be reverted by enabling "Azure.Storage.UseLegacyDefaultConcurrency" in the AppContext switch or "AZURE_STORAGE_USE_LEGACY_DEFAULT_CONCURRENCY" in the environment variable.

## 12.26.0-beta.1 (2025-11-17)

### Features Added
Expand Down
5 changes: 5 additions & 0 deletions sdk/storage/Azure.Storage.Common/src/Shared/CompatSwitches.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,10 @@ internal static class CompatSwitches

public static bool DisableExpectContinueHeader => _disableExpectContinueHeader
??= AppContextSwitchHelper.GetConfigValue(Constants.DisableExpectContinueHeaderSwitchName, Constants.DisableExpectContinueHeaderEnvVar);

private static bool? _useLegacyDefaultConcurrency;

public static bool UseLegacyDefaultConcurrency => _useLegacyDefaultConcurrency
??= AppContextSwitchHelper.GetConfigValue(Constants.UseLegacyDefaultConcurrencySwitchName, Constants.UseLegacyDefaultConcurrencyEnvVar);
}
}
16 changes: 14 additions & 2 deletions sdk/storage/Azure.Storage.Common/src/Shared/Constants.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

using System;
using System.Collections.Generic;
using System.ComponentModel;

namespace Azure.Storage
{
Expand Down Expand Up @@ -129,6 +130,9 @@ internal static class Constants
public const string DisableExpectContinueHeaderSwitchName = "Azure.Storage.DisableExpectContinueHeader";
public const string DisableExpectContinueHeaderEnvVar = "AZURE_STORAGE_DISABLE_EXPECT_CONTINUE_HEADER";

public const string UseLegacyDefaultConcurrencySwitchName = "Azure.Storage.UseLegacyDefaultConcurrency";
public const string UseLegacyDefaultConcurrencyEnvVar = "AZURE_STORAGE_USE_LEGACY_DEFAULT_CONCURRENCY";

public const string DefaultScope = "/.default";

/// <summary>
Expand Down Expand Up @@ -241,7 +245,9 @@ internal static class Append

internal static class Block
{
public const int DefaultConcurrentTransfersCount = 5;
[EditorBrowsable(EditorBrowsableState.Never)]
public const int DefaultConcurrentTransfersCount = LegacyDefaultConcurrentTransfersCount;
Copy link
Member

@jaschrep-msft jaschrep-msft Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get that this is to avoid altering a ton of files but i think it's probably better for code health to do that. We don't want to be confused about whether we're using old or new values. We want to know it's legacy when we see it. I'd rather this const just not exist.

Copy link
Member Author

@nickliu-msft nickliu-msft Dec 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So Constants.DefaultConcurrentTransfersCount is no longer being referenced at all, we are replacing all usage with the new Constants.LegacyDefaultConcurrentTransfersCount. The reason why we hid it rather than removed it is to avoid a breaking change

public const int LegacyDefaultConcurrentTransfersCount = 5;
public const int DefaultInitalDownloadRangeSize = 256 * Constants.MB; // 256 MB
public const int Pre_2019_12_12_MaxUploadBytes = 256 * Constants.MB; // 256 MB
public const long MaxUploadBytes = 5000L * Constants.MB; // 5000MB
Expand Down Expand Up @@ -378,7 +384,13 @@ internal static class DataLake
/// <summary>
/// Default concurrent transfers count.
/// </summary>
public const int DefaultConcurrentTransfersCount = 5;
[EditorBrowsable(EditorBrowsableState.Never)]
public const int DefaultConcurrentTransfersCount = LegacyDefaultConcurrentTransfersCount;

/// <summary>
/// Legacy default concurrent transfers count.
/// </summary>
public const int LegacyDefaultConcurrentTransfersCount = 5;

/// <summary>
/// Max upload bytes for less than Service Version 2019-12-12.
Expand Down
9 changes: 8 additions & 1 deletion sdk/storage/Azure.Storage.Common/src/Shared/PartitionedUploader.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,11 @@ public struct Behaviors
/// </summary>
private readonly string _operationName;

/// <summary>
/// The default conconcurrency transfer count.
/// </summary>
private readonly int DefaultConcurrentTransfersCount = Math.Min(Math.Max(Environment.ProcessorCount * 2, 8), 32);

public PartitionedUploader(
Behaviors behaviors,
StorageTransferOptions transferOptions,
Expand Down Expand Up @@ -235,7 +240,9 @@ public PartitionedUploader(
}
else
{
_maxWorkerCount = Constants.Blob.Block.DefaultConcurrentTransfersCount;
_maxWorkerCount = CompatSwitches.UseLegacyDefaultConcurrency
? Constants.Blob.Block.LegacyDefaultConcurrentTransfersCount
: DefaultConcurrentTransfersCount;
}

// Set _singleUploadThreshold
Expand Down
Loading