[Feature Req] Eagerly signal session lock lost when underlying connection/link fault and signal ProcessSessionMessageEventArgs.CancellationToken

[hpendyalamsft]

Library name and version

Azure.Messaging.ServiceBus 7.18.4

Describe the bug

When using ServiceBusSessionProcessor to process messages from sessions and in the event of SessionLockLost due to ConnectionReset during processing a message M1,we are observing
a new connection is being created immediately and the M1 getting redelivered and processed by a separate invocation of ProcessMessageHandler.

The initial invocation of ProcessMessageHandler processing M1 is unaware of a ConnectionReset event.
Neither SessionLockProcessSessionMessageEventArgs.CancellationToken is getting canceled nor SessionLockLostAsync event is being raised.

This is leading to a situation where there is no way handle an occurence of event like this in our code and
2 instances of M1 are being concurrently processed.

Expected behavior

SessionLock lost due to ConnectionReset should be captured by either SessionLockProcessSessionMessageEventArgs.CancellationToken
or SessionLockLostAsync event and allow us handle this event appropriately in code.

Actual behavior

Neither SessionLockProcessSessionMessageEventArgs.CancellationToken is getting canceled nor SessionLockLostAsync event is being raised
when session-lock is lost due to ConnectionReset.
Any subsequent operation on the message (either settle the message, release session etc.) results in SessionLockLostException due to ConnectionReset.

Reproduction Steps

Attaching a simple project to repro the issue but defining the high level steps here:

1. Create a queue with sessions enabled and following settings

    MaxDeliveryCount = 10
    DefaultMessageTimeToLive = TimeSpan.FromMinutes(60)
    LockDuration = TimeSpan.FromSeconds(60)
    RequiresSession = true

2. Queue a message M1 with session S1
3. Create a ServiceBusSessionProcessor with following settings

    ReceiveMode = ServiceBusReceiveMode.PeekLock
    MaxConcurrentSessions = 5
    MaxConcurrentCallsPerSession = 1
    MaxAutoLockRenewalDuration = Timeout.InfiniteTimeSpan
    AutoCompleteMessages = false

4. Setup ProcessMessageHandler to honor cancellationToken and listen to SessionLockLostAsync events
5. Recieve M1
6. Force close connection using SysInternals-TcpView
7. Observation: M1 gets redelivered immediately and neither is cancellationToken being canceled nor SessionLockLostAsync being invoked.

SessionLockLostConnectionReset.zip

Environment

.NET SDK:
Version: 8.0.407
Commit: cac65f27eb
Workload version: 8.0.400-manifests.011c56c6
MSBuild version: 17.11.26+2b19be476

Runtime Environment:
OS Name: Windows
OS Version: 10.0.26100
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\8.0.407\

Host:
Version: 9.0.3
Architecture: x64
Commit: 831d23e561

.NET SDKs installed:
6.0.427 [C:\Program Files\dotnet\sdk]
8.0.407 [C:\Program Files\dotnet\sdk]
9.0.201 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Visual Studio : Version 17.13.4

[github-actions]

Hello @hpendyalamsft. I'm an AI assistant for the azure-sdk-for-net repository. I have some suggestions that you can try out while the team gets back to you.

• The issue appears to be that a connection reset isn’t triggering the cancellation token or the SessionLockLostAsync event for already started message processing. This behavior can lead to duplicate processing of the same message.

• As a workaround, consider wrapping your message operations in try/catch blocks to catch SessionLockLostExceptions so you can gracefully abandon any further operations on the message.

• You might also compare your processor settings – for example, using a finite MaxAutoLockRenewalDuration rather than an infinite one – to see if that change causes the expected cancellation behavior.

• Additionally, review the Service Bus troubleshooting guidelines (see ServiceBusException troubleshooting) for details on handling lock renewal issues.

• It might be helpful to log connection events, including any transient errors, to better handle unexpected redelivery. If this behavior persists, additional diagnostic logging or a targeted repro might be needed so we can investigate if a change is required in the SDK behavior.

The team will get back to you shortly, hopefully this helps in the meantime.

[jsquire]

Hi @hpendyalamsft. Thanks for reaching out and we regret that you're experiencing difficulties. Due to the design of the AMQP transport and the nature of transient network failures, the processor itself has no knowledge that a connection has been reset until such time that a network operation is requested. As a result, the processor is unaware that the session lock was lost until it attempts to perform an operation such as renewing the lock or reading additional messages. At that time, it does raise the associated events, including SessionLockLostAsync.

Unfortunately, without non-trivial updates to the AMQP transport, there is no reasonable way for the processor to receive notification that an AMQP connection or link has broken or been reset. If you'd like to make a feature request for the Mcirosoft.Azure.Amqp library, you're welcome to do so in their repository. We'd be happy to make use of it in the Azure SDK packages once available.

One additional and important note - Service Bus has an at-least-once delivery guarantee. Your application must be tolerant of duplicates and ensure idempotency in its processing. Even were this scenario detectable, there are still other scenarios where Service Bus may send duplicate messages or lock timing/expiration may cause a perceived overlap in processing.

[github-actions]

Hi @hpendyalamsft. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[bramley-jetcharge]

Hi there @hpendyalamsft ! I'm having a potentially related issue over in #49056. Were you able to determine the root cause of your AMQP connections being reset?