[SB] `The connection was inactive for more than the allowed 240000 milliseconds`

[nhtgl]

• Package Name: azure-servicebus
• Package Version: 7.11.1
• Operating System: macOS/ubuntu
• Python Version: 3.8, 3.9, 3.10

Describe the bug
After resolving #29961 we've updated servicebus to 7.11.1 and now we're not seeing END frame errors like we used to, but instead we're seeing a lot of Connection closed with error: [b'amqp:connection:forced', b"The connection was inactive for more than the allowed 240000 milliseconds and is closed by container 'de9c0655b1684a31b24a94d4a733ed5e_G4S2'.", None] errors. That definitely is coming from

azure-sdk-for-python/sdk/servicebus/azure-servicebus/azure/servicebus/_pyamqp/_connection.py

Lines 478 to 485 in ee85ddd

	if frame[0]:
	self._error = AMQPConnectionError(
	condition=frame[0][0], description=frame[0][1], info=frame[0][2]
	)
	_LOGGER.error(
	"Connection closed with error: %r", frame[0],
	extra=self._network_trace_params
	)

.

I'm not sure whether this should be considered an error message, but I don't really understand how this amqp protocol works.

I want to be clear about one thing that may have something to do with it: we create an instance of ServiceBusClient and use it outside of context manager, and the same for topic or queue senders. The reason for this approach is performance - we prefer connection being closed due to inactivity over manual connection closing (like with calling close or context manager). This seems reasonable to us, because other solution would be to initialize service bus worker thread that would monitor connection time and send messages asynchronously.

Most errors coming from 3rd party libraries I treat as something I should probably take care of or there is an issue I should resolve somehow. In this case, I don't think I could do anything more, so on one hand I would silence this logger, but other error may occur that I wouldn't know about.

I would be very grateful for your support 🙏

To Reproduce
Steps to reproduce the behavior:

1. Probably leaving a connection hanging with no traffic for more than 240000 milliseconds

Expected behavior
Dunno, maybe this shouldn't be considered error? Just a warning that service was disconnected would be sufficient.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
We talked with @kashifkhan in #29961.
#29961 (comment)
#29961 (comment)
#29961 (comment)
#29961 (comment)

[kashifkhan]

Thanks for opening a new issue @macieyng . Ill check with the team and update.

In the meantime is it normal in your set up to not receive messages from the queue ( or subscription ) for 30 seconds ? Largely this error can be ignored as the client should retry connecting to the service and resume properly. If its happening a lot and there is supposed to be a steady stream of messages, it can indicate some network issues.

[nhtgl]

Yes. This is this normal. We process orders from one time zone, so during night time there's little to no traffic.

Right now, this is something we observe on dev and test environments, where traffic is generated only by our QA engineers, but I expect this to appear on our production once this is released. On production I would assume that there might be no traffic for 15 to 30 minutes on certain queues during night time.

[kashifkhan]

Thank you for the update @macieyng . We will be working on this issue during this sprint and will update this issue with the resolution

[nhtgl]

I think that we won't be able to provide more feedback, because we observed a 2x slowdown with pyamqp and we decided to use uamqp for as long as it will be possible on all three of our environments.

Also I would like to add that recommended way of using context manager for creating service bus client and then new context for service bus sender also adds latency big time (up to 4x).

[kashifkhan]

@macieyng We will be updating the error messages to show up only during DEBUG and will be releasing it in the upcoming release.

I'm very interested in resolving the performance issues you are experiencing as we run performance tests nightly to make sure pyamqp is running well. Infact for service bus ( and event hub before it) the goal was to have pyamqp be at par if not faster than uamqp, and overall pyamqp was much faster.

Our goal is pyamqp for Event Hubs and Service Bus and usage of uaqmp only as an emergency mitigation. If you are willing, I can share my email address and you can send me details about your environment, set up etc. so we can try to reproduce the issue on our end.