Challenges with the migration to the topic-per-event topology

[danielmarbach]

This discussion topic serves as a reference point to capture challenges with the migration to the topic-per-event topology. Please share with us the challenges you faced or rough edges you discovered during the migration.

There is also a discuss group entry you can use should you prefer this avenue.

For cases when specific data or insights of the project needs to be disclosed, feel free to raise a support request.

Additional Context

• Topology documentation
  
• Upgrade guide
• Migration sample

Additional ideas

• If needed, codify a live migration approach similar to Remove hierarchy migration sample docs.particular.net#6989

[dvdvorle]

A challenge we encountered while migrating to the new topology is that we can no longer manage all Azure resources using Terraform. This decouples topic creation from permission management and makes it harder to clean up temporary dev/PR environments.

Previously we created the bundle topic using Terraform, where we also assigned the appriopriate permissions (Azure Service Bus Data Sender role) to our endpoint specific managed identities. The NServiceBus installer only managed the subscription rules.

In the new topology, we can't manage the topic permissions via Terraform because we don't have any knowledge about the topic/events there. While the NServiceBus installer can create the event-specific topics, it doesn’t seem to assign the necessary permissions. As a result, our endpoints are currently unable to publish events.

We've explored a few mitigation options:

1. Assign Azure Service Bus Data Sender at the servicebus namespace level.
   • This violates the principle of least privilege, especially since we share the namespace with unrelated (non-NServiceBus) topics and queues.
   • It doesn’t address the housekeeping challenges in temporary environments.
2. Manually keep a list of topic events in sync in terraform
   • This is tedious and error-prone.
   • We prefer to decouple infrastructure definitions from application code
3. Create some tooling around NServiceBus to export a list of events we can import in terraform
   • This adds maintenance overhead
   • Ideally, NServiceBus could provide a manifest (e.g., JSON/YAML) describing the required topics/subscriptions/queues. Akin to the compile-time SqlScripts.

None of these are ideal, so this is definitly a 'rough edge' for us.

---

To help visualize our setup, here’s a simplified sketch of our solution:

Endpoints:
- Name: Endpoint-A
  Identity: MI-A
  Events:
  - Event-a
- Name: Endpoint-B
  Identity: MI-B
  Events:
  - Event-b

We have a company-wide Azure ServiceBus namespace. Endpoint-A is owned by Team A, Endpoint-B by Team B. Each has its own deployment pipeline. During deployment we run the installers under a more privileged identity (MI-installer-x). At runtime each endpoint has its own least-privilege identity.

Old topology

Topics:
- Name: Bundle1 # terraform managed
  Permissions:
  - MI-A: Send # terraform managed
  - MI-B: Send # terraform managed
  - MI-installer-A: Owner # inherited from namespace
  - MI-installer-B: Owner # inherited from namespace
  Subscriptions:
  - Endpoint-A # NServiceBus managed
  - Endpoint-B # NServiceBus managed
- Name: Unrelated-Non-NServiceBus-Topic
  Permissions:
  - MI-Unrelated: Send
-

New topology
Missing Send permissions

Topics:
- Name: Event-a # NServiceBus managed
  Permissions:
  - MI-installer-A: Owner # inherited from namespace
  Subscriptions:
  - Endpoint-B # NServiceBus managed
- Name: Event-b # NServiceBus managed
  Permissions:
  - MI-installer-B: Owner # inherited from namespace
  Subscriptions:
  - Endpoint-A # NServiceBus managed
- Name: Unrelated-Non-NServiceBus-Topic
  Permissions:
  - MI-Unrelated: Send

[danielmarbach]

@dvdvorle First of all, thank you very much for taking the time to engage with us and provide extensive context. One of our highest priorities is considering various perspectives to make the right set of tradeoffs. The more detailed insights we have about different use cases (ideally in advance, though we don’t always achieve optimal engagement levels), the better we can fine-tune our approach.

I'll do my best to support you in addressing these concerns, but please allow me a brief detour to provide additional background to ensure clarity and context.

Historically, Azure Service Bus topologies were designed as system-wide solutions, enabling various system components to exchange events. The previous forwarding topology used a single bundled entity for multiplexing many events, filtering them based on subscriptions and associated rules. Unfortunately, this method incurs significant costs—CPU and memory consumption increase dramatically, as our extensive testing and discussions with Microsoft have confirmed. This leads to namespace-level throttling, affecting all endpoints. While some customers might rarely encounter this issue, a substantial number are directly impacted by these previous design choices.

To address these limitations, we've developed a new topology designed for scalability and intended to guide users towards the "pit of success" from the start, eliminating multiplexing by default.

We recognize, however, that multi-tenancy scenarios and situations like yours necessitate careful consideration of premium namespace costs and achieving higher entity density. Many customers optimize resource use by co-locating multiple functionalities within a single namespace. The catch with the old topology is that it reaches a high CPU and memory usage threshold much sooner, requiring costly additional messaging units. The new topology allows for greater density before hitting resource limits but, as you rightly noted, introduces different tradeoffs: requiring different permission management or potentially requiring more entities and therefore approaching quota limits sooner.

This is precisely why we designed the new topology with flexibility in mind, enabling overrides for topic mapping on both publisher and subscriber sides. For high-density scenarios, you can adjust your topology to reduce the total number of events—even reverting to a single "bundle topic" if needed—while other deployments still benefit from the default design's advantages.

Additionally, I'd like to emphasize the flexibility of this override functionality in greater detail. This approach allows you to start with minimal entity creation, adjusting your strategy incrementally as your requirements evolve. It provides a practical balance between initial simplicity and future scalability.

One aspect we couldn’t adjust due to Azure Service Bus control-plane restrictions is permission management. The previous topology could manage rules with minimal rights during auto-subscription, but the new topology inherently requires manage rights for subscriptions when subscribing to events. From your responses, this doesn't appear to be an issue since you're managing subscriptions via deployment scripts anyway is my understanding correct?

Opting into multiplexing, as previously discussed, means you wouldn't need to maintain separate topic lists explicitly, allowing you to retain your current approach. Additionally, the transport includes an options layer for topology instantiation:

https://docs.particular.net/transports/azure-service-bus/configuration#entity-creation-topology-options https://docs.particular.net/samples/azure-service-bus-netstandard/options/

Since each endpoint is self-contained (aware of its publishing and subscribing events), you could use this options layer as a single source of truth, automatically informing your deployment scripts and reducing duplication. Or to rephrase if your endpoints ship the topic and subscription map as artifacts to the deployment pipeline that structured data can easily be consumed by your terraform script acting as the single source of truth.

To recap:

• If you prefer you can map all published and subscribed events to a single topic with the new topology as long as you are aware of the tradeoffs involved and manage the rules yourself with your deployment in terraform
• If you want to opt into multiple topics, which I would highly suggest because of the other benefits mentioned above, you can use the options layer as an artifact / single source of truth to automate that permissions assignment

The additional upside of this is that you get even tighter control and also visibility because everything that is not explicitly mapped will not have the necessary permissions at runtime to send or receive (via the subscriptions) and you can run in truly least privilege mode even with multiple topics because it becomes clear which managed identity is a data sender or data receiver on a specific topic.

Let me know how I can be of further assistance

[dvdvorle]

Many thanks for your well-thought-out and detailed response. Good to hear you are aware of multi-tenant scenarios, and the trade-offs your customers face. We were not aware that multiplexing was still possible if needed. That's great to hear!

I understand that using ASB comes with limitations where choices (including trade-offs) need to be made, and you want to support your customers in choosing their optimal configuration.

One aspect we couldn’t adjust due to Azure Service Bus control-plane restrictions is permission management. The previous topology could manage rules with minimal rights during auto-subscription, but the new topology inherently requires manage rights for subscriptions when subscribing to events.

Ah, I think I understand, Azure Service Bus Data Owner allows you to create subscriptions and manage rules, but not assign roles for other identities. You'd need the appropriate control plane permissions for that (eg Owner or RBAC admin role). Though this could be documented as an additional requirement, or even option, I think.

It would be nice if the minimum required permissions would be mentioned somewhere anyway, maybe here?

From your responses, this doesn't appear to be an issue since you're managing subscriptions via deployment scripts anyway is my understanding correct?

So far we let NServiceBus handle the subscriptions through running the installers with a slightly privileged identity. Permissions on the actual topic / queues are set via deployment scripting. Or schematically:

Terraform provisions:

• Identities
• Topic, incl. role assignments
• Queue, incl. role assignments
• Container app env
• Installer job, which provisions:
  • Subscription (rules) + forwarding

Since each endpoint is self-contained (aware of its publishing and subscribing events), you could use this options layer as a single source of truth, automatically informing your deployment scripts and reducing duplication. Or to rephrase if your endpoints ship the topic and subscription map as artifacts to the deployment pipeline that structured data can easily be consumed by your terraform script acting as the single source of truth.

Ah, this is the inverse of our third mitigation option. If I understand correctly this would require us to explicitly map all events in JSON, which could be used both by our terraform scripts and NServiceBus alike. Or is there a way to export a serialized options file that would contain the mapping that'd be generated using TopicTopology.Default?

This is definitely an additional option, and a better one than all of our earlier ones. Though I do think it could be more polished still, in that it still forces us to be explicit about every event, while I'd rather adhere to convention over configuration. Though this might be inexperience on my part, maybe being explicit is better here.

Thanks again, this discussion has given us a few directions we'll definitely explore further.

[danielmarbach]

Ah, I think I understand, Azure Service Bus Data Owner allows you to create subscriptions and manage rules, but not assign roles for other identities. You'd need the appropriate control plane permissions for that (eg Owner or RBAC admin role). Though this could be documented as an additional requirement, or even option, I think.

It would be nice if the minimum required permissions would be mentioned somewhere anyway, maybe here?

That is a good idea. Currently this change is somewhat buried in the migration / upgrade guide and it doesn't hurt lift it up.

If I understand correctly this would require us to explicitly map all events in JSON, which could be used both by our terraform scripts and NServiceBus alike. Or is there a way to export a serialized options file that would contain the mapping that'd be generated using TopicTopology.Default?

It doesn't necessarily require you to map all events in JSON but it requires you to map all events. When you call some of the registration APIs like topology.PublishTo will populate the options which you can than dump to JSON. Unfortunately as of today there is no overload that simply allows you to call topology.PublishTo without explicitly specifying a topic in case you'd want the default convention of the full name you would have to call the overload with the full name yourself.

Though I do think it could be more polished still, in that it still forces us to be explicit about every event, while I'd rather adhere to convention over configuration. Though this might be inexperience on my part, maybe being explicit is better here.

I think both ways are fine and it really depends on the requirements and/or taste. When we implemented the topology we have decided to not auto populated the convention derived names back to the options because it would have forced the public API surface of the options type to be concurrency safe.

Please keep your ideas / thoughts coming. They are tremendously helpful input for any future changes we might be making to the transport 🙇

[markphillips100]

@danielmarbach I'm attempting to get my head around how to migrate from the single topic to the topic-per-event topology. Nothing I have is production yet so free to change whatever I need to do.

Much like @dvdvorle I'm using IaC (via Crossplane compositions and gitops) to define the endpoint queues, the single bundle topic and endpoint forwarding rules, with the default rule disabled. For a given set of endpoints this configuration largely remains the same. Only if I add a new endpoint, or decide to merge some together, would it need to evolve.

For subscriptions, each endpoints app identity has minimal privilege but can make use of AutoSubscribe to register their filter rules. This is necessary because each endpoint itself is largely unaware of the handlers it hosts. It's merely a container for other logical components that encapsulate their event handlers. An endpoint's separation from other endpoints is more to serve scaling and performance needs than defining any boundary. The boundaries so to speak are supported by the logical components themselves.

So from this architecture point of view, the endpoint itself is the source of truth for configuration of subscriptions. If I want this to be preconfigured via IaC then a CI pipeline would need some information to achieve its goal of defining IaC manifests for the ASB setup. I can see it as being problematic to have to manually manage configuration at each endpoint in this situation, considering each endpoint is meant to not have knowledge of the event handlers it is running due to boundary separation.

As @dvdvorle suggested, ideally some build-time configuration generated by the transport would be useful here. Maybe something like how the NServiceBus.Persistence.Sql package generates sql scripts for sagas, or how dotnet makes use of designtime factories for dbcontexts and migrations.

[markphillips100]

@danielmarbach just to add a question to that. Does AutoSubscribe in this new topology create both the topic and the forwarding rule to the queue for the endpoint, and does it do this without need for installers being enabled? See in the interim I may have to accept the endpoint identities having a higher privilege on the namespace but only if it means not using installers.

[markphillips100]

So from a bit of testing the auto-subscribe creates the subscription (makes sense) and forwarding rule to the relevant endpoint queue. It does not create the topic for the event and consequently the process logs an exception indicating the specific event that cannot be subscribed to.

I could of course then proceed to create the relevant topic, restart the process and see which is the next event needing a topic, and so on till all are accounted for. Instead I've created an MSBuild task package to scan for all the events in all the projects referenced assemblies, using a regex for fullname or interface to identify an event. For those it finds, it then checks for those consumed in handlers, also including those consumed through package references. Basically everything AutoSubscribe would normally find.

Results are dumped to a json file which I can use to produce the required IaC manifests.

[hazel-bohon]

Hi, @markphillips100! Thank you for your feedback. We have received some other requests around script generation for IaC.

This issue appears to be relevant. #876

[SzymonPobiega]

@markphillips100 the Subscribe call (either manual via IEndpointInstance or via the AutoSubscribe feature) creates topic only when installers are enabled (see here).

If I understand correctly what you would like to achieve (as a stop-gap) is a mode in which you don't enable installers (queues, databases etc. are all created via IaC) but the Subscribe method does attempt to create topics. Is that correct?

[markphillips100]

@SzymonPobiega I'd prefer for build-time tooling to output the required topics so IaC can act on it, rather than escalate privileges for a process (pod in kubernetes) at runtime as would be necessary to create a topic.

I have this working using an MSBuild task (you are welcome to the code) albeit it does not take into account any APIs that would otherwise alter the actual topics subscribed to by an endpoint, i.e. DisableFor() etc. I've only ever used AutoSubscribe so never needed those APIs - codebase isn't live (ongoing battles for years to get there) so free to change however I want.

Of course something more robust would be needed to support non-AutoSubscribe based subscriptions to identify actual required topics, but this fulfils my immediate needs:

Instead I've created an MSBuild task package to scan for all the events in all the projects referenced assemblies, using a regex for fullname or interface to identify an event. For those it finds, it then checks for those consumed in handlers, also including those consumed through package references.