feat(controller): optimize memory with queue when archiving is rate-limited. Fixes #13418

[ChenRussell]

Fix #13418

Motivation

When archiving workflows operations are severely rate-limited，there is a big memory usage increase in workflow controller

Modifications

To solve this problem, we can add a archived_wf_queue to archive the workflow asynchronously when listening the Add and Update events instead of archiving the workflow synchronously

Verification

run locally and tested

[agilgur5]

Thanks for root causing the memory increase and optimizing it!

I think we'd want to have a few people review this since it adds a new queue (with its own metrics, for instance) and a new flag.
There's also a consideration around naming those two, especially since there are potentially multiple different archive workers (e.g. GC'ing archived workflows a la archiveTTL is another one)

[agilgur5]

Makes sense to me. @Joibel could you double-check this when you're back from vacation? Specifically as this adds a new queue with metrics. Also if you have any thoughts on the naming

[ChenRussell]

Makes sense to me. @Joibel could you double-check this when you're back from vacation? Specifically as this adds a new queue with metrics. Also if you have any thoughts on the naming

@Joibel could you review this PR when you are free, thanks

[Joibel]

@Joibel could you review this PR when you are free, thanks

Yes, I'll take a look.

[ChenRussell]

@Joibel CI / Windows Unit Tests failed，could you rerun it or merge this PR without it, thanks

[ChenRussell]

@Joibel all checks have passed now, could you merge this PR? thanks

[Joibel]

Thank you for working through this.

[agilgur5]

@ChenRussell small reminder on the refactor discussed above

[ChenRussell]

Yes, I will finish it