Skip to content

Conversation

@AndriiLab
Copy link
Collaborator

Improvements to ITransport.AmazonSQS

  • Added ability to create topic if it is not found in the topics cache
  • Improved functions naming

Improvements to SQS policy generation

Problem 1
In the production, we have faced limitation of SQS policy - size 8192 b
Proposed solution
In previous PR #808 I reduced policy size by combining statements for SNS access into one. Here I propose to replace all ARNs with wildcard if the prefixes for SNS and SQS name matches.
For example, there is SQS name MyQueue and three related topics:

"Resource": "arn:aws:sqs:us-east-1:MyQueue-v1",
"Condition": {
    "ArnLike": {
        "aws:SourceArn": "arn:aws:sns:us-east-1:MyQueue-FirstTopic",
        "aws:SourceArn": "arn:aws:sns:us-east-1:MyQueue-SecondTopic",
        "aws:SourceArn": "arn:aws:sns:us-east-1:MyQueue-ThirdTopic",
        "aws:SourceArn": "arn:aws:sns:us-east-1:SecondQueue-FirstTopic",
        "aws:SourceArn": "arn:aws:sns:us-east-1:SecondQueue-SecondTopic",
   }
}

If SNS name MyQueue mathes with topic prefixes this may be compacted to

"Resource": "arn:aws:sqs:us-east-1:MyQueue-v1",
"Condition": {
    "ArnLike": {
        "aws:SourceArn": "arn:aws:sns:us-east-1:MyQueue-*",
        "aws:SourceArn": "arn:aws:sns:us-east-1:SecondQueue-FirstTopic",
        "aws:SourceArn": "arn:aws:sns:us-east-1:SecondQueue-SecondTopic",
   }
}

If the SNS name does not match (case with SecondQueue), topics would not be compacted.

Problem 2
Use of _snsClient.SubscribeQueueToTopicsAsync broke compactization of SQS policies.
Solution
After investigation of AWS SDK, it was found that SubscribeQueueToTopicsAsync internally calls regeneration of SQS policy, but it would not recognize correctly wildcards, thus adds explicitly ARN of each SNS topic. Since the policy is already generated in the function above, usage of SubscribeQueueToTopicsAsync was replaced with SubscribeAsync. This just ensures subscription to topic but does not regenerate policy.

@yang-xiaodong yang-xiaodong self-requested a review July 27, 2021 10:01
Copy link
Member

@yang-xiaodong yang-xiaodong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks

@yang-xiaodong yang-xiaodong merged commit 4aeca71 into dotnetcore:master Jul 28, 2021
@AndriiLab AndriiLab deleted the feature/aws-updates branch July 28, 2021 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants