Fast AWS SQS Redrive
Posted on 2022-07-28, by Mark Watson
Quickly posting messages to SQS can be challenging. The most naive approach would be to have a sequential loop posting messages in groups of ten (the max supported batch size). In order to submit messages faster, it becomes important to parallelize the approach. This article provides an overview of my attempts to do so.
In the best case we should be able to submit millions of messages in less than a minute. Depending on message size, this should not consume very much bandwidth, and the bottleneck will just be coordinating that many HTTP requests. Since AWS SQS is a distributed system, it can handle a very high amount of requests per second.
Thread Pool Approach
import boto3
if __name__ == "__main__":
pass