Choosing a Message Queue: SQS, RabbitMQ, Kafka, or Redis Streams

For a client sending transactional emails and push notifications, we used SQS and would again without hesitation. It's fully managed, we've never had to think about broker capacity, and the semantics — at-least-once delivery, visibility timeouts, dead-letter queues — cover this use case completely. The trade-off is that SQS doesn't do pub/sub natively; fanning a single event to multiple consumers meant pairing it with SNS, which is an extra piece but a well-understood one on AWS.

For a client running background workflows with complex routing — different message types needing to reach different combinations of workers based on content, not just a single queue name — we used RabbitMQ. Its exchange and binding model made that routing logic explicit and configurable rather than something we'd have hand-rolled in application code. The cost is that we own the broker: patching, clustering for HA, and monitoring queue depth ourselves.

Kafka came up once, for a client building a clickstream analytics pipeline processing several million events per day where consumers needed to replay historical events and multiple independent services needed to read the same stream at their own pace. That's Kafka's actual strength — log retention and independent consumer offsets — and it was worth the operational overhead of running (or in this case, paying for Confluent Cloud to manage) a Kafka cluster specifically because replay was a hard requirement, not a nice-to-have.

Redis Streams got used for a much smaller case: a client already running Redis for caching needed simple job queuing without wanting to introduce an entirely new piece of infrastructure for a modest volume of background tasks. It's not as durable or feature-rich as the others, but for a team of two engineers who didn't want to operate another system, 'good enough and something we already run' won over 'technically superior and new.'

The question we ask first isn't 'which is best' but 'do we need replay, do we need complex routing, and do we already operate a broker.' Needing replay points to Kafka. Needing routing flexibility points to RabbitMQ. Wanting zero operational burden on AWS points to SQS. Already running Redis for something else, at modest volume, points to Streams. Picking based on what's trending rather than these questions is how teams end up operating Kafka for a workload that never needed more than SQS.