Queues, Pub/Sub & Streams
Queues distribute work, pub/sub distributes a message to multiple subscribers, and streams keep an orderly sequence readable. Delivery and storage rules characterize processing.
- QueueDistributes work to a worker.
- Pub/SubEvery subscriber receives the message.
- StreamOrdered sequence rereadable.
Context
Why this is important
Not every message needs the same distribution. Those who choose the form incorrectly lose work, reach too few recipients or cannot read again. Technical events deal with event-driven architecture.
This building block compares message transports. For business events and reactions, see Event-driven Architecture.
At a glance · One topic, many subscribers
Who gets the message?
The shop processes invoice orders, distributes orderPaid for shipping and analysis and keeps order status changes for later evaluation.
This is how you read the picture
- Queue In Queue, only invoice is green: a worker gets the task “create invoice”.
- Pub/Sub At Pub/Sub, shipping and analysis are green and receive orderPaid once each.
- Stream At Stream, shipping and analysis read the status changes from their own location.
- SendOrdersets up the task “Create invoice”
- ReceiptInvoiceA worker gets the job
- UninvolvedShippingreceives nothing
- UninvolvedAnalysisreceives nothing
One worker per orderConsignee
Benefits & limitations
Helps if:
- Queue, pub/sub and stream arrange messages according to the recipient group and desired storage.
- The appropriate messaging model lets each recipient resume from its own progress or replay the history it needs.
- A durably retained stream makes earlier events available to new consumers.
Boundaries
- The distribution model determines recipients and replay capabilities; each consumer must verify its guarantees.
- Transport acknowledgement and business processing are distinct states; Delivery Guarantees covers the transport.
- Delay and backlog remain consequences of temporal decoupling; see Asynchronous communication.
In practice
- Choose a Queue, Pub/Sub or Stream according to recipients, retention and replay needs. Record who owns the message and processing progress.
- A joint transport gives different business processing operations their own status. Save progress per consumer and link it to the delivery identifier.
- Test retention and catch-up behaviour under backlog. Asynchronous Communication explains temporal decoupling; Delivery Guarantees specifies redelivery.
Read the detailed explanation
Queues, Pub/Sub and Streams distribute messages with different rules for recipients, retention and replay. A queue distributes pending work; Pub/Sub gives independent subscriptions their own deliveries; a stream retains ordered entries for later reading.
The choice determines who stores progress, how long data remains available and whether new consumers can read the history. Delivery guarantees and business effects remain separate commitments.
Asynchronous Communication explains temporal decoupling; Delivery Guarantees specifies loss, redelivery and acknowledgement for the chosen transport.
Kafka is a widely used streaming platform. A Consumer Group shares a topic's messages among its members.
Check your understanding
Test yourself in three stages
ReadingCheck in the picture
Who receives orderPaid at Pub/Sub?
View the solution with justification
Dispatch and analysis, once each.
ApplyingTransferred to a new situation
The shop wants to evaluate order status changes later again. What form do you choose?
View the solution with justification
The stream. He considers the episode to be a fixed duration, and readers have positions of their own.
EvaluateWeighing an alternative
Should orderPaid run over a queue instead of pub/sub? How do you weigh up?
View the solution with justification
A queue distributes work to workers, each message goes to one. However, shipping and analysis should receive the event independently once, pub/sub provides.
Explore further
These optional deepenings complement the learning goal of this page.
Priority Queue and Competing Consumers
A Priority Queue can place urgent work ahead of normal work; Competing Consumers distribute pending messages across workers. Prioritisation serves urgent work sooner but may starve normal messages when urgent traffic remains high. Fairness can be provided through separate queues, reserved capacity or limits on consecutive priority messages. Workers may receive messages again, so processing must handle duplicates.
For testingTest priority and fairness. Divide shop tasks into urgent and normal work, simulate competing workers and a sustained stream of urgent messages. Define a fair scheduling rule and verify that redelivery of the same message does not duplicate its business effect.
Further building blocks
References
- Microsoft publisher subscriber (opens in new tab)Declares decoupled event distribution and delivery conditions.
- RabbitMQ · Queues (opens in new tab)Explains queue characteristics, consumers and news life cycle.
- Apache Kafka · Design (opens in new tab)Describes persistent logs, consumer and delivery semantics.
- Microsoft · Priority Queue (opens in new tab)Process messages by priority and take into account the operation of low priorities.
- Microsoft · Competing Consumers (opens in new tab)Distribute work among several recipients; take into account order and repetitions.