Competing Consumers
Competing consumers distribute queued jobs across a worker pool.
- LockAssigns a job temporarily to exactly one worker.
- AcknowledgementOccurs only after durable business effect.
- ScalingUses queue age, duration, and resource limits.
Context
Why this is important
The throughput gain is safe only when redelivery recognizes the same business effect.
The four invoice workers are hypothetical; currently seven modules run together.
Worker failures and broker reactions
The current shop creates invoices in a modular monolith with seven modules. Hypothetically, Order places invoice jobs on a queue read competitively by four workers. Order ID and invoice version form the idempotency key. The team terminates a worker after storing the PDF but before broker acknowledgement; after lock expiry, another worker takes over, detects the existing effect, and acknowledges without a second invoice. Queue age and duration control worker count.
This is how you read the picture
- Lock Longer than duration or renewable
- Acknowledgement Effect before ack
- Scaling Age rather than count
| Comparison | Failure point | Broker | Protection |
|---|---|---|---|
| Before effect | Lock expires | Job reappears | Safe restart |
| After effect, before ack | Job reappears | Effect exists | Idempotency key |
| After ack | Job completed | No regular redelivery | Audit links effect |
Benefits & limitations
Helps if:
- Independent background jobs have variable volume.
- Exactly one worker should process each job at a time.
- Ordering is sufficient per order or customer key.
Boundaries
- At-least-once delivery can redeliver the same job.
- Locks that are too short create concurrent duplicate work; overly long locks delay recovery.
- Global ordering conflicts with unrestricted parallelism.
In practice
- Start multiple invoice workers and distribute test jobs. Check that each message is claimed by only one worker at a time and throughput rises under load.
- Stop a worker after the business commit but before message acknowledgement. Verify redelivery and a single invoice effect through idempotency.
- Assess horizontal scaling against ordering of one customer’s jobs, poison messages, and dead-letter handling. Define what must remain serialized per key.
Read the detailed explanation
The broker locks a message for one worker. After successful effect the worker acknowledges; if the lock expires first, another worker may receive the same job.
More workers increase parallelism only for independent work. Shared resources, ordering, or uneven jobs limit throughput; queue age is therefore more informative than count alone.
Queue describes the work channel. Idempotency protects the business effect on redelivery after a crash.
Check your understanding
Test yourself in three stages
ReadingCheck in the picture
How does the pattern differ from pub/sub?
View the solution with justification
One worker processes the job; subscribers do not receive separate copies.
ApplyingTransferred to a new situation
How does the example prevent a second invoice?
View the solution with justification
Order ID and version detect the existing business effect.
EvaluateWeighing an alternative
Which scaling metric is especially useful?
View the solution with justification
The age of the oldest message together with processing time.
Explore further
These building blocks continue the thoughts of this page.
References
- Azure · Competing Consumers Pattern (opens in new tab)Primary documentation relevant to Competing Consumers.
- Enterprise Integration Patterns · Competing Consumers (opens in new tab)Primary documentation relevant to Competing Consumers.
- Apache Kafka · Consumers (opens in new tab)Primary documentation relevant to Competing Consumers.