Software ArchitectureAtlas

Software Architecture Atlas

Gain an overview. Understand the connections.

Failure modeChapter III · Operations · Building block 3 of 10

Retry Storm

A retry storm occurs when many clients repeat simultaneously in the event of a joint malfunction. The additional traffic can further overwhelm the burdened dependence.

By Michael Meierhoff · Meierhoff Systems

  1. Load amplificationEach level repeats, the load multiplies.
  2. LimitsFewer attempts per level brake the shaft.
  3. Jitter and QueueRandom pauses and a queue distribute the rest.

Context

Why this is important

In the event of a malfunction, many clients repeat at the same time, and the gateway repeats additionally. The burden grows exactly when the provider would have to recover. Retry and Backpressure deepen their own building blocks.

This building block handles coordinated retry load. Retry defines rules of a client; backpressure regulates inflow and queue capacity.

At a glance · Retries amplify load

Each layer retries failed calls, multiplying the load; a limit and jitter slow the wave.
Shop example · Playing through stages

Retry load: clients × attempts per level

After a short interruption, shop clients and gateway send payment status queries simultaneously. A limited queue and random breaks distribute the re-traffic.

This is how you read the picture

  1. Load amplification The load is clients × 3 × 3; 1.000 Clients are those 9.000 Questions.
  2. Limits With switch: only 2 Tests per level, gain decreases by x9 on ×4.
  3. Jitter and Queue With a switch, the load remains at a maximum of 1.500 Requests under capacity.
  • 1.800 RequestsLoad on the provider
  • ×9Reinforcement

200 × 3 × 3 = 1.800 Questions. The provider is almost busy; every further stage overloads him.

Values illustrative

Increase the clients and observe the load on the payment provider (capacity) 2.000 Questions. Turn on jitter, limit and queue and compare.

Benefits & limitations

Helps if:

  • Many similar callers can generate synchronised additional traffic after a shared failure.
  • Independent retry rules in several technical layers need a common load budget.

Boundaries

  • Unbounded backoff prolongs failures and resource occupation; bound attempts and total time per request.
  • Additional instances provide no relief if they load the same store or payment provider; assess the shared bottleneck.
  • Jitter spreads retries out but does not prevent overload; combine it with Backpressure and a Circuit Breaker.

In practice

  1. Count attempts end to end. Count original calls and retries at each participating layer and derive the maximum amplification. Record where another attempt is triggered for each operation. This gives an overall picture of additional load.
  2. Set a shared retry budget. Bound attempt count and total time, spread waits with jitter and define a retry-traffic rate. Check the limit against regular load and dependency capacity. This makes additional load predictable.
  3. Control and rehearse recovery. Define how accumulated work is released after a failure and which signals permit the next step. Simulate simultaneous reconnection under load. This shows whether the rules protect recovery or cause renewed overload.
Read the detailed explanation

A retry storm occurs when many callers retry within the same period or multiple layers multiply attempts. The additional load hits an already failing dependency and can prolong its outage. Bounded attempts, backoff and jitter reduce synchronisation; a shared budget complements individual Retry rules. A Circuit Breaker can temporarily reject subsequent calls during persistent failures.

After a failure, accumulated work can still cause a sudden surge during recovery. Load quotas, rate limits and gradual re-admission therefore belong in operational planning. Count retries across the entire request path, including client libraries and gateways. Autoscaling helps only when extra instances do not compete for the same exhausted bottleneck.

Thundering Herd describes the related case in which many waiting callers react to the same event simultaneously and overload a resource.

Check your understanding

Test yourself in three stages

  1. ReadingCheck in the picture

    What is the burden on the provider at 500 Clients without switches, and how when switched on?

    View the solution with justification

    Without switches 4.500 Requests ()500 × 3 × 3), over the capacity of 2.000. With a switch 1.500 Questions.

  2. ApplyingTransferred to a new situation

    After a short interruption, many clients ask for the payment status to B-1042 and other orders. How do you avoid the top?

    View the solution with justification

    Limit attempts over a common time budget and distribute them with jitter. A limited queue keeps the inflow to the provider within its capacity.

  3. EvaluateWeighing an alternative

    Only repeat the clients, the gateway no more. What speaks in favour, what speaks against?

    View the solution with justification

    For this: The reinforcement over two levels is eliminated. On the other hand, the gateway no longer intercepts short disturbances, and each client must comply with the agreed limits.

Explore further

These building blocks continue the thoughts of this page.

OperationsRetryRetry repeats a failed operation within defined limits and conditions.Opening OperationsBackpressureBackpressure adapts message inflow to available processing capacity.Opening OperationsCircuit BreakerA Circuit Breaker pauses calls to a failing dependency and protects other workflows.Opening

References

Continue on the learning path

Next in Runtime & operations: Circuit Breaker.

10 Building blocks

Continue withCircuit Breaker