Software ArchitectureAtlas

Software Architecture Atlas

Gain an overview. Understand the connections.

Standalone conceptMechanismChapter II · Distribution · Building block 3 of 11

Failure Detection

Slow and failed components can produce similar remote signals.

By Michael Meierhoff · Meierhoff Systems

  1. SignalSignal: An observable clue triggers an evaluation.
  2. SuspicionSuspicion: A timeout raises suspicion, not proof.
  3. ResponseResponse: The response follows criticality and evidence.

Context

Why this is important

Detection provides a sound basis for a targeted response.

Detection provides a sound basis for a targeted response.

Shop example · Mapping

Failure Detection: inspect the decision

The shop example uses the existing shared database and seven modules. Any added service or deployment model is a hypothetical extension. During checkout, a worker sends a heartbeat every five seconds. After several missed reports, the controller marks it suspect and routes new work to healthy instances. A delayed heartbeat can result from network load while the worker is processing a payment. The in-flight job therefore remains queryable by its identifier; detection does not trigger a second charge. The team measures detection time and false alarms, then tunes the threshold to recovery objectives.

This is how you read the picture

  1. Signal Signal
  2. Suspicion Suspicion
  3. Response Response
Failure Detection: inspect the decision
ConceptMeaningReview question
SignalAn observable clue triggers an evaluation.What does a missing heartbeat mean for the worker?
SuspicionA timeout raises suspicion, not proof.Which measurement shows whether detection meets the recovery objective?
ResponseThe response follows criticality and evidence.When should the team change the threshold?
Heartbeats, aktive Health Checks und Zeitüberschreitungen liefern zeitgebundene Signale. Ihre Aussage hängt von Intervall, Schwelle, Netzwerkpfad und Prüfungsumfang ab. Eine Erkennung startet eine abgestimmte Reaktion; sie beweist keinen Prozessabsturz. Use the questions in the second column to test this scenario.

Benefits & limitations

Helps if:

  • Detection provides a sound basis for a targeted response.
  • Define heartbeat timing, threshold, and follow-up action together.

Boundaries

  • One missing signal cannot distinguish a network fault from a process crash.
  • Probe intervals trade faster response for a higher risk of false alarms.

In practice

  1. Set the heartbeat interval and suspicion threshold from the recovery objective, and record which follow-up action begins when a worker is marked suspect.
  2. Delay heartbeats during a payment deliberately and compare false suspicions with actual process termination; ensure the same order identifier cannot trigger a second payment.
  3. Measure detection time and false-positive rate under normal and elevated network load, and adjust the threshold only when its effect on recovery and duplicate processing is clear.
Read the detailed explanation

A component may be slow, isolated, or crashed; from a distance these states can look alike. Failure detection therefore separates observation, suspicion, and confirmed diagnosis.

Heartbeats, active health checks, and timeouts provide time-bound signals. Their meaning depends on interval, threshold, network path, and probe scope. Detection starts a coordinated response; it does not prove a process crash.

The shop example uses the existing shared database and seven modules. Any added service or deployment model is a hypothetical extension. During checkout, a worker sends a heartbeat every five seconds. After several missed reports, the controller marks it suspect and routes new work to healthy instances. A delayed heartbeat can result from network load while the worker is processing a payment. The in-flight job therefore remains queryable by its identifier; detection does not trigger a second charge. The team measures detection time and false alarms, then tunes the threshold to recovery objectives. heartbeat and load balancing help compare this decision with adjacent concerns.

Check your understanding

Test yourself in three stages

  1. ReadingCheck in the picture

    What does a missing heartbeat mean for the worker?

    View the solution with justification

    It marks the worker as suspect; network load and process failure remain possible causes.

  2. ApplyingTransferred to a new situation

    Which measurement shows whether detection meets the recovery objective?

    View the solution with justification

    Measure detection time and false-positive rate across representative faults.

  3. EvaluateWeighing an alternative

    When should the team change the threshold?

    View the solution with justification

    When measurements show response time or false positives miss the agreed objective.

Explore further

Find variants and links to standalone concepts here.

Heartbeat

A heartbeat shows that a process sent a signal recently; the threshold determines when that signal counts as late. It supports timely detection, while remaining sensitive to interval, network path, and process pauses.

Further building blocks

DistributionPartial failuresContext and comparisonOpening OperationsLoad BalancingConceptual connectionOpening

References

Continue on the learning path

Next in Distributed systems: Delivery guarantees.

11 Building blocks

Continue withDelivery guarantees