Partial failures
A partial failure occurs when one part of a distributed sequence fails and other parts continue to work. The caller often knows the final state only after a targeted clarification.
- Intermediate stateThe result is still unknown.
- Status checkThe provider calls the real state.
- Payment identification codeAllows for targeted demand.
Context
Why this is important
For distributed processes, a response may be missing, even though the provider has executed the action. Without its own intermediate state, the shop advises and risks double or lost payments. Repetition and waiting time deepen your own building blocks.
This building block deals with incomplete results across service boundaries. Retry describes the controlled repetition; Timeout limits the waiting time.
At a glance · No response, state unclear
Payment with unexplained result
The shop stores an order and calls up the payment provider. The connection breaks off after sending, while the order management continues.
This is how you read the picture
- Intermediate state The box “Clarification is running”: Answer remained out, result is unknown.
- Status check The arrows from “clarification runs” to “confirmed” and to “rejected”.
- Payment identification code The box “Payment requested”: order and payment identifier are stored.
Wide diagram · Move laterally if necessary
Read diagram as text
- Payment requested: Order and payment identification are stored.
- Clarification underway: Answer remained; result is unknown.
- Confirmed: Provider status confirms the payment.
- Rejected: Provider status confirms the rejection.
- Payment requested → Clarification underway: Response connection breaks
- Clarification underway → Confirmed: Status Request: Successful
- Clarification underway → Rejected: Request for status: rejected
Benefits & limitations
Helps if:
- A call can end without a response after a possible commit; an explicit unresolved state keeps the operation open until a status query resolves it.
- Services that fail independently need distinct handling for success, rejection and unresolved outcomes.
- Stable identifiers link status queries and resumption when another call could repeat a business effect.
Boundaries
- A Timeout does not determine the remote outcome; treat it as unresolved until a query with the same identifier confirms it.
- Resending can duplicate a payment; combine retries with Idempotency or a status query that resolves the outcome.
- The intermediate state needs a time limit and a rule for what happens afterwards; otherwise B-1042 remains pending indefinitely.
In practice
- Store unresolved outcomes separately from success and rejection. A stable operation identifier links status queries and resumption across process boundaries.
- Test failures before and after the commit. A timeout means no response arrived; it does not establish whether the remote system already committed its change.
- Retry with the same identifier and only once the effect is understood. Timeout defines the waiting limit; Circuit Breaker protects against persistent unavailability.
Read the detailed explanation
A partial failure occurs when participants in a distributed process operate independently and some connections or responses are unavailable. A timeout describes what the caller observes; the recipient may already have committed the change.
The outcome remains unknown until its status is resolved. A stable operation identifier links retries, queries and results. Systems should store and display “rejected”, “completed” and “unresolved” as distinct states.
Timeout defines the waiting limit; Circuit Breaker limits calls during persistent failures. Both leave a completed commit as a possible outcome.
A difficult-to-detect variant is Grey Failure: a component is still running but responds slowly or returns some errors, while health checks continue to report it as healthy.
Check your understanding
Test yourself in three stages
ReadingCheck in the picture
Which two outcomes lead out of “Clarification runs”?
View the solution with justification
To "Confirmed" and to "Rejected", respectively after the status query with the provider.
ApplyingTransferred to a new situation
The customer opens B-1042, while the status query is still unanswered. Can the shop show the order as paid?
View the solution with justification
No. The state "Confirmed" is only reached when the provider status proves the payment. Until then, “clarification is ongoing” remains visible.
EvaluateWeighing an alternative
The shop could immediately set “Rejected” if there is no answer. What speaks in favour, what speaks against?
View the solution with justification
The result is quick and simple. On the other hand: The provider may have booked the payment, and a new payment could charge twice. The status query avoids this guessing.
Explore further
These optional deepenings complement the learning goal of this page.
Heartbeat and failure detection
A missing heartbeat suggests failure but does not prove it: network delays, overload or a paused process can produce the same observation. Timestamps, missed signals and subsequent responses help interpret the detection. In the shop, the inventory service may keep working even though its heartbeat arrives late.
Initially mark the service as suspected of failure and preserve the operation state. A defined failure rule or fresh status response determines the next action; detection alone does not resolve a reservation already initiated.
For testingInterrupt the inventory service heartbeat in the shop while its processing continues. Check that your system initially marks the service as suspected of failure, avoids blindly sending a second reservation request and resolves the operation using its identifier and a fresh status response.
Further building blocks
References
- AWS – Timeouts, retries, and backoff with jitter (opens in new tab)Explains timeouts, limited repetitions, and jitter based on distributed failures.
- Google SRE – Handling Overload (opens in new tab)Explains how limited resources and overload increase failures.
- AWS – Challenges with distributed systems (opens in new tab)Organizes uncertainty, network failures and mutually independent error states.
- Unmesh Joshi · HeartBeat (opens in new tab)Periodic signs of life as the basis of a time-dependent failure detection.