Chapter III · Area 09 of 12

Runtime & operations

The Question for Our ShopWhat does the shop do if the payment provider has not responded after one second?

Here you will find building blocks for stable runtime, controlled releases and the diagnosis of faults.

10 Building blocks

Starting hereTimeout

Three questions for every ongoing service

  1. How does the shop limit waiting time and load?

    Call protection

    After one second without a reply the shop stops waiting; payment P-731 stays open, and a retry uses pay-B-1042.

  2. How does the team know how the system is doing?

    Observing

    One trace links cart, payment and the shipping query; the goal: order confirmations visible within two seconds.

  3. How are changes delivered safely?

    Release & recovery

    A new tax rule first gets a small share of traffic; a backed-up order index is restored in isolation and reconciled with the payment system.

This is how the building blocks hang together

Timeout, retry, circuit breaker, bulkhead and backpressure protect the service from the consequences of faults and overload. Observability shows their effect; SLOs, rollouts and recovery connect measurement with operational decisions.

Quality attributeObservabilityObservability combines signals that let you inspect a system's internal state.MechanismTimeoutlimits the waiting time; signals show frequency and affected dependenciesFailure modeRetry Stormshows up in growing call load and retry metricsPatternCircuit Breakerlimits calls and makes its state visible through signalsPatternBulkheadseparates resources; utilisation per pool shows where bottlenecks ariseMechanismBackpressurethrottles the inflow; queue length and processing rate show the pressureApproachSLIs, SLOs & error budgetstranslate operational signals into metrics, targets and responses to deviationsDecision spaceDeployment & rolloutuses metrics to assess the effect of a rolloutActivityRecoveryis triggered by signals and closed with verified metrics
Relations: Timeout limits the waiting time; signals show frequency and affected dependencies; Retry Storm shows up in growing call load and retry metrics; Circuit Breaker limits calls and makes its state visible through signals; Bulkhead separates resources; utilisation per pool shows where bottlenecks arise; Backpressure throttles the inflow; queue length and processing rate show the pressure; SLIs, SLOs & error budgets translate operational signals into metrics, targets and responses to deviations; Deployment & rollout uses metrics to assess the effect of a rollout; Recovery is triggered by signals and closed with verified metrics.

Your learning path through the area

Here you clarify what happens when the payment provider has not responded after one second. From this you derive deadlines, protection mechanisms, measurement targets and safe rollouts. Each building block shows its theme at our shop.

  1. Timeout

    Protection

    This is what you learnYou can model a timeout as the end of the waiting period and keep an unexplained result technically visible.

    In the shopThe order module passes an authorization to the payment module that calls up the external provider. The shop time budget ends after one second; the provider response arrives later.

    Timeout Open
  2. Retry

    Protection

    This is what you learnYou can define a limited repetition with appropriate errors, waiting times, and cancellation conditions.

    In the shopThe shop loads the shopping cart again after a short connection error. It repeatedly limits and shows the stored cart status after the last attempt.

    Retry Open
  3. Retry Storm

    Protection

    This is what you learnYou can detect a repetition wave as a load amplification and contain it with coordinated limits and breaks.

    In the shopAfter a short interruption, shop clients and gateway send payment status queries simultaneously. A limited queue and random breaks distribute the re-traffic.

    Retry Storm Open
  4. Circuit Breaker

    Protection

    This is what you learnYou can configure a Circuit Breaker to protect a disturbed dependency from additional call pressure.

    In the shopThe shop calls the external payment provider. After a series of technical errors, the breaker opens for further authorization attempts.

    Circuit Breaker Open
  5. Bulkhead

    Protection

    This is what you learnYou can use separate resource quotas so that a slow shop function does not fully occupy other processing capacity.

    In the shopThe modular shop uses a worker pool for invoices and its own pool for order confirmations. Delayed invoices do not prove the order confirmation capacity.

    Bulkhead Open
  6. Backpressure

    Protection

    This is what you learnYou can link in-depth work to the actual processing capacity available.

    In the shopAfter a discount promotion, orders arrive faster than invoice creation processes them. The shop limits the queue and throttles new background orders.

    Backpressure Open
  7. Observability

    Measuring

    This is what you learnYou can combine metrics, logs and traces to match an operational question and derive a next test step from this.

    In the shopOrders take longer after a release. A trace links shopping cart, payment, and shipping query; metrics show the latency increase by version.

    Observability Open
  8. This is what you learnYou can define an SLI measurably, set an SLO for a time window, and connect an error budget with a reaction.

    In the shopThe team evaluates successful order confirmations that become visible in the shop within two seconds. A monthly goal and a response rule control risky changes.

    SLIs, SLOs & error budgets Open
  9. This is what you learnYou can plan a rollout with compatible versions, observable criteria, and clear return options.

    In the shopThe shop publishes a new tax rule initially for a small share of traffic. Old and new versions read the same additive database schema during the rollout.

    Deployment & rollout Open
  10. Recovery

    Changes

    This is what you learnYou can schedule a recovery process based on recovery goal, verified data, and business verification.

    In the shopAn error damages the order index. The team restores a secure stand in an isolated environment and compares orders with the payment system.

    Recovery Open
Afterwards in the learning path · area 10Security & risks