Software ArchitectureAtlas

Software Architecture Atlas

Gain an overview. Understand the connections.

PatternChapter II · Styles & patterns · Building block 8 of 8

Pipe & Filter

Pipe & Filter processes data through a series of independent steps. Each filter takes inputs in a known format and delivers defined outputs.

By Michael Meierhoff · Meierhoff Systems

  1. FiltersOne step with exactly one task.
  2. Intermediate formatDefined input and output per filter.
  3. Plugged togetherInsert filters without changing others.

Context

Why this is important

A single big import step hides where an error arises. Pipe & Filter breaks down processing into filters with clear intermediate formats that can be combined, known from Unix pipes (cat import.csv | grep ... | sort) and ETL. For business flows with redemption across multiple transactions, see Saga.

This building block explains linear data processing. For business flows with redemption across multiple transactions, see Saga.

At a glance · Filter chain with pipes

Data flows from the source through filters 1 and 2 to the sink; pipes connect the steps, each filter has input and output.
Shop example · Line through the pipeline

Send a line through the pipeline

The catalogue imports a CSV file with new cups every morning. Invalid lines should be logged without falsifying valid product data.

This is how you read the picture

  1. Filters The stations read, check, normalize and catalogue, each with its own result.
  2. Intermediate format Under each filter, the format, such as raw data set or valid data set.
  3. Plugged together “+ Filter Remove Duplicates” is between checking and normalizing.

What is available after each filter

  1. Read moreRohdatensatz

    nr T-7, price ’24,95 EUR’

  2. Validategültiger Datensatz

    Mandatory fields and price available

  3. NormalizationProduktimport

    Price amount of money(s)2495 Cent, EUR

  4. Catalogueim Katalog

    T-7 Designed for

The line runs through all filters; only normalizing makes the text an amount of money in cents.

Lines illustrative

Select a CSV line and, if necessary, turn on the duplicate filter; the lines are illustrative.

Benefits & limitations

Helps if:

  • Individually testable filters give ordered transformations clear handovers.
  • Stable input and output formats allow processing steps to be reused or exchanged.
  • A separate failure path reveals why records failed while allowing the remaining processing to continue.

Boundaries

  • Many branches and loops exceed a linear chain's capabilities; use explicit process control, as described in Orchestration.
  • Partial state: earlier filters may already have written data when a later filter fails. Define resumption points.
  • Each handover has overhead; splitting extremely small steps may not be worthwhile.

In practice

  1. For each filter, define input, output, and error form. Stable intermediate formats allow targeted tests and show which processing step is responsible for a value.
  2. Limit filters to a transformation and keep common changeable state out of the chain. As a result, a faulty step can be checked and replaced in isolation.
  3. Define where failed records are stored and how processing resumes, then measure backlog at each stage. See Asynchronous Communication for temporal decoupling and Backpressure for load control.
Read the detailed explanation

A pipeline consists of filters with defined inputs and outputs. Each filter performs one processing step; a pipe passes its result to the next. The order and format of these handovers determine the flow.

Small filters with little state are easy to test and replace individually. Failure handling, resumption and side effects still need coordination across the pipeline. Many branches, loops or long-lived states call for explicit process control.

Asynchronous Communication decouples processing steps in time; Backpressure governs how a chain signals overload upstream.

Unix pipes are a familiar example: small programs connect through standard input and output. ETL pipelines and stream APIs such as Java Streams follow the same idea. Enterprise Integration Patterns describes the messaging form, with channels between filters.

Check your understanding

Test yourself in three stages

  1. ReadingCheck in the picture

    row 2 There is no price. Where does it end, and what happens to it?

    View the solution with justification

    Check in the filter. It is written with line number in the error log; normalization and catalog never see it.

  2. ApplyingTransferred to a new situation

    New rule: Prices under 1 EUR shall be rejected. What filter does it belong in?

    View the solution with justification

    In checks, because there is a decision about valid or invalid. Reading, normalizing and catalogue remain unchanged.

  3. EvaluateWeighing an alternative

    Should testing and normalization be combined into one step? How do you weigh up?

    View the solution with justification

    One step saves a handover. Separately, however, remains visible where an error arises, and normalization only processes what has been tested. Recommendation: Leave separate, as long as both steps have their own rules.

Explore further

These optional deepenings complement the learning goal of this page.

Blackboard

The Blackboard pattern provides a shared knowledge base to which specialised components contribute incrementally until a solution emerges. It suits tasks where available knowledge determines the processing order and several approaches work together. Unlike a Pipe & Filter chain, it does not prescribe a linear order; contributions, conflicts and completion criteria need explicit rules.

For testingCompare a linear discount calculation with a Blackboard for complex product recommendations in the shop. Several components could contribute product knowledge and refine a selection. Define which contributions each component may publish, how conflicting suggestions are handled and what makes the solution complete.

Further building blocks

PrinciplesSeparation of responsibilitiesSeparation of responsibilities assigns tasks with different reasons for change to suitable building blocks.Opening CommunicationAsynchronous communicationAsynchronous communication accepts work and enables it to be processed later.Opening DataSQL & NoSQLYou choose SQL or NoSQL persistence based on access patterns, consistency and operational requirements.Opening

References

Continue on the learning path

With that, you have Styles & patterns run through. It continues with Data & state.

8 Building blocks

Next areaData & state