Modes of Dataflow: Databases, Services, Messages

The three ways encoded data travels between processes (through a database, through service calls, through a message broker) and what each one demands from compatibility, coupling and failure handling.

Overview

Whenever one process sends data to another process that does not share its memory, the data is encoded into bytes and decoded on the other side. There are three common paths for those bytes: through a database, through service calls (REST or RPC), and through asynchronous messages via a broker. Each path puts different versions of code in contact with each other, so each one has its own compatibility rules, its own coupling and its own way of failing.

Three paths for the same data

Timing — reader may come much later — caller waits for the answer — receiver processes later
Both sides up at once? — no — yes — no, the broker buffers
Who must be compatible — every code version ever, with every record ever stored — clients and servers during rollouts — producers and consumers, for as long as messages are kept
Typical tools — PostgreSQL, MySQL, object storage — REST over HTTP, gRPC — Kafka, RabbitMQ, SQS

Through a database: sending data to the future

Writing to a database is a message to whoever reads it later, which may be your own code after the next deploy. Two things make this hard.

First, during a rolling deploy, old and new code read and write the same rows. New code may add a field that old code reads, changes and writes back without it, losing the field. Old code should therefore keep fields it does not understand.

Second, data outlives code. A deploy replaces all the code in minutes, but rows written years ago stay in their old shape unless someone rewrites them. Rewriting a large table is expensive, so most databases let old and new shapes coexist: PostgreSQL, for example, adds a column with a constant default by changing only metadata, and old rows get the default when read.

Old code rewrites new records — new fields silently dropped — keep unknown fields, or update only changed columns
Very old rows — readers meet shapes no one remembers — defaults for new fields, backward-compatible reads
Several services share one database — a schema change must be agreed by every service — one owning service per table, others use its API or events

Through service calls: REST and RPC

In a service call, a client sends a request over the network and waits for a response. REST is a style built on HTTP: resources named by URLs, standard methods, usually JSON bodies. RPC (remote procedure call) frameworks such as gRPC let the client call a method that looks local; gRPC generates the client stub from a .proto file and encodes messages with Protobuf by default.

Servers are usually upgraded before their clients, and some clients (mobile apps, partners) may never upgrade. So a service must accept old requests (backward compatibility for requests) and old clients must cope with new responses (forward compatibility for responses). Public APIs often keep several versions alive for this reason, named in the URL or in a header.

A remote call only looks like a local call. A 1994 Sun Microsystems paper, A Note on Distributed Computing, argued that hiding the difference is a mistake, because of latency, separate memory and partial failure.

Result — returns or throws — returns, fails, or times out with no answer
After a timeout — not possible — the request may or may not have run
Speed — nanoseconds, predictable — milliseconds, varies widely
Arguments — pointers to shared memory — everything encoded into bytes
Retrying — never needed — needed, and can run the action twice

The timeout row is the one to remember. A client that times out does not know whether the server did the work, so a blind retry can charge a card twice. Make such operations idempotent, for example by sending a unique request id that the server remembers. Also set a deadline on every call: gRPC sets none by default, so a client can otherwise wait forever.

Through messages: fire and forget

With asynchronous message passing, the sender hands a message to a message broker, and the broker delivers it to one or more receivers later. The sender does not wait for the receiver to finish, and it does not even need the receiver to be running.

Buffering — the receiver can be slow or down without losing messages
Fan-out — one event feeds many independent consumers
Redelivery — a consumer that crashes gets the message again
Decoupling — the sender does not need to know who the receivers are

Kafka, for example, keeps events for a configured retention period whether or not they have been read, keeps events with the same key in one partition, and lets every consumer of a partition read it in the order it was written. By default it guarantees at-least-once delivery: nothing is lost, but a message can arrive twice, for example when a consumer crashes after processing a message but before saving its position (its offset), and the message is delivered again on restart. Consumers must therefore be idempotent.

Messages carry the same compatibility demand as a database, because a topic may hold events written by older producers, and a consumer may replay from the start. A schema registry that checks each new schema is the usual guard.

Choosing a mode

The user is waiting for the answer (read a profile, check a price) — service call — the caller needs the result now
Payment authorization before confirming an order — service call, with an idempotency key — must know the outcome before going on
Several teams react to one business event — messages — fan-out, each consumer independent
The receiver may be slow or down — messages — the broker absorbs the gap
Data kept for later reads and reports — database, owned by one service — storage is the point; one owner controls its schema

Most real systems use all three. A common shape: a service writes its own database, answers its callers synchronously, and publishes events for everyone else.

What you will be able to do

Name the three modes of dataflow and say who must be compatible with whom in each.
Explain why data in a database outlives code and what that means for schema changes.
Explain why a network call is not a local call, and how timeouts, retries, idempotency and deadlines follow from that.
Explain what a message broker adds, and why its consumers must handle duplicates.
Choose a mode for a given interaction in a design interview and defend the choice.

Key concepts

Dataflow through a database: One process writes, another (or the same code later) reads. Every stored record must stay readable by every code version that may read it.
Synchronous service call: The client sends a request and waits for the response (REST, gRPC). Simple to reason about, but both sides must be up and fast at the same time.
Asynchronous message passing: The sender hands a message to a broker and moves on; receivers process it later. Decouples the two sides in time and availability.
Message broker: A service that stores messages and delivers them to receivers, buffering, fanning out and redelivering as needed.
Idempotent operation: An operation that has the same effect whether it runs once or several times. Needed wherever retries or redelivery can happen.
Deadline: The latest time a client is willing to wait for a remote call. Past it the call fails, instead of hanging.
At-least-once delivery: The broker never loses a message but may deliver it more than once. Kafka’s default guarantee.
Partial failure: Some parts of a distributed system fail while others keep working, so a caller can be left not knowing what happened.

When to use it

Choosing between a synchronous call (REST, gRPC) and an event on a broker (Kafka, SQS) for a step in a design.
Explaining why microservices should not share one database.
Planning a rollout where clients, servers, producers and consumers upgrade at different times.
Designing retries for calls that change state, such as payments.

Interview questions

When would you use Kafka instead of REST for service-to-service communication?
Why is it a problem for several microservices to share one database?
A client calls a payment service and the request times out. What should it do?
In each mode of dataflow, whose code must be compatible with whose?
How do you make a message consumer safe against duplicate deliveries?