Encoding Formats: JSON, Protobuf, Avro, Thrift

How programs turn in-memory data into bytes, and how text formats like JSON compare with schema-based binary formats like Protocol Buffers, Thrift and Avro.

Overview

Inside a program, data lives in objects, lists and hash maps that point at each other in memory. To write that data to a file, send it over the network or put it on a queue, the program has to turn it into a flat sequence of bytes. This is encoding (also called serialization). The reverse, bytes back into objects, is decoding. Every service call, every message and every stored record goes through this step, so the format you choose affects size, speed, safety and, most of all, how easily the data can change later.

Two families of formats

Examples — JSON, XML, CSV — Protocol Buffers, Thrift, Avro
Human-readable — yes — no, needs a tool
Field names on the wire — in every record — no: numbers (Protobuf, Thrift) or nothing (Avro)
Types — loose (JSON has one number type) — declared in the schema
Schema — optional, separate (JSON Schema) — required, drives code generation
Typical use — public APIs, browsers, config — internal RPC, event streams, data files

The MessagePack format sits in between. It is a binary form of JSON: more compact types, but it still writes every field name in every record, and it has no schema.

The same record, four ways

The example record has three fields: user_id 12345, name Ada, active true. The script below encodes it by hand following each format's specification and prints the size of each.

JSON (no spaces) — 44 — field names, quotes, digits as text
MessagePack — 29 — field names, but compact numbers and lengths
Protocol Buffers — 10 — a 1-byte tag per field, values as varints
Avro — 8 — values only, in schema order

The exact numbers depend on the record, but the order is typical: names on the wire cost the most, and binary formats with a schema are several times smaller.

Protocol Buffers and Thrift: numbered fields

Protocol Buffers (Protobuf, from Google) and Thrift (started at Facebook, now Apache) work the same way. You write a schema that gives every field a field number (Thrift calls it a field id). The encoded bytes carry that number, never the field name. Code generated from the schema does the encoding and decoding.

In Protobuf, each field starts with a tag: the field number shifted left by 3 bits, combined with a 3-bit wire type that says how long the value is (varint, fixed 32 or 64 bits, or length-prefixed). Integers use varints: 7 bits of the number per byte, with the top bit saying whether another byte follows, so small numbers take one byte. Field numbers 1 to 15 fit in a one-byte tag.

Thrift has two main binary encodings. The BinaryProtocol writes a 1-byte type and a 2-byte field id before each field, and integers at fixed width. The CompactProtocol packs the change in field id and the type into one byte when it can, and writes integers as zigzag varints, much like Protobuf.

Because the wire type tells a reader how long a value is, an old reader can skip a field number it does not know. That is what makes adding fields safe.

Avro: no names and no numbers

Apache Avro, from the Hadoop world, goes one step further. The encoded record is just the values, one after another, in the order the schema declares them. There are no tags. The reader can only make sense of the bytes if it knows the exact schema the writer used, the writer's schema.

Avro then compares the writer's schema with the reader's own schema (the reader's schema) and matches fields by name. A field the reader does not know is skipped; a field the reader expects but the writer did not write is filled from the reader's default value.

So where does the reader get the writer's schema?

Large file (Avro container file) — once, in the file header
Kafka messages with Confluent Schema Registry — a schema id in each message; the reader looks it up
Two services on one connection — agreed once when the connection opens

In Confluent's classic wire format, each message starts with a 0 byte (the format version) and a 4-byte schema id, so the overhead for Avro is 5 bytes. Protobuf messages also carry a short list of message indexes after the id, saying which message type in the .proto file was used. Avro schemas are written in JSON, and because nothing in the schema is numbered, a schema can be generated from, say, a database table's columns.

Where JSON falls short

JSON is the right default for public APIs, but it has real gaps. It has no binary type, so bytes are usually sent as Base64 text, which is about a third larger. And it has one number type. RFC 8259 notes that integers are only guaranteed to be read the same everywhere between -(2^53 - 1) and 2^53 - 1, because many parsers, including JavaScript's, store numbers as 64-bit floats.

The usual fix is to send large ids as strings. A schema-based format avoids the problem with a real 64-bit integer type.

Choosing a format

Public REST API, browsers — JSON — readable, universal, no generated code
Internal service calls — Protobuf with gRPC — compact, typed, generated clients and servers
Kafka events across many teams — Avro or Protobuf with a schema registry — registry enforces compatible changes
Big data files — Avro (rows) or Parquet (columns) — schema travels with the file
Quick binary replacement for JSON — MessagePack — smaller, no schema needed

gRPC uses Protocol Buffers by default: services and messages are declared in a .proto file, and protoc generates client stubs and server code for many languages.

What you will be able to do

Explain what encoding is and why every service call and message needs it.
Say why Protobuf and Thrift are small and how field numbers let old code skip new fields.
Explain Avro's writer's schema and reader's schema, and where the writer's schema comes from.
Name JSON's real weaknesses, including the 2^53 integer limit.
Pick a format for a public API, an internal RPC and an event stream, and justify it.

Key concepts

Encoding (serialization): Turning in-memory objects into a sequence of bytes for a file, the network or a queue. Decoding turns the bytes back into objects.
Field number (tag): The integer a Protobuf or Thrift schema gives each field. The encoded bytes carry this number instead of the field name.
Wire type: Three bits in a Protobuf tag that say how the value is laid out (varint, fixed width, length-prefixed), so a reader can skip unknown fields.
Varint: A variable-length integer: 7 bits of the value per byte, with the top bit marking that more bytes follow. Small numbers take one byte.
Writer's and reader's schema: In Avro, the schema used to encode the data and the schema the reading app expects. Avro resolves the differences by matching field names.
Schema registry: A service that stores versioned schemas and hands out ids. Producers put the id in each message; consumers use it to fetch the writer’s schema.
MessagePack: A binary encoding with the same data model as JSON. Smaller than JSON text, but field names are still in every record and there is no schema.

When to use it

Choosing the wire format for internal service-to-service calls (for example Protobuf with gRPC versus JSON over HTTP).
Designing a Kafka pipeline where producers and consumers are written by different teams and schemas will change.
Explaining why a public API still uses JSON even though binary formats are smaller.
Storing large datasets in files that must carry their own schema.

Interview questions

Why would you choose Protobuf over JSON for internal service calls?
How does Avro decode a record when the bytes contain no field names or tags?
How can an old Protobuf reader handle a message with a field it has never seen?
What are the real problems with JSON as a data format?
What does a schema registry add to a Kafka pipeline?