Serialization and Protobuf
Level 2 · Data formats
How a program turns its data into text or bytes to save or send, and back. How JSON Schema checks the shape, and why Protobuf is smaller.
Course contentsLesson 4 of 11
Level 1 · Files and formats
Level 2 · Data formats
Level 3 · Config and docs files
Level 4 · Characters and escapes
Level 5 · IDs, dates and patterns
After this lesson you can
- Explain serialization in one sentence
- Read a small JSON Schema
- Say when Protobuf beats JSON
12 mincovers 3 glossary entries
Worth reading first: JSON, XML and CSV
How data leaves a program and comes back
Serialization turns the data in a program into text or bytes you can save or send. The other side reads it and builds the data back. That step is called deserialization. Emma's name and age take 24 bytes as JSON text, and just 8 bytes in Protobuf. Each saved game and each web request does this.
Inside a program, data is spread out in memory in a way only that program knows. A file or a network can only carry a line of bytes. So this step puts the data in a line, in a format both sides know.
JSON is the most common choice. People can read it, and each language can use it. In JavaScript, JSON.stringify does the packing and JSON.parse does the unpacking. Python has json.dumps and json.loads for the same two jobs.
Both sides must agree on the shape, not just the format. If one side sends age as text and the other wants a number, things break. JSON Schema is a way to write that shape down. It is itself written in JSON, and it lists each field, its type and which fields are required.
A tool can check any JSON against a schema and point to each mistake before the data is used. Code editors use schemas too, to warn you as you type a config file. The current version of JSON Schema is called 2020-12.
Protocol Buffers, or Protobuf, is a format from Google that packs data into compact bytes. You describe the data in a .proto file, giving each field a number. A tool then writes code for your language that packs and unpacks it. The bytes are small and fast to read, but people cannot read them.
Two mistakes are common. One is reading data from a stranger with no check. In some formats, that can even run an attacker's code. The other is changing a Protobuf field's number once it is in use. Then old messages land in the wrong field.
{
"type": "object",
"properties": {
"name": { "type": "string" },
"age": { "type": "integer", "minimum": 0 }
},
"required": ["name"]
}message Person {
string name = 1;
int32 age = 2;
}Worked example
Jack catches bad data early
Jack's shop gets 5,000 orders a day as JSON. Once a week, one arrives with the price as text, and the total comes out wrong.
He adds a JSON Schema that says price must be a number. Now the bad order is turned away at the door, with a message naming the field.
The words for this lesson3
Check yourself
What does serialization do?
Finished this lesson?
Marking it complete also marks its 3 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.