Principles of Serialization: Data TranslationPrinciples of Serialization: Data Translation

------

The Necessity of Data SerializationThe Necessity of Data Serialization

During frontend-backend interaction, data undergoes multiple "transformations" to travel from the server to the client.During frontend-backend interaction, data undergoes multiple "transformations" to travel from the server to the client.

Scenario 1: The data the frontend receives has "changed"Scenario 1: The data the frontend receives has "changed"

// Backend sends Date birth = new Date(1990, 5, 15) // Frontend receives { "birth": "1990-06-15T00:00:00Z" } // A string!

The frontend tries to use .getFullYear() and gets an error — because this isn't a Date object, it's a string.The frontend tries to use .getFullYear() and gets an error — because this isn't a Date object, it's a string.

Scenario 2: Chinese garbled textScenario 2: Chinese garbled text

// Expected { "name": "Zhang San" } // Actually received { "name": "å¼ ä¸" }

Character encoding issues cause Chinese characters to become garbled.Character encoding issues cause Chinese characters to become garbled.

Scenario 3: Performance bottleneckScenario 3: Performance bottleneck

// A response containing 10,000 product listings { "products": [ { "id": 1, "name": "...", "description": "...", ... }, // ... 9999 more ] } // Size: 5.2 MB, transfer time: 3.5 seconds

JSON format redundancy causes the data packet to be too large, seriously impacting performance.JSON format redundancy causes the data packet to be too large, seriously impacting performance.

------

Serialization is like "translation" — "translating" in-memory objects into a transmittable format, and the receiver "translates" them back.Serialization is like "translation" — "translating" in-memory objects into a transmittable format, and the receiver "translates" them back.

------

1. Overview of Serialization and Deserialization1. Overview of Serialization and Deserialization

Serialization is the process of converting objects into a transmittable format.Serialization is the process of converting objects into a transmittable format.

Deserialization is the process of converting a transmitted format back into objects.Deserialization is the process of converting a transmitted format back into objects.

1.1 Package Delivery Analogy1.1 Package Delivery Analogy

1.2 Motivation for needing Serialization1.2 Motivation for needing Serialization

------

2. Common Serialization Formats2. Common Serialization Formats

Try it out: Click the button below to observe the serialization process across different languages:Try it out: Click the button below to observe the serialization process across different languages:

2.1 JSON: The Most Universal2.1 JSON: The Most Universal

Advantages:Advantages:

• Good readability, easy debuggingGood readability, easy debugging
• Supported by all languagesSupported by all languages
• Browser native support (JSON.parse / JSON.stringify)Browser native support (JSON.parse / JSON.stringify)

Disadvantages:Disadvantages:

• Large size (lots of {} "" markup)Large size (lots of {} "" markup)
• Doesn't support rich data types (Date, Map, Set are converted to strings)Doesn't support rich data types (Date, Map, Set are converted to strings)

Use cases:Use cases:

• Public APIsPublic APIs
• Frontend-backend communicationFrontend-backend communication
• Configuration filesConfiguration files

2.2 XML: The Former Mainstream2.2 XML: The Former Mainstream

<?xml version="1.0" encoding="UTF-8"?> <user> <id>123</id> <name>Zhang San</name> <email>zhangsan@example.com</email> <age>28</age> </user>

Advantages:Advantages:

• Clear structure, supports commentsClear structure, supports comments
• Supports complex nested structuresSupports complex nested structures
• Has Schema validation (XSD)Has Schema validation (XSD)

Disadvantages:Disadvantages:

• Large size, slow parsingLarge size, slow parsing
• Tag redundancy ()Tag redundancy ()

Use cases:Use cases:

• Configuration files (Spring, MyBatis)Configuration files (Spring, MyBatis)
• SOAP protocolSOAP protocol
• Complex data exchangeComplex data exchange

2.3 Protobuf: The Most Efficient2.3 Protobuf: The Most Efficient

// user.proto syntax = "proto3"; message User { int32 id = 1; string name = 2; string email = 3; int32 age = 4; }

Advantages:Advantages:

• Small size (30-50% smaller than JSON)Small size (30-50% smaller than JSON)
• Fast speed (5-10x faster parsing)Fast speed (5-10x faster parsing)
• Backward compatible (adding fields doesn't affect old versions)Backward compatible (adding fields doesn't affect old versions)

Disadvantages:Disadvantages:

• Not human-readable (binary format)Not human-readable (binary format)
• Requires .proto file definitionRequires .proto file definition
• Doesn't support dynamic typesDoesn't support dynamic types

Use cases:Use cases:

• Microservice internal communicationMicroservice internal communication
• High-performance scenarios (gaming, real-time communication)High-performance scenarios (gaming, real-time communication)
• Mobile apps (saves bandwidth)Mobile apps (saves bandwidth)

2.4 MessagePack: Balancing Readability and Performance2.4 MessagePack: Balancing Readability and Performance

// MessagePack is a binary version of JSON // Same data, MessagePack is about 30% smaller than JSON

Advantages:Advantages:

• Smaller than JSON, faster than JSONSmaller than JSON, faster than JSON
• Maintains JSON's data modelMaintains JSON's data model
• Supports all JSON typesSupports all JSON types

Disadvantages:Disadvantages:

• Not human-readableNot human-readable
• Not as efficient as ProtobufNot as efficient as Protobuf

Use cases:Use cases:

• Need performance but don't want ProtobufNeed performance but don't want Protobuf
• Redis cachingRedis caching
• WebSocket messagesWebSocket messages

------

3. Serialization Methods by Language3. Serialization Methods by Language

------

4. Performance Comparison4. Performance Comparison

4.1 Size Comparison (using a user object as example)4.1 Size Comparison (using a user object as example)

4.2 Speed Comparison (serializing 10,000 times)4.2 Speed Comparison (serializing 10,000 times)

------

5. Common Issues5. Common Issues

5.1 Date Serialization Problem5.1 Date Serialization Problem

Problem: Date objects become strings after serializationProblem: Date objects become strings after serialization

// Before serialization const date = new Date('2024-01-01') // After serialization JSON.stringify(date) // "2024-01-01T00:00:00.000Z"

Solutions:Solutions:

// Option 1: Convert to timestamp { createdAt: date.getTime() } // 1704067200000 // Option 2: Convert to ISO string { createdAt: date.toISOString() } // "2024-01-01T00:00:00.000Z" // Option 3: Custom serialization JSON.stringify(obj, (key, value) => { if (value instanceof Date) { return { __type: 'Date', value: value.toISOString() } } return value })

5.2 Circular Reference Problem5.2 Circular Reference Problem

Problem: Circular references in objects cause errorsProblem: Circular references in objects cause errors

const obj = { name: 'test' } obj.self = obj JSON.stringify(obj) // TypeError: Converting circular structure to JSON

Solutions:Solutions:

// Option 1: Filter out circular references const seen = new WeakSet() JSON.stringify(obj, (key, value) => { if (typeof value === 'object' && value !== null) { if (seen.has(value)) return seen.add(value) } return value }) // Option 2: Use the flatted library import { parse, stringify } from 'flatted' stringify(obj) // Automatically handles circular references

5.3 Chinese Garbled Text Problem5.3 Chinese Garbled Text Problem

Problem: Chinese characters become garbled after serializationProblem: Chinese characters become garbled after serialization

Causes:Causes:

• Character encoding mismatch (UTF-8 vs GBK)Character encoding mismatch (UTF-8 vs GBK)
• BOM markersBOM markers

Solutions:Solutions:

# Python: Ensure UTF-8 import json json.dumps(data, ensure_ascii=False) # Don't escape Chinese characters

// Node.js: Set response header res.setHeader('Content-Type', 'application/json; charset=utf-8')

------

6. Practice: E-commerce System Serialization Solution6. Practice: E-commerce System Serialization Solution

6.1 Scenario Analysis6.1 Scenario Analysis

6.2 Code Examples6.2 Code Examples

// API response (JSON) app.get('/api/products/:id', async (req, res) => { const product = await db.getProduct(req.params.id) res.json({ code: 0, data: product }) }) // Microservice communication (Protobuf) // product.proto syntax = "proto3"; message Product { int32 id = 1; string name = 2; int32 price = 3; } // Server side const proto = require('./product.proto') const message = proto.Product.create(product) const buffer = proto.Product.encode(message).finish() // Client side const decoded = proto.Product.decode(buffer) // Redis cache (MessagePack) const msgpack = require('msgpack-lite') await redis.set( `product:${id}`, msgpack.encode(product) ) const cached = msgpack.decode(await redis.get(`product:${id}`))

------

7. Using AI to Help Choose a Serialization Solution7. Using AI to Help Choose a Serialization Solution

AI can help you choose the right serialization format based on your scenario.AI can help you choose the right serialization format based on your scenario.

7.1 Prompt Template7.1 Prompt Template

You are a senior system architect, proficient in data serialization technology. Please help me choose an appropriate serialization solution. ## Business Scenario [Describe your scenario, e.g.: e-commerce app, game backend, microservices, etc.] ## Technical Requirements [List constraints, e.g.: - Frontend-backend separation (Vue + Node.js) - High performance requirements (QPS > 10000) - Bandwidth sensitive (mobile, need to save data) - Cross-language needed (Java + Python + Go)] ## Data Characteristics [Describe data features, e.g.: - Large data volume (single response > 1MB) - Complex structure (deeply nested) - Contains dates, binary data, and other special types] ## Output Format Please output in the following format: 1. Recommended solution (JSON/Protobuf/MessagePack/XML) 2. Rationale 3. Performance comparison (estimated size and speed) 4. Code examples

7.2 Practical Example7.2 Practical Example

Input prompt:Input prompt:

You are a senior system architect. Our e-commerce app needs to optimize the product list API performance. ## Business Scenario - E-commerce app product list API - Returns 100 products per request - Each product contains: id, name, price, image, description, and 20 other fields - Currently using JSON, response size ~200KB, time ~800ms ## Technical Requirements - Frontend is Vue.js - Backend is Node.js + Go microservices - Need to balance performance and development efficiency ## Optimization Goal - Response time under 300ms - Data packet size reduced by 30%

AI output:AI output:

Recommended solution: Hybrid approach 1. **API layer**: Continue using JSON (frontend-friendly) 2. **Microservice communication**: Switch to Protobuf (performance improvement) 3. **Static data**: Enable gzip compression ### Rationale - JSON is most frontend-friendly, no extra parsing needed - Backend microservices using Protobuf can reduce data volume by 40% - gzip compression is very effective for text (JSON can be compressed by 70%) ### Performance Comparison | Solution | Size | Time | |------|------|------| | Current (JSON) | 200 KB | 800 ms | | + gzip | 60 KB | 350 ms | | + Protobuf | 50 KB | 280 ms | ### Code Examples [Specific implementation code...]

------

GlossaryGlossary