Ensiklopedia VibeKoding: Principles of Search Engines.Ensiklopedia VibeKoding: Principles of Search Engines.
You search for "red dress" on Taobao and find the most relevant results from billions of products in 0.1 seconds β how is this possible? Search engines are one of the internet's most critical infrastructure components. From Google to e-commerce site search, the core principles are the same: inverted index + relevance ranking.You search for "red dress" on Taobao and find the most relevant results from billions of products in 0.1 seconds β how is this possible? Search engines are one of the internet's most critical infrastructure components. From Google to e-commerce site search, the core principles are the same: inverted index + relevance ranking.
What will you learn in this article?What will you learn in this article?
After reading this chapter, you will gain:After reading this chapter, you will gain:
| Chapter | Content | Core Concepts |
|---|---|---|
| Chapter 1 | Inverted index | Forward index vs inverted index |
| Chapter 2 | Tokenization and analysis | Chinese word segmentation, stop words, stemming |
| Chapter 3 | Relevance ranking | TF-IDF, BM25 |
| Chapter 4 | Elasticsearch | Distributed architecture, shards, replicas |
| Chapter 5 | Search optimization | Synonyms, spell correction, autocomplete |
------
The essence of search is an information retrieval problem: given a query, find the most relevant results from a massive collection of documents and return them sorted by relevance.The essence of search is an information retrieval problem: given a query, find the most relevant results from a massive collection of documents and return them sorted by relevance.
This process has two phases:This process has two phases:
SELECT * FROM products WHERE name LIKE '%red dress%' might seem like it could work for search, but it requires a full table scan β checking each row one by one. When data reaches millions of records, this query becomes unusably slow. Inverted indexes turn this O(n) operation into an O(1) lookup.SELECT * FROM products WHERE name LIKE '%red dress%' might seem like it could work for search, but it requires a full table scan β checking each row one by one. When data reaches millions of records, this query becomes unusably slow. Inverted indexes turn this O(n) operation into an O(1) lookup.
------
Traditional databases use forward indexes: from document ID to document content. Search engines use inverted indexes: from keywords to the list of documents containing them.Traditional databases use forward indexes: from document ID to document content. Search engines use inverted indexes: from keywords to the list of documents containing them.
| Index Type | Direction | Lookup Method | Use Case |
|---|---|---|---|
| Forward index | Document β Content | Know the ID, look up content | Database primary key queries |
| Inverted index | Keyword β Document list | Know the keyword, look up documents | Full-text search |
1. Document collection: Gather all documents that need to be searchable 2. Tokenization: Split documents into individual terms 3. Build mapping: Record which documents each term appears in (along with position, frequency, etc.) 4. Persist storage: Write the index to disk for fast lookup1. Document collection: Gather all documents that need to be searchable 2. Tokenization: Split documents into individual terms 3. Build mapping: Record which documents each term appears in (along with position, frequency, etc.) 4. Persist storage: Write the index to disk for fast lookup
------
Tokenization is the first step in search engines and the biggest challenge for Chinese search. English naturally separates words with spaces, but Chinese has no delimiters β "δΉδΉηζεδΊ" could be segmented as "δΉδΉη/ζε/δΊ" or "δΉδΉ/ηζ/ε/δΊ".Tokenization is the first step in search engines and the biggest challenge for Chinese search. English naturally separates words with spaces, but Chinese has no delimiters β "δΉδΉηζεδΊ" could be segmented as "δΉδΉη/ζε/δΊ" or "δΉδΉ/ηζ/ε/δΊ".
| Tokenization Method | Description | Example |
|---|---|---|
| Standard tokenizer | Split by spaces and punctuation (English) | "hello world" β ["hello", "world"] |
| Chinese tokenizer | Segment based on dictionaries or models | "ζη΄’εΌζ" β ["ζη΄’", "εΌζ"] |
| N-gram | Sliding window of fixed length | "ζη΄’" β ["ζη΄’", "η΄’εΌ"] |
| Custom dictionary | Add business-specific terms | "iPhone16ProMax" as a single term |
Tokenization is just one step in text analysis. The complete pipeline includes: 1. Character filtering: Remove HTML tags, special characters 2. Tokenization: Split text into tokens 3. Stop word filtering: Remove meaningless high-frequency words like "η", "δΊ", "ζ―" 4. Synonym expansion: Expand "ζζΊ" (mobile phone) to "ζζΊγη΅θ―γ移ε¨η΅θ―" 5. Stemming: Reduce "running" to "run" (English)Tokenization is just one step in text analysis. The complete pipeline includes: 1. Character filtering: Remove HTML tags, special characters 2. Tokenization: Split text into tokens 3. Stop word filtering: Remove meaningless high-frequency words like "η", "δΊ", "ζ―" 4. Synonym expansion: Expand "ζζΊ" (mobile phone) to "ζζΊγη΅θ―γ移ε¨η΅θ―" 5. Stemming: Reduce "running" to "run" (English)
------
Finding matching documents is just the first step; more importantly, ranking β placing the most relevant results at the top.Finding matching documents is just the first step; more importantly, ranking β placing the most relevant results at the top.
| Algorithm | Principle | Characteristics |
|---|---|---|
| TF-IDF | Term Frequency (TF) Γ Inverse Document Frequency (IDF) | Classic algorithm, simple and effective |
| BM25 | Improved version of TF-IDF, adding document length normalization | Elasticsearch's default algorithm |
| Vector search | Convert documents and queries to vectors, compute cosine similarity | Supports semantic search |
- TF (Term Frequency): The more times a term appears in a document, the more likely the document is relevant to that term - IDF (Inverse Document Frequency): The fewer documents a term appears in, the higher its discriminative power - "η" appears in all documents (low IDF), so searching for "η" is meaningless - "Elasticsearch" appears in only a few documents (high IDF), so searching for it precisely locates relevant content- TF (Term Frequency): The more times a term appears in a document, the more likely the document is relevant to that term - IDF (Inverse Document Frequency): The fewer documents a term appears in, the higher its discriminative power - "η" appears in all documents (low IDF), so searching for "η" is meaningless - "Elasticsearch" appears in only a few documents (high IDF), so searching for it precisely locates relevant content
------
Elasticsearch is currently the most popular open-source search engine, built on Apache Lucene, providing distributed, RESTful API-based full-text search capabilities.Elasticsearch is currently the most popular open-source search engine, built on Apache Lucene, providing distributed, RESTful API-based full-text search capabilities.
| Concept | Description |
|---|---|
| Index | Similar to a database "table," storing documents of the same type |
| Document | A single record, in JSON format |
| Shard | A partition, splitting an index across multiple nodes |
| Replica | A copy, providing high availability and read scaling |
| Mapping | Field type definitions, similar to a database schema |
| Analyzer | Text analyzer, defining tokenization rules |
Elasticsearch is not meant to replace databases; it works alongside them as a search layer. Typical architecture: data is written to the database β synced to ES β search requests go to ES β detail requests go to the database.Elasticsearch is not meant to replace databases; it works alongside them as a search layer. Typical architecture: data is written to the database β synced to ES β search requests go to ES β detail requests go to the database.
------
| Optimization Method | Description | Effect |
|---|---|---|
| Synonyms | Searching "ζζΊ" (mobile) also finds "η΅θ―" (phone) | Improves recall |
| Spell correction | "iphoen" auto-corrected to "iphone" | Fault tolerance |
| Autocomplete | Typing "θΉ" suggests "θΉζζζΊ" (Apple phone) | Better UX |
| Highlighting | Matching words shown in red in search results | Visual clarity |
| Weight adjustment | Title match weight > content match weight | Improves precision |
| Filtering and aggregation | Filter by price range, brand | Narrow results |
------
Search engines are core infrastructure for internet applications. Understanding inverted indexes, tokenization, and relevance ranking β these three core concepts β means you've grasped the essence of search engines.Search engines are core infrastructure for internet applications. Understanding inverted indexes, tokenization, and relevance ranking β these three core concepts β means you've grasped the essence of search engines.
Key takeaways from this chapter:Key takeaways from this chapter: