Vector Databases Explained: Semantic Search

Search for "running shoes comfortable long distance" on a keyword-based system and you'll find products titled exactly that. You'll miss the product called "Marathon Training Footwear — Cushioned, Lightweight" — even though it's a perfect match.

The problem: keyword search matches words, not meaning. A vector database matches meaning.

This post explains how vector databases work, how data gets into them, and the search techniques available — without assuming you've worked with them before.

What Is a Vector Database?

A traditional database stores rows and finds them by exact value. WHERE category = 'shoes' either matches or it doesn't.

A vector database stores embeddings — lists of numbers that encode meaning. An embedding model (a neural network) reads your text and produces a float array like [0.12, -0.83, 0.41, ...] — typically 384 to 1536 numbers long. Similar meaning produces geometrically similar arrays.

When you search, your query gets embedded the same way. The database finds the stored vectors closest to your query vector — "closest" meaning most similar in meaning, not in exact wording.

This is why searching for "marathon footwear" surfaces products about running shoes even when neither word appears in the query.

Who Uses Vector Databases and Why

Semantic search — e-commerce, documentation search, job boards. Users search in natural language; results match intent rather than exact keywords. Amazon's current search engine uses this approach.

RAG (Retrieval-Augmented Generation) — AI assistants that need to answer questions from a large document set. The user's question is embedded, similar document chunks are retrieved, and those chunks are passed to a language model to generate an answer. This is how most enterprise AI chatbots are built.

Recommendation systems — products, articles, songs. A user's history is embedded into a profile vector; the database finds items with similar vectors. Spotify's "Discover Weekly" works this way.

Duplicate and fraud detection — find records semantically similar to a flagged entry. Two slightly differently worded support tickets may have identical meaning; a vector search surfaces them even when no keywords match.

Image and multimodal search — images can also be embedded. Search by photo, find similar products. Pinterest's visual search is built on this.

Core Concept: Embeddings

Before diving into ingestion and search, one concept to lock in: an embedding is a compressed representation of meaning as a list of numbers.

The same embedding model must be used at both ingestion time (when you store documents) and search time (when you embed the query). If you embed your documents with model A but query with model B, the numbers live in different spaces and the similarity scores are meaningless.

Semantic Space

“Marathon training footwear”

[0.12, -0.83, 0.41, 0.09, ...]

“Running shoes for long runs”

[0.11, -0.79, 0.44, 0.08, ...]

similar

“Car insurance quote”

[-0.62, 0.31, -0.18, 0.77, ...]

very different

The vector database's job is to answer "which of my stored vectors is most similar to this query vector?" — efficiently, across millions or billions of stored vectors.

Ingestion: Getting Data In

Ingestion has three stages. Each one has a decision that affects everything downstream.

Stage 1: Chunking

You rarely embed an entire document as one vector. A 10,000-word manual embedded as a single vector loses the detail from any specific section — every query returns the same result regardless of what you're actually asking about.

Chunking splits documents into smaller pieces before embedding. The tradeoff is simple:

Chunk size	Result
Too small (1–2 sentences)	Precise retrieval, but results lack surrounding context
Too large (whole pages)	Good context per result, but noisy similarity — everything matches a bit
Sweet spot (~256–512 tokens)	Good for most document and Q&A use cases

For short content like product descriptions or customer reviews, embedding the whole text as one chunk is usually right. For long documents like manuals, research papers, or support articles, chunk with a small overlap between chunks so context isn't cut off at boundaries.

Document

Running shoes need good cushioning. The heel drop matters for gait. Breathable mesh uppers reduce foot temperature during long runs...

Chunk 1

Chunk 2

overlap keeps context intact at chunk boundaries

Stage 2: Choosing an Embedding Model

The embedding model turns your text into a vector. It sets your accuracy ceiling, your dimension count (how long each vector is), and your cost.

Model	Dimensions	Best for
`all-MiniLM-L6-v2`	384	Fast, lightweight, good enough for most use cases
`bge-large-en-v1.5`	1024	Strong benchmark performance, runs locally
`text-embedding-3-small` (OpenAI)	1536	General purpose, API-hosted, pay-per-token
`bge-m3`	1024	Multilingual content

Higher dimensions generally mean better accuracy — but more storage, more RAM, and slower search. For most applications, 384 or 1024 dimensions is the practical sweet spot.

If you're prototyping, all-MiniLM-L6-v2 runs locally, is free, and is good enough to validate your idea. Switch to a stronger model when quality becomes the bottleneck.

Stage 3: Storing with Metadata

When you insert a document into a vector database, you store the vector alongside metadata — structured fields like category, price, date, or author. This matters at search time: you can filter by metadata while also doing semantic search.

json

{
  "vector": [0.12, -0.83, 0.41, "..."],
  "title": "Ultralight Trail Running Shoes",
  "category": "Footwear",
  "price": 89.99,
  "rating": 4.6
}

Design rule: add metadata fields you'll want to filter on. Filtering by unindexed metadata fields is slow.

Search: The Different Techniques

Semantic Search (Dense Search)

The standard approach. Embed the query, find the most similar vectors. Returns results that match meaning, not keywords.

Strength: Finds relevant results even when no keywords match. Weakness: Can miss results where the exact keyword matters — a user searching for "Python 3.12 release notes" wants that exact version, not semantically similar release notes for other versions.

Keyword Search (Sparse/BM25 Search)

The classic approach. Tokenizes the query, finds documents containing those tokens, ranks by term frequency. Most databases (Postgres full-text, Elasticsearch) use this.

Strength: Exact matches. Great for product codes, names, specific terms. Weakness: Misses synonyms, paraphrases, and anything not literally in the text.

Hybrid Search

Runs both semantic search and keyword search simultaneously, then fuses the two result lists into one ranked list using an algorithm called Reciprocal Rank Fusion (RRF).

Hybrid search is what Amazon, Shopify, and modern Elasticsearch all use in production. It captures exact-match precision from keyword search and semantic recall from dense search — the best of both.

Filtered Search

Combine vector similarity with metadata filters. "Find the most semantically similar products, but only within the Footwear category, under $100, rated above 4.0."

The filter runs before or alongside the vector search, narrowing the candidate pool. This keeps the result set relevant without doing a full brute-force scan.

Important: Add explicit indexes on the fields you filter by (category, price, date). Without them, filtering requires scanning all records — fast at 10,000 items, very slow at 10 million.

Reranking

A two-stage approach used when precision matters more than speed:

02
First pass: semantic search retrieves the top 50–100 candidates (fast, approximate)
04
Second pass: a more expensive model reads both the query and each candidate together and re-scores them (slow, accurate)

The second-pass model (called a cross-encoder) can consider the full relationship between query and document — it's much more accurate than the embedding similarity, but too slow to run over millions of documents. Running it over just 50 candidates is fast enough.

This is the architecture behind Amazon's A9/A10 search algorithm and most enterprise RAG pipelines.

How the Vector Index Works

The most expensive part of running a vector database is the ANN (Approximate Nearest Neighbour) search — finding the closest vectors to your query without scanning every single stored vector.

The index is the data structure that makes this fast. Different index types trade off between memory, build time, query speed, and recall (how often you get the true best match vs an approximate one).

Index	Memory	Query speed	Recall	Best for
FLAT	Low	Slowest (exact scan)	100%	< 100k vectors, testing
IVF_FLAT	Medium	Fast	~92–99%	Millions of vectors, predictable memory
HNSW	High (graph in RAM)	Fastest	~95–99.5%	Production: e-commerce, real-time search

HNSW is the right default for most production systems. It's a graph-based index — think of it as a map where closely related vectors have direct connections. Queries traverse the graph rather than scanning all vectors, getting to the right neighbourhood very quickly.

Two variants are worth knowing for edge cases: IVF_SQ8 compresses IVF_FLAT down to roughly a quarter of its memory at a small recall cost, and DISKANN keeps a similar graph on SSD instead of RAM for billion-scale datasets where memory is the hard constraint.

The recall dial: Every ANN index has a parameter that controls the speed-recall tradeoff. In HNSW it's called ef (search depth). Higher ef = check more nodes = better recall but slower. A typical production setting is ef=64 which gives ~98% recall at a few milliseconds per query.

The key tradeoff nobody warns you about: Recall is not binary. An ef=16 setting might return 9 out of the true top 10 results. Whether that 1 missed result matters depends entirely on your use case. For product search: almost never matters. For medical record retrieval: could be critical.

Similarity Metrics: Which One to Use

When you compare two vectors, you need a formula to measure "how close are these?"

Cosine similarity — measures the angle between vectors. Ignores how long the vector is, only cares about its direction. This is the right choice for text. Two sentences can be very different lengths but carry the same meaning — cosine similarity handles this correctly.

L2 (Euclidean distance) — measures the actual geometric distance between vector endpoints. Better for images and spatial data where the magnitude of the vector carries meaning.

Inner product (IP) — mathematically equivalent to cosine similarity when vectors are normalized to unit length. Slightly faster in practice. If you normalize your vectors at ingestion time (recommended), use IP.

For text: use cosine or IP with normalized vectors. Don't overthink it.

A Working Example

Here's a minimal semantic search over product descriptions using Milvus and a local embedding model.

python

from sentence_transformers import SentenceTransformer
from pymilvus import MilvusClient, DataType

model = SentenceTransformer("all-MiniLM-L6-v2")
client = MilvusClient(uri="http://localhost:19530")

# 1. Create a collection (like a table, but for vectors)
client.create_collection(
    collection_name="products",
    dimension=384,  # matches all-MiniLM-L6-v2 output size
)

# 2. Ingest: embed product descriptions and store with metadata
products = [
    {"id": 1, "text": "Cushioned marathon running shoes lightweight breathable"},
    {"id": 2, "text": "Waterproof hiking boots ankle support trail"},
    {"id": 3, "text": "Minimalist barefoot running shoes zero drop"},
    {"id": 4, "text": "Office leather loafers slip-on dress shoes"},
]

client.insert(
    collection_name="products",
    data=[
        {
            "id": p["id"],
            "vector": model.encode(p["text"]).tolist(),
            "text": p["text"],
        }
        for p in products
    ],
)

# 3. Search: embed the query, find similar vectors
query = "shoes for running a marathon"
query_vector = model.encode(query).tolist()

results = client.search(
    collection_name="products",
    data=[query_vector],
    limit=3,
    output_fields=["text"],
)

for hit in results[0]:
    print(f"Score: {hit['distance']:.3f} — {hit['entity']['text']}")

Output:

text

Score: 0.841 — Cushioned marathon running shoes lightweight breathable
Score: 0.712 — Minimalist barefoot running shoes zero drop
Score: 0.289 — Waterproof hiking boots ankle support trail

The barefoot shoe ranks second even though it doesn't contain the word "marathon" — it's semantically closer to a marathon query than hiking boots are. The loafer doesn't appear in the top 3 at all.

That's vector search working as intended.

The Tradeoffs That Matter in Practice

Chunk size vs. retrieval quality: Smaller chunks give more precise results but strip context. Larger chunks give more context but noisier similarity scores. There's no universal answer — test on your actual data.

Embedding model quality vs. cost: A stronger embedding model raises your accuracy ceiling but costs more per embedding call (if API-hosted) or requires more compute (if self-hosted). Start cheap; upgrade when you can measure the quality gap.

HNSW recall vs. memory: HNSW keeps the entire graph in RAM. At 10 million vectors with 1024 dimensions, that's roughly 40GB of RAM just for the index. If you're memory-constrained, IVF_SQ8 cuts this 4× at a small recall cost.

Hybrid search vs. pure semantic: Hybrid search is almost always better than pure semantic in production. The extra complexity (running two retrieval passes and fusing them) is worth it. Users type in real keywords and expect exact matches — pure semantic search alone occasionally misses obvious results.

Reranking vs. raw retrieval: Add a reranker only when precision in the top 5–10 results is critical. It adds 20–100ms of latency per query. For a real-time search box, that's noticeable. For a background RAG pipeline, it's fine.

Vector Databases Explained: Embeddings, Semantic Search, and Production Tradeoffs