Scaling Vector Search Without Losing Relevance

Semantic search looks simple in a prototype: embed documents, store vectors, embed a query, and return the nearest neighbors. At production scale, each of those verbs hides an engineering decision that affects relevance, latency, cost, and operability.

The central mistake is treating a vector database as a smarter replacement for a conventional search index. It is not. Vector search is a probabilistic candidate-generation technique. Strong systems combine it with lexical retrieval, metadata constraints, reranking, and disciplined data operations.

Start with the retrieval contract

Before selecting a database, define what the search system must return and under which constraints. “Find similar content” is not a useful specification.

A retrieval contract should state:

• The searchable unit: product, paragraph, ticket, code symbol, image, or another entity.
• Required filters, such as tenant, language, region, entitlement, status, or date.
• The target recall and precision for representative query classes.
• Latency objectives at p50, p95, and p99.
• Freshness requirements for inserts, updates, and deletions.
• Expected corpus size, query volume, and growth rate.
• Whether exact ordering matters or approximate candidates are acceptable.

These requirements determine the architecture. A catalog search serving shoppers has different failure costs from duplicate-incident detection or source-code navigation. A system that must honor access rules before retrieval also needs different filtering guarantees from one searching public data.

Embeddings are part of the schema

Teams often treat the embedding model as a replaceable API dependency. In practice, it defines the geometry of the index and should be managed like a schema version.

Record the model identifier, model version, vector dimensions, normalization method, distance metric, input template, and creation timestamp with every vector. If any of these change, assume old and new vectors are incompatible unless testing proves otherwise.

Model selection should follow evaluation on your domain, not leaderboard position. Test real query-document pairs, including acronyms, identifiers, misspellings, multilingual terms, and hard negatives. A general model may understand prose while performing poorly on SKU codes or internal technical language.

Chunking is equally consequential. Small chunks improve localization but lose context and increase index size. Large chunks preserve context but dilute the signal. Prefer boundaries that reflect the domain: functions for code, sections for policies, issue-and-resolution pairs for support cases, and complete attribute sets for products.

Do not silently overwrite vectors during a migration. Create a new embedding version and index, run both against the same evaluation set and production shadow traffic, then switch through a controlled alias. Rollback should be an index-pointer change, not an emergency re-embedding job.

Approximate nearest neighbor indexes make explicit tradeoffs

Exact search compares a query with every vector. That becomes too expensive as collections grow, so most vector databases use approximate nearest neighbor, or ANN, indexes.

HNSW graph indexes generally provide strong recall and low query latency, but consume substantial memory and can be operationally awkward for high-churn workloads. Parameters controlling graph connectivity and search breadth trade memory, build time, and latency for recall.

IVF-style indexes partition vectors into clusters and search selected partitions. They can be more memory-efficient, but require representative training data and careful tuning of how many clusters each query probes. Product quantization compresses vectors further, reducing memory and sometimes improving cache behavior at the cost of distance accuracy.

There is no universally correct index. Benchmark with your vectors, filters, insertion pattern, and concurrency. Published queries-per-second numbers rarely transfer because vector dimensions, hardware, recall targets, and filter selectivity differ.

Measure recall against an exact-search baseline on a representative sample. An ANN configuration that is fast because it misses useful candidates is not an optimization.

Filtering changes the search problem

Metadata filters are not an implementation detail. They often define whether results are valid.

Consider a query restricted to one tenant and documents updated in the past month. If the engine first retrieves global nearest neighbors and then applies filters, it may discard every candidate even though relevant allowed records exist. Increasing the candidate count can reduce the symptom but raises latency and provides no guarantee.

Understand whether the database performs filtering before ANN traversal, during traversal, or after candidate generation. Test highly selective and skewed filters, not only common cases. Some engines handle low-cardinality categories well but degrade when each tenant represents a tiny portion of the index.

Security-sensitive constraints should be enforced by the query path and verified independently. Never rely on prompt instructions or downstream omission to hide unauthorized results.

Use hybrid retrieval and reranking

Embeddings are good at conceptual similarity but weaker on exact strings, rare names, version numbers, and negation. Lexical search has the opposite profile. Production search usually needs both.

Run vector and lexical retrieval in parallel, then combine ranked lists using a method such as reciprocal rank fusion. Avoid directly averaging raw scores unless they have been calibrated; cosine similarity and BM25 scores have different distributions and meanings.

For valuable queries, rerank the combined candidate set with a cross-encoder or another model that scores the query and candidate together. This adds compute, so reserve it for tens or hundreds of candidates rather than the full corpus.

A practical pipeline is:

1. Apply mandatory metadata constraints.
2. Generate candidates from ANN and lexical indexes.
3. Fuse the candidate lists.
4. Rerank the top candidates with a stronger model.
5. Apply deterministic business rules and diversity controls.
6. Return results with stable identifiers and score diagnostics.

This layered design is easier to tune than expecting one embedding model to solve every retrieval case.

Scale by isolating bottlenecks

Vector systems hit several limits: memory capacity, index-build throughput, write amplification, network transfer, and tail latency. Sharding is not automatically the first answer.

Estimate memory from vector dimensions, numeric precision, record count, and index overhead. A 1,536-dimensional float32 vector occupies roughly 6 KB before graph links, metadata, replicas, and database overhead. Hundreds of millions of vectors therefore require deliberate compression or partitioning.

Partition along stable routing boundaries when possible. Tenant, region, language, or content type can reduce fan-out if queries naturally target one partition. Pure hash sharding balances storage but requires every query to contact many shards, making p99 latency depend on the slowest participant.

Replicas improve read capacity and availability but multiply memory cost. Batch writes, asynchronous embedding generation, and immutable index segments can improve ingestion efficiency. Separate the operational objectives for fresh data and optimized historical indexes if one structure cannot satisfy both.

Operate vectors as derived data

The source record remains authoritative. Embeddings and ANN indexes should be reproducible derived assets.

Maintain an ingestion ledger with source version, chunk identifier, content hash, embedding version, index status, and deletion state. Make writes idempotent. Propagate deletions explicitly and audit that they reached every replica and index version.

Monitor more than uptime:

• End-to-end and per-stage latency percentiles.
• Candidate counts before and after filters.
• Empty-result and fallback rates.
• ANN recall against periodic exact-search samples.
• Score and query distribution drift.
• Index age, replication lag, and failed deletions.
• Cost per indexed item and per search.

Keep an offline relevance set segmented by query type, then supplement it with judged production samples. Aggregate metrics can hide serious regressions for exact identifiers, minority languages, or small tenants.

Takeaway

Treat embeddings as versioned schema, ANN as a measured approximation, and vector search as one stage in a hybrid ranking pipeline. Scale decisions should follow observed recall, filter behavior, memory use, and tail latency—not database feature lists.