vector-databases

# Vector Databases (Deep Workflow) Vector search is **approximate nearest neighbor (ANN)** at scale—**not** magic semantic understanding. Success requires **embedding model alignment**, **index parameters**, **metadata filters**, and **evaluation** against real queries. ## When to Offer This Workflow **Trigger conditions:** - Building RAG, similarity search, dedup, recommendations, anomaly clustering - Comparing **managed vector DB** vs **pgvector** vs **search engine kNN** - **Recall** issues, **stale** vectors, **slow** queries, or **cost** explosions **Initial offer:** Use **six stages**: (1) problem & metrics, (2) embeddings & schema, (3) index & parameters, (4) hybrid & filtering, (5) operations & cost, (6) evaluation & iteration. Confirm **scale** (vectors, QPS, dimension) and **latency SLO**. --- ## Stage 1: Problem & Metrics **Goal:** Define **what “similar” means** for the product—not only cosine similarity. ### Questions 1. **Query types**: short keyword vs long paragraph? multilingual? 2. **Precision vs recall** preference: legal/medical may need **high precision** 3. **Freshness**: how often do vectors change? **Real-time** upserts? 4 **Ground truth**: any labeled **relevant** pairs for eval? ### Metrics - **Recall@k**, **MRR**, **nDCG** when judgments exist; otherwise **human** spot checks + **proxy** tasks **Exit condition:** **Success metric** and **minimum acceptable** recall/latency stated. --- ## Stage 2: Embeddings & Schema **Goal:** **Stable embedding pipeline** with **versioning** and **metadata** design. ### Embeddings - **Model choice**: domain fit (code vs general text); **dimension**; **distance metric** (cosine vs dot vs L2)—**match** DB defaults - **Chunking** strategy upstream—bad chunks → bad retrieval regardless of DB ### Schema - **Payload/metadata** per vector: `doc_id`, `tenant_id`, `acl`, `source`, **timestamps** - **Multi-vector** per doc (passages) vs single centroid—**trade-offs** ### Versioning - **Re-embed** all on model change—**plan** downtime or **dual-write** period **Exit condition:** **ID strategy** + **metadata filter** needs documented. --- ## Stage 3: Index & Parameters **Goal:** Pick **index type** and **build params** for data size and recall. ### Common families (vendor-specific names) - **HNSW**: strong latency/recall; **memory** hungry; **tunable** `efConstruction`, `M` - **IVF**: better memory; needs **training** **nlist**; **probe** tuning - **PQ/OPQ**: compression—**recall** hit; good for **huge** scale ### Tuning loop - Start **defaults**; **sweep** parameters with **benchmark** queries - Watch **insert** throughput during **index build** on large backfills **Exit condition:** **Benchmark** results: p95 latency vs recall at fixed **k**. --- ## Stage 4: Hybrid Search & Filtering **Goal:** Combine **vector** similarity with **structured** constraints—most production needs this. ### Patterns - **Pre-filter** metadata (tenant, date) **before** ANN when supported—**verify** filter selectivity - **Hybrid**: BM25 + vector with **weighted** fusion or **rerank** stage - **Reranking**: cross-encoder on **top-k** candidates—**quality** boost, **latency** cost ### Pitfalls - **Filtering** that leaves **too few** candidates—empty results despite “similar” existing in other tenants **Exit condition:** **Query plan** documented: ANN → filter → rerank (as applicable). --- ## Stage 5: Operations & Cost **Goal:** **Reliable** ingestion, **monitoring**, and **predictable** bills. ### Ops - **Upsert** idempotency; **delete** tombstones for compliance - **Backups**, **multi-region** if needed—**eventual consistency** semantics per vendor - **Capacity**: memory per node vs **sharding**; **replication** factor ### Cost - **Managed** per **dimension × count**; **egress**; **query** units—**estimate** from **peak QPS** **Exit condition:** **Runbook** for **reindex**, **scaling**, and **incident** “search degraded.” --- ## Stage 6: Evaluation & Iteration **Goal:** **Continuous** improvement with **labeled** or **proxy** eval. ### Loop - **Golden** query set updated when product changes - **A/B** embedding models or **rerankers** with **guardrails** on latency - **Monitor** **click-through**, **thumbs**, or **human** grading in RAG ### Debugging bad retrieval - **Chunk** inspection, **metadata** leaks, **wrong** tenant filter, **stale** index --- ## Final Review Checklist - [ ] Metrics and embedding/model versioning plan - [ ] Index family chosen with benchmark evidence - [ ] Hybrid/filter strategy matches product needs - [ ] Ops: upsert, delete, scaling, backup understood - [ ] Eval set and iteration process in place ## Tips for Effective Guidance - **Never** promise “semantic search understands intent”—**ground** with eval. - **pgvector** vs **specialized**: trade-offs on **scale**, **ops**, **features**—state honestly. - Warn: **high-cardinality** filters + ANN can be **slow**—**design** metadata carefully. ## Handling Deviations - **Tiny corpus**: **brute force** or **simple** index may suffice—avoid over-engineering. - **Multimodal**: **separate** embedding spaces or **unified** model—**fusion** strategy required.