On-Prem RAG: Qdrant vs pgvector for a Sovereign Retrieval Tier

Most teams that set out to keep their AI stack in-house get the generation tier right and leave the retrieval tier wide open. They stand up a self-hosted model server, harden the GPU nodes, write a policy that says no prompt leaves the network — and then wire the whole thing to a hosted vector service, where every document chunk they own is embedded by someone else's API, stored in someone else's tenancy, and queried through someone else's control plane. The prompts stayed home. The corpus did not.

This is the retrieval-augmented-generation blind spot, and it is structural rather than careless. Generation is the visible, expensive, obviously sensitive part; retrieval feels like plumbing. But retrieval is where the proprietary corpus lives — the contracts, the patient notes, the incident history, the source code, the pricing models. In a RAG system, the vector store holds a lossy but often reversible projection of your most sensitive text, plus the metadata payloads that make it queryable, plus a query log that reveals exactly what your organization is asking about. If sovereignty means anything, it means that tier is yours.

This article is the decision and deployment guide for that tier. It covers the three credible open-source candidates and how to choose between them, the embedding server most teams forget to bring in-house, the index tuning that decides whether your recall is real, the memory arithmetic that decides what it costs, and the exit ramp that keeps the decision reversible. It assumes you have already read our guide to running the generation tier on Kubernetes with vLLM — that post is about serving tokens; this one is about everything that happens before the first token.

Where Sovereignty Actually Leaks in a RAG Pipeline

Draw the data flow honestly and the leak points are obvious. A RAG pipeline has two paths that touch your corpus, and both of them are easy to accidentally route through a third party.

The ingest path reads source documents, splits them into chunks, converts each chunk into a vector, and writes the vector plus a metadata payload into a store. If the embedding step is an API call, every chunk of every document you have ever indexed has been transmitted to a vendor in plaintext. Not a summary. Not a hash. The text.

The query path takes a user question, embeds it with the same model, searches for nearest neighbours, filters by metadata, reranks the candidates, assembles a prompt, and calls the generation model. If the search step is a hosted vector service, your query embeddings and your filter predicates — which encode tenant IDs, document classifications, and date ranges — transit that vendor's control plane on every request.

The usual reassurance is that embeddings are not the original text. That is only partly true. Embedding inversion is an active research area, and the practical posture in a regulated environment should be that a vector derived from personal or confidential data inherits the classification of its source. Treat embeddings as regulated data and the architecture follows: the store lives on hardware you control, in a network you control, with the same access controls, encryption, backup, and retention obligations you already apply to the source system.

Click to zoom

The Three Credible Candidates

The vector database market is loud, but the set a serious platform team should actually consider is small. The filter: a permissive open-source licence, a real self-hosted deployment story rather than a crippled community edition, and an operational surface your team can carry at three in the morning.

pgvector — the store you probably already run

pgvector is a PostgreSQL extension, and that single sentence is most of its value proposition. Your vectors live in the same database as the rows they describe, under the same transaction, the same backup, the same point-in-time recovery, the same access control, and the same audit trail. There is no second system to secure, no dual-write consistency problem, and no separate disaster-recovery rehearsal. If you already run PostgreSQL on Kubernetes with CloudNativePG, adding a retrieval tier is an extension and a migration, not a new platform.

It is also genuinely capable. Version 0.7.0 added the sparsevec type — supporting indexing of up to 1,000 non-zero dimensions — and halfvec, a half-precision type that roughly halves storage and, importantly, allows indexing vectors above the 2,000-dimension limit that applies to the full-precision type. Version 0.8.0, released in October 2024, added iterative index scans, which fixed the single worst failure mode in filtered vector search (more on that below). Version 0.8.1 followed in September 2025 with PostgreSQL 18 support and a faster binary_quantize.

The honest limitation is memory. HNSW indexes in pgvector want to be resident; when the index no longer fits, latency degrades sharply rather than gracefully. The commonly cited practical ceiling is somewhere in the range of ten million vectors on a well-provisioned instance, and that number moves with dimensionality, so treat it as a prompt to benchmark rather than a hard constant.

Qdrant — the purpose-built store that behaves like infrastructure

Qdrant is a Rust vector database under Apache-2.0, and it is what you reach for when retrieval stops being a feature of your application and starts being a tier of your platform. It ships as a single binary or a container, runs CPU-based HNSW for all approximate search, and gives you the two things pgvector cannot: first-class payload indexes with filter-aware graph construction, and a quantization story that changes the cost model by an order of magnitude.

The filtering behaviour is the part that matters most in production. Qdrant builds payload indexes on the fields you filter by, and the HNSW graph is optimized for payload filtering only when the graph is built after the payload index exists. That ordering constraint is easy to get wrong and expensive to discover late: create your payload indexes before you ingest at volume, not after.

Weaviate — when hybrid search is the requirement

Weaviate is a Go vector database under the BSD-3-Clause licence, with Helm charts, native multi-tenancy, and built-in hybrid search that fuses BM25 keyword scoring with vector similarity. If your retrieval quality problem is that pure semantic search keeps missing exact identifiers — part numbers, error codes, statute references — hybrid search is not a nice-to-have, and Weaviate gives it to you without assembling it yourself. The trade is a larger operational surface and, in most published comparisons, lower raw throughput than Qdrant under heavy filtering, which is unsurprising given the language and architecture differences.

There is a fourth option worth naming for teams committed to PostgreSQL that are hitting the memory ceiling: disk-resident index extensions such as Timescale's pgvectorscale, which implements a StreamingDiskANN index under the PostgreSQL licence, and VectorChord. Both publish benchmarks showing large advantages over stock pgvector and, in Timescale's case, over Qdrant at high recall on a 50-million-vector corpus. Those are vendor-published numbers on vendor-chosen workloads — useful as evidence that the approach works, not as a substitute for benchmarking your own corpus.

Click to zoom

Bring the Embedding Model In-House Too

Self-hosting the vector store while calling a hosted embedding API is the most common half-measure in this space, and it defeats the purpose entirely. The fix is a self-hosted embedding server, and the default answer is Hugging Face's text-embeddings-inference (TEI): a purpose-built inference engine for embedding and reranking models with token-based dynamic batching, optimized transformer kernels via Flash Attention, Candle and cuBLASLt, small container images with fast boot times, OpenTelemetry tracing, and Prometheus metrics out of the box.

That last pair matters more than it sounds: an embedding server that exports Prometheus metrics and OpenTelemetry spans drops straight into the self-hosted observability stack you already run, instead of becoming an invisible dependency.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tei-embeddings
  namespace: retrieval
spec:
  replicas: 2
  selector:
    matchLabels: { app: tei-embeddings }
  template:
    metadata:
      labels: { app: tei-embeddings }
    spec:
      containers:
        - name: tei
          image: registry.internal.example/tei:1.8-cuda
          args:
            - "--model-id=/models/bge-m3"
            - "--max-batch-tokens=32768"
            - "--max-concurrent-requests=256"
            - "--auto-truncate"
          ports:
            - { name: http, containerPort: 80 }
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - { name: models, mountPath: /models, readOnly: true }
          readinessProbe:
            httpGet: { path: /health, port: http }
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: embedding-models

Two operational rules follow from this deployment. First, pin the model by digest and store it in your own registry or PVC — an ingest pipeline that pulls weights from the public internet at pod start is neither reproducible nor sovereign. Second, treat the embedding model as part of the index contract: query-time embeddings must come from exactly the same model and revision as ingest-time embeddings, because a model upgrade silently invalidates every vector you have stored. Record the model digest in the collection metadata so the mismatch is detectable rather than mysterious.

If you are sharing GPUs between the embedding server, the reranker, and the generation model, this is where MIG partitioning and time-slicing earn their keep. Embedding workloads are bursty and small; generation is steady and large. Giving the embedding server a dedicated MIG slice keeps a bulk re-index from starving interactive inference.

The Filtered-Search Trap, and How Each Store Escapes It

Almost every real RAG query is filtered: not the whole corpus, but this tenant's documents, from this date range, with this classification, that this user may see. Naive approximate-nearest-neighbour search plus a filter is where recall quietly dies.

The mechanism is worth understanding precisely, because it is counterintuitive. In a post-filtering design, the index returns its top candidates first and the filter is applied afterwards. With pgvector's default hnsw.ef_search of 40 and a predicate matching 10% of rows, roughly four rows survive the filter — so a query asking for the top ten returns four, and nobody gets an error. The system silently returns less context, the model answers with less grounding, and the failure surfaces as "the AI seems worse this month" rather than as an alert.

pgvector 0.8.0 addressed this with iterative index scans: the planner keeps scanning more of the index until enough rows survive the filter. Two modes exist — strict_order, which preserves exact distance ordering across iterations, and relaxed_order, which permits slight reordering in exchange for better recall. Bounds are set by hnsw.max_scan_tuples (default 20,000) and hnsw.scan_mem_multiplier (default 1), which cap how far a pathological query can run.

-- One store, one transaction boundary.
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunk (
  id           bigserial PRIMARY KEY,
  document_id  bigint      NOT NULL REFERENCES document(id) ON DELETE CASCADE,
  tenant_id    uuid        NOT NULL,
  classification text      NOT NULL,
  published_at date        NOT NULL,
  body         text        NOT NULL,
  embedding    halfvec(1024) NOT NULL   -- half precision: ~50% of the storage
);

-- Build the ANN index and the predicate indexes the filter will use.
CREATE INDEX ON chunk USING hnsw (embedding halfvec_cosine_ops)
  WITH (m = 16, ef_construction = 128);
CREATE INDEX ON chunk (tenant_id, classification, published_at);

-- Without iterative scans, this filter silently starves the result set.
SET hnsw.iterative_scan = relaxed_order;
SET hnsw.ef_search = 100;
SET hnsw.max_scan_tuples = 40000;

SELECT id, document_id, body
FROM chunk
WHERE tenant_id = $1
  AND classification = ANY($2)
  AND published_at >= $3
ORDER BY embedding <=> $4
LIMIT 10;

Qdrant approaches the same problem from the other end: rather than scanning further after the fact, it makes the graph itself filter-aware, using payload indexes built before the graph. The practical consequence is that Qdrant degrades far more gracefully as filters get selective, which is exactly the regime multi-tenant RAG lives in.

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant.retrieval.svc:6333")

client.create_collection(
    collection_name="corpus",
    vectors_config=models.VectorParams(
        size=1024,
        distance=models.Distance.COSINE,
        on_disk=True,                      # originals on disk...
    ),
    hnsw_config=models.HnswConfigDiff(m=32, ef_construct=256),
    quantization_config=models.BinaryQuantization(
        binary=models.BinaryQuantizationConfig(always_ram=True),  # ...index in RAM
    ),
)

# Payload indexes FIRST — the graph is only filter-optimized if these exist.
for field, schema in [
    ("tenant_id", models.PayloadSchemaType.KEYWORD),
    ("classification", models.PayloadSchemaType.KEYWORD),
    ("published_at", models.PayloadSchemaType.DATETIME),
]:
    client.create_payload_index("corpus", field_name=field, field_schema=schema)

hits = client.query_points(
    collection_name="corpus",
    query=query_vector,
    limit=10,
    search_params=models.SearchParams(hnsw_ef=256, exact=False),
    query_filter=models.Filter(must=[
        models.FieldCondition(key="tenant_id", match=models.MatchValue(value=tenant)),
    ]),
).points

On parameters: m controls graph connectivity and lands between 16 and 32 for production workloads; higher values raise recall and memory together. ef_construct governs build-time search breadth and trades index build time for quality. The query-time parameter — hnsw_ef in Qdrant, hnsw.ef_search in pgvector — is the one you tune in production, and the guidance for recall targets above 0.97 is to raise Qdrant's default of 100 to around 256. Measure recall against an exact-search baseline on your own corpus before you accept any of these numbers.

Memory Is the Cost Model

Vector search economics are RAM economics; work the arithmetic once and sizing becomes concrete. One million vectors at 1,536 dimensions in float32 is roughly 6.14 GB of raw vector storage, plus about 20–30% overhead for the HNSW graph itself — call it 8 GB for an unquantized million-vector collection.

Quantization changes that by an order of magnitude. Scalar quantization to int8 gives roughly 4× compression, taking the same million vectors to about 2 GB. Binary quantization gives roughly 32× compression, taking them to about 200 MB — which is why the standard production advice for Qdrant collections above about 100,000 vectors is to enable it. The pattern that makes this safe is oversampling with rescoring: search the compressed index for a wider candidate set, then rescore those candidates against the full-precision vectors held on disk. You pay one disk read per candidate and recover most of the lost precision.

The same arithmetic drives the storage layout. Keep the graph resident and the original vectors on disk; put those originals on fast local NVMe rather than networked storage, because rescoring is a random-read workload and network round-trips will dominate it. If your object tier is Ceph, SeaweedFS, or Garage, use it for source documents and snapshots, not for the hot rescore path.

What an Auditor Will Ask About Your Vector Store

A retrieval tier holding regulated data attracts the same questions as any other in-scope system. Five come up reliably, and most teams have not prepared answers for them.

• Deletion. When a data subject exercises erasure, what happens to the vectors derived from their records? In pgvector, a foreign key with ON DELETE CASCADE makes this automatic and provable in one transaction. In a separate vector store, it is an application-level responsibility you must implement, test, and evidence.
• Lineage. For any chunk returned in an answer, can you name the source document, its version, the chunker revision, and the embedding model digest? Store all four in the payload. Without them, an answer cannot be explained after the fact.
• Access control. Is the tenant filter enforced server-side by identity, or is it a predicate the application promises to add? The second is a prompt-injection and coding-error away from cross-tenant disclosure.
• Encryption and residency. Where do the volumes live, who can attach them, and are they encrypted at rest with keys you hold? For the strongest posture, confidential computing with remote attestation extends the guarantee to memory while the index is loaded.
• Recoverability. Can you restore the collection to a known point in time, and have you rehearsed it? A vector store rebuilt by re-embedding is a multi-hour GPU job, which makes it a real disaster-recovery scenario rather than an afterthought.

The deletion question is the one that most often decides the architecture. If your regulatory posture demands that erasure be atomic and demonstrable — as it does under most privacy regimes, and as it does for PHI under HIPAA — then keeping vectors in the same transactional store as the rows they describe is not a convenience. It is the control.

The Exit Ramp: Designing for the Store You Will Outgrow

You will pick wrong at least once. Corpora outgrow forecasts, filter requirements arrive from compliance rather than product, and the embedding model you standardized on will be superseded. The goal is therefore not to pick perfectly — it is to make the pick cheap to revise.

Three properties make a retrieval tier portable. First, the source of truth for chunks and their metadata is your own database or object store, never the vector index — the index is derived state that can be rebuilt from scratch. Second, retrieval sits behind a narrow internal interface (search(query, filters, k) -> chunks) so swapping the backing store is one implementation, not a refactor across every service. Third, the embedding model is self-hosted and versioned, so re-indexing is a job you schedule rather than a bill you negotiate.

With those three in place, migrating from pgvector to Qdrant is a batch re-upsert against a store you already have the vectors for, run in parallel with the old path, cut over behind the interface, and rolled back by flipping a flag. Without them, it is a quarter of work. This is the same interface discipline we apply to the model tier itself, where a self-hosted LiteLLM gateway keeps the choice of generation model behind one seam.

The Long Game

The retrieval tier will outlive the model tier by years. Generation models turn over every few months; the corpus, its chunking discipline, its metadata schema, and the lineage you attach to every chunk are institutional assets that compound. A team that owns its retrieval path can adopt a new model in an afternoon. A team that rented it is renegotiating a contract.

That is the whole argument, and it is not really about Qdrant or pgvector — both are good software under permissive licences. The durable decision is that the corpus, the embeddings derived from it, the index built over them, and the queries run against it all stay inside a boundary you control, because that is the layer where your organization's accumulated knowledge actually lives, and the cost of getting it back later only ever goes up.

Pick the store your team can operate. Bring the embedding model in-house on day one. Put the whole thing behind an interface you can swap. Then treat the vector store like what it is: a database holding your most sensitive text, subject to every control you already apply to the systems it was derived from.