§02/SOVEREIGN AI
Sovereign AI
Routing regulated data through a third-party inference API is a structural risk posture decision, not a technical shortcut. We deploy private inference inside your perimeter — vLLM or Ollama on owned GPUs, private RAG pipelines, governed agents — so prompts, embeddings, and outputs never leave the cluster.
§01/THE INFERENCE PERIMETER
Why the API route is a risk posture decision
When an engineer routes a prompt to a third-party inference API, several things happen simultaneously: sensitive data leaves the perimeter; it transits infrastructure the organisation does not control; it is processed under terms-of-service that can change without notice; and the model version is not pinned, meaning the system's behaviour can shift without a deployment event. For consumer applications with low-sensitivity data, these risks may be acceptable. For regulated organisations — healthcare handling PHI, legal processing privileged communications, financial services with material non-public information — they are structural risk posture decisions that deserve the same scrutiny as any other data-handling choice.
The organisations we work with in regulated industries arrive at sovereign inference through different paths: a compliance requirement that explicitly restricts PHI to defined infrastructure, a legal opinion on privilege waiver risk in attorney-client communications, a board decision to treat model providers as a category of concentration risk. The technical implementation is the same in each case. What varies is the urgency — and the cost of the conversation that did not happen before the deployment went to production.
18–30 mo
TCO parity horizon (representative range)
API-based inference vs. owned GPU cluster, for sustained high-volume workloads — varies with token volume, hardware configuration, and vendor pricing trajectory
§02/INFERENCE INFRASTRUCTURE
Private inference on owned GPUs
We deploy inference stacks on Kubernetes with GPU device plugin scheduling: vLLM for high-throughput multi-GPU deployments, Ollama for simpler single-node inference with broader model support, HuggingFace TGI as an alternative where its specific model optimisations are relevant. Model selection is driven by your workload requirements — task performance, context window needs, and GPU memory available. We do not have preferred model vendors.
GPU allocation is governed by Kubernetes device plugins and resource quotas so inference workloads compete fairly with other cluster workloads and no single model deployment can exhaust available GPU capacity. Inference endpoints serve both chat-completion and embedding formats — the same cluster runs your LLM inference and your RAG embedding pipeline, with consistent authentication and audit logging. Open WebUI provides governed self-service access to these endpoints for internal users.
Model versioning is treated with the same rigour as application versioning: models are pinned, stored in a private registry, signed for provenance, and promoted through environments via GitOps. A model update is a deployment event — reviewable, auditable, and rollback-capable. This is the EVIDENCE-BY-DEFAULT pillar applied to AI: model governance is evidence infrastructure, not documentation overhead.
§03/PRIVATE RAG
Retrieval-augmented generation inside the perimeter
Retrieval-augmented generation addresses the model's knowledge cutoff and context window limitations by injecting relevant document chunks at inference time. The architecture has two stages — document ingestion and retrieval — each of which must stay inside the perimeter to maintain data sovereignty.
Document ingestion: source documents are chunked, embedded using an embedding model running inside the cluster (never a third-party embedding API), and stored in a vector database inside the cluster — pgvector on managed PostgreSQL, Qdrant, or Weaviate depending on workload size and query characteristics. Retrieval at inference time runs a semantic similarity search against the vector store and injects the top-k chunks into the prompt context. Latency is profiled for each deployment; sub-200ms retrieval is achievable at typical enterprise document corpus sizes.
We include an accuracy evaluation framework in every RAG deployment — a curated set of questions with known answers against which retrieval quality can be measured and tracked over time as the document corpus changes.
§04/AGENT GOVERNANCE
Governed agents with audit trail from day one
Agent frameworks — systems where an LLM drives a sequence of tool calls to complete a task — introduce a class of risk that single-turn inference does not: the model's decisions compound across steps, and an incorrect early decision can cascade into unexpected downstream actions. For regulated environments, this requires explicit governance design from the start, not a retrofit.
The governance controls we implement include: prompt and response logging with tamper-evident retention (every input and output persisted before the tool call executes); guardrails for PII detection and sensitive output classification; human-in-the-loop gates for tool calls above a defined risk threshold; retry and fallback logic that degrades gracefully to a defined safe state; and an incident classification framework for model-related failures distinct from infrastructure failures. Agent frameworks are scoped tightly — each agent has a defined tool surface and no access beyond what the task requires. This is the principle of least privilege applied to model capabilities.
§05/GPU ECONOMICS
The honest GPU TCO
GPU ownership economics are genuinely complex. H100 hardware costs are material; power and cooling in a data centre or colocation facility add ongoing cost; GPU memory limits the model classes you can run; and the staffing to manage the infrastructure is real overhead. We build a workload-specific TCO model before any hardware commitment: projected token volume, model class requirements, hardware options (owned, leased, or a hybrid with reserved GPU cloud capacity for burst), and the staffing cost of management.
The comparison point is not 'free API access' — it is the true projected cost of API-based inference at your expected token volume, including the vendor's pricing schedule and the trajectory of that pricing as demand grows. For teams with sustained, predictable inference workloads and a data sensitivity requirement, owned GPU infrastructure typically reaches TCO parity with API-based inference within 18–30 months. For teams with low-volume or highly variable workloads, a hybrid — private infrastructure for sensitive data, API-based inference for non-sensitive workloads — may be the better answer. We will tell you which is which.
Deliverables
Inference stack deployment
vLLM or Ollama on Kubernetes with GPU device plugin scheduling, model versioning, and automated health checks — serving both chat-completion and embedding endpoints.
Private RAG pipeline
Document ingestion, chunking, embedding, and retrieval using pgvector or Qdrant — with latency profiling, accuracy evaluation, and index maintenance automation.
Model governance framework
Policy document and tooling: model registry, version pins, evaluation harness, change control process, and incident classification for model-related failures.
Agent architecture design
Scoped, governed multi-agent frameworks with tool call logging, prompt/response retention for audit, PII guardrails, and retry/fallback logic.
Data residency documentation
Mapping of every data path through the AI stack — ingestion, embedding, inference, storage — with residency classification for regulatory review.
GPU TCO and capability model
Workload-specific comparison: owned GPU cluster vs. per-token API spend across projected token volume, including hardware, power, and staffing cost.
Stack
Trade-off
Model capability ceiling vs. data control — the trade-off that does not resolve cleanly
Self-hosted inference caps your accessible model classes at what runs on hardware you own or can lease. Frontier models — GPT-4-class, Gemini Ultra — are only accessible through their providers' APIs. That is a real capability ceiling, and we will not pretend otherwise. The performance gap between frontier API models and open-weight models like Llama 3.1 405B or Mixtral 8x22B is real for some tasks and negligible for others; it must be measured against your specific workloads, not assumed from general benchmarks. What is not negotiable for regulated organisations: if PHI, privileged legal communications, or material non-public financial data enters the prompt, the API route is an unacceptable risk posture. For workloads where data sensitivity is low and frontier capability is essential, API-based inference may be the right answer. We will tell you which is which.
Architecture
Sovereign AI — reference architecture
§06/FAQ
Questions, answered plainly.
The questions we hear most from CTOs, engineering directors, and founders considering a sovereignty engagement.
Which regulated industries is sovereign AI most relevant for?
Healthcare organisations handling PHI, legal services processing attorney-client communications, financial services with material non-public information, and government or defence entities with classification requirements are the most common cases. The commonality is not sector — it is the combination of data sensitivity and a regulatory or fiduciary obligation that defines how that data may be handled. If your organisation has a data classification policy that includes a 'restricted' or 'confidential' category, the question of where inference runs is already answered by that policy; the infrastructure needs to follow.
Can we run frontier models like GPT-4 or Claude on-prem?
Not in a meaningful sense. Frontier proprietary models are not released as weights — they are only accessible through their providers' APIs. Open-weight models in the 70B–405B parameter range are the current ceiling for self-hosted inference, and they perform well across a wide range of tasks. For specific workloads, we run a task-specific evaluation against both open-weight and frontier API models before recommending an approach. The capability gap is real in some domains (complex multi-step reasoning, frontier coding quality) and minimal in others (document classification, summarisation, structured extraction).
What GPU hardware do we need to get started?
It depends on the model class and throughput requirements. For 7B–13B parameter models at low to moderate concurrency, a single A100 or H100 40GB is sufficient. For 70B+ parameter models, a multi-GPU configuration is required — typically two to four H100 80GB GPUs for comfortable headroom. We model your specific requirements — model class, concurrent users, peak throughput — before recommending hardware. Leased GPU servers from colocation providers are a reasonable starting point before committing to hardware purchase.
How do you handle model updates and version pinning?
Model updates are treated as deployment events. Models are stored in a private registry, pinned to specific versions in the GitOps configuration, and promoted through environments with the same review and approval process as application code. A model update requires: a changelog entry noting the version change and the upstream release notes; evaluation against the task-specific test harness; a canary deployment to a subset of traffic; and a structured approval before full promotion. Rollback is a GitOps operation — reverting to the previous model version is the same as reverting any other infrastructure change.
What compliance documentation do we receive for the AI stack?
We deliver a data residency map covering every data path through the AI stack — document ingestion, embedding, inference, and storage — with residency classification for regulatory review. For HIPAA engagements, this includes a Business Associate Agreement analysis for each component. For SOC 2 engagements, each AI component is mapped to the relevant control objectives with evidence pointers. For GDPR, we document the lawful basis for processing, data minimisation controls, and the Subject Access Request response path for AI-processed data.
Can private RAG replace a commercial semantic search product?
For most enterprise document search use cases, yes — at significantly lower per-query cost and with full data sovereignty. The operational trade-off is real-time indexing latency (seconds to minutes vs. sub-second for optimised commercial products) and the operational responsibility for maintaining the vector index as your document corpus grows. For high-volume, real-time search over frequently-changing document sets, commercial products may retain an advantage. For compliance-driven search over internal document repositories where the update cadence is measured in hours, private RAG performs well. We profile both against your specific use case before recommending.
§00/The Five Pillars
Sovereignty
OWN-THE-PERIMETER
Engineering teams owning the full stack they depend on — no invisible landlords, no rented foundations.
Open source as method
OSS-AS-DISCIPLINE
Open source is a discipline of review, contribution, and independence — not a license type to tick on an audit form.
Audit-grade rigor
EVIDENCE-BY-DEFAULT
Security and compliance aren't retrofit — they are the bar that makes self-hosting safe in regulated, serious environments.
Optionality
EXIT-RAMPS-DESIGNED-IN
Anti-lock-in by architecture: every layer has a documented exit ramp so no vendor can hold you hostage.
The long game
BUILT-FOR-DECADES
Systems proportioned to outlast the tools, vendors, and leadership changes that will come in the decade after delivery.
Executive Briefing
Thirty minutes to clarify your infrastructure risk
Walk us through your vendor footprint and regulatory constraints. We will tell you honestly where sovereignty creates leverage — and where it does not. No pitch deck. No obligation.