A regulated healthcare SaaS

The company had built a clinical workflow SaaS used in regulated healthcare settings — outpatient and procedural care settings with strict PHI handling obligations. Their engineering team had identified AI-assisted clinical documentation as the highest-leverage feature they could ship: the documentation burden for their users was well-documented, the opportunity was large, and competitive pressure from well-funded AI-native entrants was building quarter by quarter. The first attempt used a third-party LLM API. The prototype was strong — inference quality was high, latency was acceptable, and the integration was straightforward. The legal review stopped it within two weeks. The problem was not the API provider's security posture or the quality of their Business Associate Agreement. The problem was structural: any PHI that transited a third-party API — even encrypted, even under a signed BAA — constituted a disclosure event under the specific interpretation of HIPAA obligations that several of the platform's largest enterprise clients held and had written into their contracts. Several of those clients sent formal written notices: PHI inference on external APIs would trigger contract review under the data handling provisions of their SaaS agreements. The product team faced a binary choice. They could accept the regulatory risk, betting that clients wouldn't detect the PHI transit or wouldn't act on it. They could restrict the AI feature to de-identified data only, which would gut its clinical utility. Or they could build an architecture where the inference happened entirely inside the organisation's controlled infrastructure boundary — making the third-party disclosure question structurally moot rather than legally managed.

The commercial stakes were direct. The accounts that had sent formal notices represented a material portion of ARR. Losing them would have set the financing round back significantly and damaged the company's reputation in a market segment where reference customers were the primary sales motion. But not shipping AI features ceded positioning to competitors who were either accepting the regulatory risk (betting their clients wouldn't notice) or operating in lower-acuity segments where the PHI interpretation was different and the constraint didn't apply.

The legal team's position was unambiguous: PHI inference must happen entirely inside the company's infrastructure boundary. Not "encrypted in transit to a trusted third party with a BAA" — inside the boundary, meaning the company's own hardware or infrastructure it controls and operates. That was the bar.

A secondary risk surfaced during the initial analysis that had not been part of the original problem statement: the company had no model governance process. If a model was updated by an external API vendor, the clinical behaviour of the inference output could change without a release process, without a rollback plan, and without a documented approval chain. A clinician-facing feature whose behaviour could change silently on a vendor's schedule was not a posture any compliance officer at a regulated healthcare organisation would approve, regardless of the PHI question. Model governance — the ability to say "this version of this model, approved on this date, by these reviewers, with these known evaluation results, deployed via this change process" — became a first-class requirement alongside the PHI boundary requirement.

There was also a longer-term competitive consideration the product leadership articulated clearly: a proprietary inference infrastructure that the company owned and operated was a durable competitive asset. Inference quality from open-weight models was improving rapidly; a company that had built the capability to select, fine-tune, and serve models on its own hardware would compound that advantage over time in ways that a company dependent on a third-party API would not.

The engineering team had strong Python and application development skills but no prior experience operating GPU infrastructure or large model serving. They had not run a CUDA workload in production, had not managed GPU memory allocation under load, and had not operated inference serving infrastructure at any scale. Any platform delivered at the end of the engagement had to be operable by that team within a reasonable ramp period — which meant the operational model could not depend on deep GPU infrastructure expertise for routine operations.

The architecture also had to satisfy two distinct reviewer audiences simultaneously: the company's CISO, who needed to see technical controls they could verify independently, and the enterprise clients' compliance officers, who needed policy documentation that matched their understanding of HIPAA requirements. These are materially different documents. A CISO review focuses on actual controls — what enforces what, where the logs go, how you detect a violation. A compliance officer review focuses on narrative attestation — does this architecture satisfy the HIPAA Security Rule's administrative, physical, and technical safeguards, and can you show me where each one lives? Producing both required structuring the work so that the technical controls and the compliance narrative were generated from the same source of truth.

Model lifecycle governance was a hard constraint with process implications. Every model update needed a documented approval chain — a designated clinical reviewer, a defined evaluation set run against the candidate model before promotion, a sign-off record, and a deployment event in the GitOps history. The process had to be lightweight enough that it did not slow the team's iteration pace, but rigorous enough that a compliance auditor reviewing it would recognise it as a defensible change control process.

Physical infrastructure was constrained by the company's existing co-location arrangement. GPU hardware had to fit in the existing rack footprint. This influenced the GPU selection (high memory density per rack unit was prioritised over raw aggregate compute) and the inference serving configuration (multi-model serving on shared GPU hardware was required, not one GPU per model).

The engagement opened with a PHI data-flow audit: a complete trace of every path PHI could take through the product during an AI-assisted documentation session. Where does PHI enter the system? Which services touch it? What is the network path for each touch? What are the exit points — intentional and unintentional? This produced a data-flow diagram that served three purposes: the basis for the HIPAA control mapping, the specification for the network boundary enforcement, and the evidence document that enterprise clients' security reviewers received first.

The architectural answer was straightforward in principle: all inference happens on owned hardware, all PHI stays on owned storage, no network path to a third-party model API exists in the production environment. The implementation depth was in making that principle structurally true rather than just policy-stated — enforced by network configuration and admission control, not by developer discipline.

We selected vLLM for inference serving after evaluating three alternatives. vLLM's key properties for this use case: efficient batched inference with PagedAttention (which eliminates KV-cache memory fragmentation, reducing memory waste under variable load and improving throughput per GPU), OpenAI-compatible API surface (minimising application code changes), multi-model serving via separate vLLM processes with GPU memory partitioned at the process level using CUDA_VISIBLE_DEVICES, and Apache 2.0 licence (no enterprise licensing friction). Models were sourced from Hugging Face, pinned by commit hash, verified by SHA-256 checksum, and stored in self-hosted MinIO. No runtime model downloads in production — the MinIO bucket was the single source of truth for model artefacts, version-locked.

The RAG pipeline used PostgreSQL with pgvector — already in the company's operational footprint, familiar to the engineering team, with row-level security and audit logging inherited from the existing database operations practices. Clinical context (de-identified procedure documentation fragments, clinical terminology, organisational context) was embedded using a self-hosted embedding model also served by vLLM. The text never left the infrastructure boundary at any point in the retrieval or generation pipeline.

Policy enforcement was layered. OPA (Open Policy Agent) with Rego policies handled application-level enforcement: every inference request that carried PHI passed through a policy evaluation before the request was forwarded to the vLLM serving layer. Kyverno handled Kubernetes admission control: no workload could be scheduled that lacked the required labels for audit routing, and no container could run without a read-only root filesystem and a non-root user context. Both policy sets were version-controlled alongside application code and tested in CI on every merge — the policy test suite ran before any deployment, with failures blocking promotion.

The legal and security sign-off came in under six weeks — faster than any comparable infrastructure security review the company had previously conducted, because the evidence was pre-structured and queryable rather than assembled retrospectively. The enterprise accounts that had sent formal contract-review notices reviewed the control-mapping document, confirmed the PHI boundary was architecturally enforced, and retracted the notices. No on-site audit was required.

The clinical documentation AI feature shipped inside the infrastructure boundary. PHI did not transit any third-party API. The model governance process — release tagging, clinical sign-off entry, evaluation run documentation, GitOps deployment — produced an auditable artefact on every model update that satisfied both the CISO's change control requirements and the clinical team's need for a documented approval chain they could defend to their own medical directors.

Representative outcomes across comparable engagements (not precise client figures): Zero bytes of PHI transmitted to third-party APIs — the architecture makes it structurally impossible, enforced by network policy and OPA admission control, not by developer convention. Legal and security sign-off in under six weeks in engagements where comparable reviews had previously taken six months or stalled entirely, primarily because the evidence was structured and queryable from the first day of the review rather than assembled on request. Full HIPAA Security Rule control mapping, enforced by running policy-as-code rather than by attestation alone — every control claim is a pointer to a verifiable running component. Model governance process produces a complete auditable artefact on every model update, including evaluation results and clinical reviewer sign-off.

The existing PostgreSQL infrastructure, extended with pgvector rather than replaced. The existing application layer — the AI documentation feature was an addition, not a rewrite; the existing clinical workflow code was unchanged. The OPA and Kyverno policy framework, which the engineering team extended to cover non-AI compliance requirements after the initial deployment. The audit pipeline (Loki with structured events), now used for security event logging beyond the AI feature scope. The HIPAA control-mapping document, maintained as a living document alongside architecture changes — not a one-time engagement deliverable. The engineering team operates the vLLM inference cluster, manages model updates through the governance process, and extends the policy set independently. No Stribog involvement is required for routine operations.