Self-Hosted Coding Assistants: Continue, Tabby, and vLLM On-Prem

Most of the sovereign-AI conversation has been about the production inference path — the model your application calls at runtime, the audit log a regulator will ask for, the GPU bill at scale — and we have written about it at production grade. But there is a second AI tier that almost every engineering organization adopted faster, with less scrutiny, and with a far higher concentration of valuable intellectual property flowing through it: the developer inner loop. The autocomplete, the chat panel, the "explain this" and "write a test for this" interactions that now happen hundreds of times a day on every engineer's machine.

When a developer accepts a ghost-text completion, the model that produced it received the surrounding code as context. When they ask the chat panel to refactor a class, the class went to the model. Over a working week, a SaaS coding assistant sees a representative sample of your entire codebase — the parts under active development, which is to say the parts that are not yet public, not yet patented, and most likely to matter. For a company whose source is its product, routing that stream to a vendor's cloud is a larger exposure than the production inference path, and it was adopted with a fraction of the governance.

This article is the engineering path to closing that gap without taking the autocomplete away. By 2026 the open-weight code models closed enough of the quality gap that a self-hosted assistant is a credible, daily-driver Copilot alternative rather than a downgrade you tolerate for compliance reasons. The stack is three components — an IDE client, a completion server, and an inference engine — and the whole thing runs inside your perimeter on hardware you already know how to operate.

The Inner Loop Is the Real Exposure Surface

Security reviews tend to focus on data at rest and data in transit through production systems. The developer assistant slips past that lens because it feels like a tool, not a data pipeline. It is a data pipeline. Every inline completion request carries the prefix and suffix around the cursor — frequently the full open file and several related files pulled in as context. Every chat turn carries whatever the developer selected or referenced. A repository-aware assistant additionally indexes the codebase to ground its answers, which means it has read everything, not just the open buffer.

The vendors are not cartoon villains about this. Most enterprise tiers contractually promise not to train on your code and offer zero-retention modes. But a contractual promise is a different category of assurance than an architectural guarantee, and the distinction is exactly the one our sovereignty thesis turns on. A zero-retention clause can change at the next renewal. A US-incorporated provider remains subject to CLOUD Act process regardless of where the bytes are stored. And a promise not to retain is unfalsifiable from your side — you cannot audit the absence of a log you have no access to. For a team whose confidentiality obligations are contractual — a defense subcontractor, a firm under a customer NDA, a team inside a clean-room acquisition — "trust us" is not a control that survives the audit.

Two Surfaces, One Served Model: Completion vs. Chat

A complete coding assistant is really two distinct products that engineers conflate because they live in the same editor. The first is inline completion — the low-latency ghost text that appears as you type, which must return in well under 500 milliseconds (ideally under 300) or developers turn it off. The second is the conversational surface — chat, multi-file edits, explanations, test generation — which tolerates a few seconds of latency in exchange for stronger reasoning. They have different latency budgets, different prompt formats, and frequently want different models.

Inline completion is a fill-in-the-middle (FIM) task: the model is given a prefix and a suffix and asked to produce the span between them. This is not the same as chat completion, and it needs a base model trained with FIM objectives and prompted with the model's specific sentinel tokens. The conversational surface is a standard instruction-following task and wants an instruct-tuned model. The clean architecture serves both from the same vLLM tier — possibly two model deployments behind one gateway — and uses the right client for each surface.

In practice the division of labor is: Tabby owns inline completion, because it is a purpose-built completion server with its own FIM handling, repository indexing, and editor plugins; Continue owns the conversational surface, because it is a backend-agnostic IDE client with excellent chat, edit, and context UX. Both point at inference you control. You can run either alone, but together they cover the full Copilot feature surface with two focused open-source projects instead of one closed product.

Picking the Code Model: Qwen2.5-Coder, DeepSeek-Coder, and the License Trap

The model choice is where the sovereignty argument either holds or quietly leaks. As of 2026 the open-weight code models are genuinely competitive: Qwen2.5-Coder has reported HumanEval scores in the high-80s — in the neighborhood of frontier closed models on that benchmark — and ships in sizes from 0.5B to 32B, so you match the model to the GPU. DeepSeek-Coder-V2 uses a mixture-of-experts design that keeps active parameters low, letting a capable model run on a consumer-class card. The day-to-day completion experience on either is not a compromise.

The trap is licensing, and it is precisely an optionality trap. Benchmark leaderboards will point you at models like Codestral, which posts the highest FIM pass@1 numbers of anything measured — but it ships under a non-production license that restricts commercial use, which means adopting it for your team's daily assistant either violates the license or pulls you toward the vendor's paid API, reintroducing exactly the lock-in you self-hosted to escape. A truly open-weight model under Apache 2.0 or a comparably permissive license (most Qwen2.5-Coder sizes, the DeepSeek-Coder family) is the choice that preserves the exit ramp. The point of anti-lock-in as a design principle is that you do not trade a SaaS dependency for a weights-license dependency.

• Completion model: a FIM-capable *base* model sized to your latency budget — Qwen2.5-Coder-1.5B or 7B (base) gives sub-300ms ghost text on a single mid-range GPU.
• Chat / edit model: a larger *instruct* model — Qwen2.5-Coder-14B/32B-Instruct or a DeepSeek-Coder-V2 instruct variant — where a few seconds of latency buys materially better reasoning.
• License: confirm the exact size's license, not the family's headline. Some flagship sizes carry stricter terms than the smaller ones. Apache 2.0 / MIT keeps the exit ramp open.
• Provenance: pin the weight digest and verify it on load, the same supply-chain discipline you apply to any dependency.

Serving the Code Model with vLLM

vLLM is the inference engine for this tier for the same reasons it is the right choice in production: PagedAttention gives high throughput under concurrent load, it exposes an OpenAI-compatible API that both Continue and Tabby speak natively, and it runs as a standard container you can schedule on a GPU node. For a team assistant, throughput-under-concurrency is the property that matters — twenty developers each firing completion requests as they type is a bursty, high-QPS workload, and naive serving collapses under it where vLLM's continuous batching holds up. The deeper serving architecture — GPU sizing math, the inference gateway, audit logging — is covered in the production vLLM deployment guide; here we focus on the code-assistant specifics.

Serving a code model is the same command as serving any other, with two wrinkles: completion needs the /v1/completions endpoint (not just chat) so the FIM client can drive it directly, and you want generous concurrency because the completion workload is spiky.

# Completion server: FIM-capable BASE model, tuned for low-latency single-line ghost text.
# Smaller model + short max-len keeps time-to-first-token under the 300ms budget.
vllm serve Qwen/Qwen2.5-Coder-7B \
  --served-model-name code-completion \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --enable-chunked-prefill \
  --max-num-seqs 64 \
  --trust-remote-code false \
  --port 8001

# Chat / edit server: larger INSTRUCT model, latency-tolerant, stronger reasoning.
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct \
  --served-model-name code-chat \
  --dtype bfloat16 \
  --quantization awq \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code false \
  --port 8002

# Both sit behind one gateway that adds OIDC auth, per-developer rate limits,
# and the audit record. Never expose vLLM directly to the editor.

Note --trust-remote-code false: as in production, you do not let a downloaded model config execute arbitrary Python on your inference node. Note also the split between a small base model for completion and a larger instruct model for chat — this is the single most impactful tuning decision for the inner-loop experience, because it lets you spend your latency budget where it is felt (the keystroke loop) and your reasoning budget where it pays off (the chat panel).

Click to zoom

Wiring Continue to Your Endpoint

Continue is the backend-agnostic IDE client. It does not ship a model; you point it at one. That is the property that makes it sovereign-friendly: it speaks the OpenAI-compatible protocol vLLM exposes, so configuring it against your own endpoint is a matter of an apiBase and a model name. The configuration below registers your self-hosted chat model and an autocomplete model, with no path to any external provider — which is itself the control. If there is no API key for a vendor in the config, there is no way for code to reach one.

name: Sovereign code assistant
version: 0.0.1
schema: v1

models:
  - name: code-chat
    provider: openai            # OpenAI-compatible wire protocol — not OpenAI the company
    model: code-chat            # matches vLLM --served-model-name
    apiBase: https://ai-gw.internal.example.com/v1
    apiKey: ${{ secrets.GW_TOKEN }}   # short-lived OIDC token, not a vendor key
    roles:
      - chat
      - edit
      - apply

  - name: code-autocomplete
    provider: openai
    model: code-completion
    apiBase: https://ai-gw.internal.example.com/v1
    apiKey: ${{ secrets.GW_TOKEN }}
    roles:
      - autocomplete

context:
  - provider: code
  - provider: diff
  - provider: currentFile
  # No 'codebase' embeddings provider pointed at a SaaS embedder:
  # if you index the repo, the embedding model is self-hosted too.

The discipline worth enforcing across the team is that this file contains no external apiBase and no vendor API key — distribute it through your platform's golden-path tooling so the secure configuration is the default one, not a thing each developer has to assemble correctly. If you also want repository-wide context (embeddings-based codebase retrieval), the embedding model is self-hosted on the same vLLM tier; otherwise you have moved the data-residency problem from the chat model to the embedder and accomplished nothing.

Wiring Tabby for Team Completion

Tabby is the purpose-built completion half. It is a single Rust binary that hosts the completion logic, indexes your repositories for context, ships VS Code and JetBrains plugins, exposes an OpenAPI interface, and adds team administration and SSO — a self-contained team server with no external database or cloud dependency. You can let Tabby run its own bundled inference for completion, or, to consolidate on one GPU tier, point its completion backend at your vLLM /v1/completions endpoint and supply the FIM prompt template for your chosen model. Qwen2.5-Coder's FIM sentinels are shown below; using the wrong template is the usual cause of garbled completions.

# ~/.tabby/config.toml
[model.completion.http]
kind = "openai/completion"
model_name = "code-completion"
api_endpoint = "https://ai-gw.internal.example.com/v1"
api_key = "env:TABBY_GW_TOKEN"

# Qwen2.5-Coder fill-in-the-middle sentinels. The model is given prefix + suffix
# and produces the span between them — this is NOT a chat prompt.
prompt_template = "<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"

[model.chat.http]
kind = "openai/chat"
model_name = "code-chat"
api_endpoint = "https://ai-gw.internal.example.com/v1"
api_key = "env:TABBY_GW_TOKEN"

# Repository context: indexed locally, never shipped out. Tabby reads these repos
# to ground completions; the index lives on the Tabby server inside your perimeter.
[[repositories]]
name = "monorepo"
git_url = "[email protected]:platform/monorepo.git"

Hosting Tabby as a shared team server is the operationally sane pattern: one GPU box serves completion for everyone, so individual laptops do not each need a card, and the repository index is built once and reused. A single workstation-class GPU comfortably serves inline completion for a team of five to ten developers; beyond that you scale the vLLM tier behind the gateway exactly as you would any other inference workload. The editor plugins are thin clients — model, index, and context all stay server-side, inside the boundary.

Proving Code Never Leaves: The Data-Residency Boundary

The reason to do all of this is to be able to make one statement defensibly: proprietary source code does not leave infrastructure we control. With the SaaS assistant that statement rests on a contract. With the self-hosted stack it rests on the network topology — and a network topology is something a security team can verify and a regulator can be shown. The same data-residency logic that makes on-prem inference mandatory for PHI under HIPAA applies to source code under an NDA or export-control regime: the control has to be the boundary, not the promise.

Making the boundary real and auditable is a small number of concrete controls, enforced at the inference gateway — the same gateway pattern from the production tier, reused rather than reinvented.

• Egress policy: a default-deny network policy on the inference namespace so the model tier physically cannot reach the public internet — the strongest form of the residency guarantee is that the path does not exist.
• Authenticated gateway: every completion and chat request carries a short-lived OIDC token, so access is tied to an identity and revocable, and there is no shared static key to leak.
• Audit record: the gateway logs who requested inference and when (not necessarily the code itself), giving you the access trail an auditor expects without building a second copy of your source.
• No external embedder: if you index the repo for context, the embedding model is self-hosted too — a SaaS embedding endpoint is the most common quiet leak in an otherwise sovereign setup.

Click to zoom

Latency, Concurrency, and GPU Sizing for the Inner Loop

The inner loop has an unusual performance profile that distinguishes it from batch or production inference. Completion is latency-bound and bursty: a developer's keystrokes generate a stream of requests, most of which are cancelled before they finish because the next keystroke supersedes them. Throughput matters less than tail latency — a p99 of 800ms makes the assistant feel broken even if the median is 200ms. The levers are model size (smaller completes faster), max-model-len (a shorter context shrinks the prefill), and serving the completion model on its own GPU so a heavy chat request cannot stall the keystroke loop.

The asymmetry is the whole sizing story: chat traffic arrives in seconds-apart bursts rather than per-keystroke, so the instruct model can run hotter on memory utilization, while the completion model is tuned purely for tail latency. Spend silicon on the two independently — combining them onto one model serves neither well.

Optionality: The Exit Ramp You Build In

The architecture is deliberately boring at its seams, and that is the optionality argument made concrete. Continue and Tabby both speak the OpenAI-compatible protocol, so the model behind the endpoint is a swap, not a migration. When a better open-weight code model ships — and on the current cadence one ships every few months — adopting it is a new vLLM deployment and a config change, with the old model kept warm until the new one proves out. There is no client rewrite, no SDK churn, no renegotiated contract.

Contrast the SaaS exit: every developer reconfigures their editor, you renegotiate or cancel a contract, and any workflow built on vendor-specific features has to be rebuilt. The self-hosted stack inverts that — because each layer is an open component talking a standard protocol, you can replace any one of them (IDE client, completion server, model, GPU) without disturbing the others. The exit ramp is not a feature you bolt on later; it is a consequence of choosing components that compose over products that capture.

The Long Game: Codebase Intelligence You Own

There is a longer arc here, beyond the immediate confidentiality argument. The assistant that has indexed your monorepo, completes in the idioms your team actually uses, and knows your internal libraries is accumulating institutional knowledge about how your organization writes software. As open-weight models become cheap to fine-tune or adapter-tune on a private corpus, the team that owns its inference tier can turn its own codebase into a private model advantage that compounds over years, on infrastructure it controls. The team renting its assistant cannot, because the substrate that would learn from its code belongs to someone else.

This is the sovereign-AI capability applied to the place engineers spend their day. It is not the flashiest tier — there is no board demo in a faster autocomplete — but it is the one with the highest ratio of IP exposure to governance, and the one where the open ecosystem has matured enough that self-hosting is a lateral move in experience and a decisive move in control. Closing it is a few containers, two config files, and a default-deny egress rule. The result is an assistant your engineers like using and a statement you can defend: the code never left.