Tabby vs Continue: Self-Hosted Coding Assistants on vLLM

Most of the sovereign-AI conversation has been about the production inference path — the model your application calls at runtime, the audit log a regulator will ask for, the GPU bill at scale — and we have written about it at production grade. But there is a second AI tier that almost every engineering organization adopted faster, with less scrutiny, and with a far higher concentration of valuable intellectual property flowing through it: the developer inner loop. The autocomplete, the chat panel, the "explain this" and "write a test for this" interactions that now happen hundreds of times a day on every engineer's machine.

When a developer accepts a ghost-text completion, the model that produced it received the surrounding code as context. When they ask the chat panel to refactor a class, the class went to the model. Over a working week, a SaaS coding assistant sees a representative sample of your entire codebase — the parts under active development, which is to say the parts that are not yet public, not yet patented, and most likely to matter. For a company whose source is its product, routing that stream to a vendor's cloud is a larger exposure than the production inference path, and it was adopted with a fraction of the governance.

This article is the engineering path to closing that gap without taking the autocomplete away. By 2026 the open-weight code models closed enough of the quality gap that a self-hosted assistant is a credible, daily-driver Copilot alternative rather than a downgrade you tolerate for compliance reasons. The stack is three components — an IDE client, a completion server, and an inference engine — and the whole thing runs inside your perimeter on hardware you already know how to operate.

The Inner Loop Is the Real Exposure Surface

Security reviews tend to focus on data at rest and data in transit through production systems. The developer assistant slips past that lens because it feels like a tool, not a data pipeline. It is a data pipeline. Every inline completion request carries the prefix and suffix around the cursor — frequently the full open file and several related files pulled in as context. Every chat turn carries whatever the developer selected or referenced. A repository-aware assistant additionally indexes the codebase to ground its answers, which means it has read everything, not just the open buffer.

The vendors are not cartoon villains about this. Most enterprise tiers contractually promise not to train on your code and offer zero-retention modes. But a contractual promise is a different category of assurance than an architectural guarantee, and the distinction is exactly the one our sovereignty thesis turns on. A zero-retention clause can change at the next renewal. A US-incorporated provider remains subject to CLOUD Act process regardless of where the bytes are stored. And a promise not to retain is unfalsifiable from your side — you cannot audit the absence of a log you have no access to. For a team whose confidentiality obligations are contractual — a defense subcontractor, a firm under a customer NDA, a team inside a clean-room acquisition — "trust us" is not a control that survives the audit.

Two Surfaces, One Served Model: Completion vs. Chat

A complete coding assistant is really two distinct products that engineers conflate because they live in the same editor. The first is inline completion — the low-latency ghost text that appears as you type, which must return in well under 500 milliseconds (ideally under 300) or developers turn it off. The second is the conversational surface — chat, multi-file edits, explanations, test generation — which tolerates a few seconds of latency in exchange for stronger reasoning. They have different latency budgets, different prompt formats, and frequently want different models.

Inline completion is a fill-in-the-middle (FIM) task: the model is given a prefix and a suffix and asked to produce the span between them. This is not the same as chat completion, and it needs a base model trained with FIM objectives and prompted with the model's specific sentinel tokens. The conversational surface is a standard instruction-following task and wants an instruct-tuned model. The clean architecture serves both from the same vLLM tier — possibly two model deployments behind one gateway — and uses the right client for each surface.

In practice the division of labor is: Tabby owns inline completion, because it is a purpose-built completion server with its own FIM handling, repository indexing, and editor plugins; Continue owns the conversational surface, because it is a backend-agnostic IDE client with excellent chat, edit, and context UX. Both point at inference you control. You can run either alone, but together they cover the full Copilot feature surface with two focused open-source projects instead of one closed product.

Picking the Code Model: Qwen2.5-Coder, DeepSeek-Coder, and the License Trap

The model choice is where the sovereignty argument either holds or quietly leaks. As of 2026 the open-weight code models are genuinely competitive: Qwen2.5-Coder has reported HumanEval scores in the high-80s — in the neighborhood of frontier closed models on that benchmark — and ships in sizes from 0.5B to 32B, so you match the model to the GPU. DeepSeek-Coder-V2-Lite (16B total, 2.4B active) is small enough for a consumer-class card, and its MoE routing keeps per-token compute — and therefore latency — low; the full DeepSeek-Coder-V2 is 236B total parameters and needs a multi-GPU server (DeepSeek documents 80GB×8 for BF16) regardless of how few experts are active per token. The day-to-day completion experience on either is not a compromise.

The trap is licensing, and it is precisely an optionality trap. Benchmark leaderboards will point you at models like Codestral, which Mistral reported as leading on fill-in-the-middle pass@1 at its May 2024 release versus DeepSeek Coder 33B — but it ships under a non-production license that restricts commercial use, which means adopting it for your team's daily assistant either violates the license or pulls you toward the vendor's paid API, reintroducing exactly the lock-in you self-hosted to escape. A truly open-weight model under Apache 2.0 or a comparably permissive license (most Qwen2.5-Coder sizes) is the choice that preserves the exit ramp; the DeepSeek-Coder family permits commercial self-hosting but ships under the DeepSeek License Agreement with use-based restrictions you must pass downstream — permissive, but not Apache 2.0. The point of anti-lock-in as a design principle is that you do not trade a SaaS dependency for a weights-license dependency.

• Completion model: a FIM-capable *base* model sized to your latency budget — Qwen2.5-Coder-1.5B or 7B (base) gives sub-300ms ghost text on a single mid-range GPU.
• Chat / edit model: a larger *instruct* model — Qwen2.5-Coder-14B/32B-Instruct or DeepSeek-Coder-V2-Lite-Instruct — where a few seconds of latency buys materially better reasoning.
• License: confirm the exact size's license, not the family's headline — in Qwen2.5-Coder every size is Apache 2.0 except the 3B, which ships under the Qwen Research License. Apache 2.0 / MIT keeps the exit ramp open.
• Provenance: pin the weight digest and verify it on load, the same supply-chain discipline you apply to any dependency.

Serving the Code Model with vLLM

vLLM is the inference engine for this tier for the same reasons it is the right choice in production: PagedAttention gives high throughput under concurrent load, it exposes an OpenAI-compatible API that both Continue and Tabby speak natively, and it is a standard container you can schedule on a GPU node. For a team assistant the property that matters most is throughput-under-concurrency — twenty developers firing completion requests as they type is a bursty, high-QPS workload, and naive serving collapses under it where continuous batching holds up. The deeper sizing, gateway, and audit design live in the production vLLM deployment guide; here we cover the code-assistant specifics.

Serving a code model is the same command as any other, with two wrinkles: completion needs /v1/completions (not just chat) so the FIM client can drive it, and generous concurrency because the workload is spiky.

# Completion server: FIM-capable BASE model, tuned for low-latency single-line ghost text.
# Smaller model + short max-len keeps time-to-first-token under the 300ms budget.
vllm serve Qwen/Qwen2.5-Coder-7B \
  --served-model-name code-completion \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --enable-chunked-prefill \
  --max-num-seqs 64 \
  --no-trust-remote-code \
  --port 8001

# Chat / edit server: larger INSTRUCT model (AWQ weights), latency-tolerant.
# Use the -AWQ repo: --quantization awq loads AWQ weights; it does not quantize bf16.
# Classic AWQ needs FP16 activations — --dtype half (not bfloat16). float16 is an alias.
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct-AWQ \
  --served-model-name code-chat \
  --dtype half \
  --quantization awq \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --no-trust-remote-code \
  --port 8002

# Both sit behind one gateway that adds auth, per-developer rate limits,
# and the audit record. Never expose vLLM directly to the editor.

Note --no-trust-remote-code: as in production, you do not let a downloaded model config execute arbitrary Python on your inference node. Note also the split between a small base model for completion and a larger instruct model for chat — that is the highest-leverage tuning choice in this stack. Spend the latency budget on the keystroke loop and the reasoning budget on the chat panel, rather than forcing one model to serve both poorly.

Click to zoom

Wiring Continue to Your Endpoint

Continue is the backend-agnostic IDE client. It does not ship a model; you point it at one. That is the property that makes it sovereign-friendly: it speaks the OpenAI-compatible protocol vLLM exposes, so configuration is an apiBase and a model name, not a vendor account. The file below registers a self-hosted chat model and a self-hosted autocomplete model with no path to an external provider — that absence is the control. If there is no vendor API key in the config, there is no path for code to reach a vendor.

name: Sovereign code assistant
version: 0.0.1
schema: v1

models:
  - name: code-chat
    provider: openai            # OpenAI-compatible wire protocol — not OpenAI the company
    model: code-chat            # matches vLLM --served-model-name
    apiBase: https://ai-gw.internal.example.com/v1
    apiKey: ${{ secrets.GW_TOKEN }}   # gateway bearer from secrets store, not a vendor key
    roles:
      - chat
      - edit
      - apply

  - name: code-autocomplete
    provider: openai
    model: code-completion
    apiBase: https://ai-gw.internal.example.com/v1
    apiKey: ${{ secrets.GW_TOKEN }}
    roles:
      - autocomplete
    # model: code-completion matches no built-in FIM heuristic; set Qwen template explicitly
    promptTemplates:
      autocomplete: "<|fim_prefix|>{{{prefix}}}<|fim_suffix|>{{{suffix}}}<|fim_middle|>"

context:
  - provider: code
  - provider: diff
  - provider: currentFile
  # No 'codebase' embeddings provider pointed at a SaaS embedder:
  # if you index the repo, the embedding model is self-hosted too.

The discipline worth enforcing across the team is that this file contains no external apiBase and no vendor API key — distribute it through your platform's golden-path tooling so the secure configuration is the default one, not a special setup the careful people use. If you want repository-wide embeddings context, host the embedder on the same vLLM tier; otherwise you have moved the residency problem from the chat model to the embedder.

Wiring Tabby for Team Completion

Tabby is the purpose-built completion half. It is a single Rust binary that hosts the completion logic, indexes your repositories for context, ships VS Code and JetBrains plugins, exposes an OpenAPI interface, and adds team administration; SSO requires a paid Tabby licence (Enterprise tier), so on the free tier identity is enforced at the inference gateway instead — a self-contained team server with no external database or cloud dependency. You can let Tabby run its own bundled inference for completion, or, to consolidate on one GPU tier, point its completion backend at your vLLM /v1/completions endpoint and supply the FIM prompt template for your chosen model. Qwen2.5-Coder's FIM sentinels are shown below; using the wrong template is the usual cause of garbled completions.

# ~/.tabby/config.toml
# Render api_key from a Secret at deploy time — Tabby does not expand env: prefixes.
[model.completion.http]
kind = "vllm/completion"
model_name = "code-completion"
api_endpoint = "https://ai-gw.internal.example.com/v1"
api_key = "replace-with-gateway-token"

# Qwen2.5-Coder fill-in-the-middle sentinels. The model is given prefix + suffix
# and produces the span between them — this is NOT a chat prompt.
# kind vllm/completion honors prompt_template; openai/completion hardcodes FIM and
# sends a suffix field that vLLM's /v1/completions rejects.
prompt_template = "<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"

[model.chat.http]
kind = "openai/chat"
model_name = "code-chat"
api_endpoint = "https://ai-gw.internal.example.com/v1"
api_key = "replace-with-gateway-token"

# Repository context: indexed locally, never shipped out. Tabby reads these repos
# to ground completions; the index lives on the Tabby server inside your perimeter.
[[repositories]]
name = "monorepo"
git_url = "git@git.internal.example.com:platform/monorepo.git"

Hosting Tabby as a shared team server is the operationally sane pattern: one GPU box serves completion for everyone, so individual laptops do not each need a card, and the repository index is built once rather than per-machine. A workstation-class GPU serves five to ten developers; beyond that, scale the vLLM tier behind the gateway the same way you would any other inference workload. The editor plugins remain thin clients — the model, the index, and the context all stay server-side inside the boundary.

Proving Code Never Leaves: The Data-Residency Boundary

The reason to do all of this is to be able to make one statement defensibly: proprietary source code does not leave infrastructure we control. With the SaaS assistant that statement rests on a contract; with the self-hosted stack it rests on network topology a security team can verify. The same residency logic that makes on-prem inference mandatory for PHI under HIPAA applies to source under NDA or export control: the control is the boundary, not the promise.

Making the boundary real and auditable is a small number of concrete controls, enforced at the inference gateway — the same gateway pattern from the production tier, reused rather than reinvented.

• Egress policy: a default-deny network policy on the inference namespace so the model tier physically cannot reach the public internet — the strongest form of the residency guarantee is that the path does not exist.
• Authenticated gateway: every completion and chat request carries a static gateway bearer (Continue's apiKey / Tabby's api_key) from your secrets store — optionally one secret per developer for attribution and per-developer rate limits — that the gateway validates and can rotate or revoke; neither Continue nor Tabby implements OIDC login or token refresh, so treat the value as a rotated secret rather than an IdP-issued short-lived token.
• Audit record: the gateway logs who requested inference and when (not necessarily the code itself), giving you the access trail an auditor expects without building a second copy of your source.
• No external embedder: if you index the repo for context, the embedding model is self-hosted too — a SaaS embedding endpoint is the most common quiet leak in an otherwise sovereign setup.

Click to zoom

Latency, Concurrency, and GPU Sizing for the Inner Loop

The inner loop has an unusual performance profile that distinguishes it from batch or production inference. Completion is latency-bound and bursty: a developer's keystrokes generate a stream of requests, most of which are cancelled before they finish because the next keystroke supersedes them. Throughput matters less than tail latency — a p99 of 800ms makes the assistant feel broken even if the median is 200ms. The levers are model size (smaller completes faster), max-model-len (a shorter context shrinks the prefill), and serving the completion model on its own GPU so a heavy chat request cannot stall the keystroke loop.

The asymmetry is the whole sizing story: chat traffic arrives in seconds-apart bursts rather than per-keystroke, so the instruct model can run hotter on memory utilization, while the completion model is tuned purely for tail latency. Spend silicon on the two independently — combining them onto one model serves neither well.

Optionality: The Exit Ramp You Build In

The architecture is deliberately boring at its seams, and that is the optionality argument made concrete. Continue and Tabby both speak the OpenAI-compatible protocol, so the model behind the endpoint is a swap, not a migration. When a better open-weight code model ships — and on the current cadence one ships every few months — adopting it is a new vLLM deployment and a config change, with the old model kept warm until the new one proves out. There is no client rewrite, no SDK churn, no renegotiated contract.

Contrast the SaaS exit: every developer reconfigures their editor, you renegotiate or cancel a contract, and any workflow built on vendor-specific features has to be rebuilt. The self-hosted stack inverts that — because each layer is an open component talking a standard protocol, you can replace any one of them (IDE client, completion server, model, GPU) without disturbing the others. The exit ramp is not a feature you bolt on later; it is a consequence of choosing components that compose over products that capture.

The Long Game: Codebase Intelligence You Own

There is a longer arc here, beyond the immediate confidentiality argument. The assistant that has indexed your monorepo, completes in the idioms your team actually uses, and knows your internal libraries is accumulating institutional knowledge about how your organization writes software. As open-weight models become cheap to fine-tune or adapter-tune on a private corpus, the team that owns its inference tier can turn its own codebase into a private model advantage that compounds over years, on infrastructure it controls. The team renting its assistant cannot, because the substrate that would learn from its code belongs to someone else.

This is the sovereign-AI capability applied to the place engineers spend their day. It is not the flashiest tier — there is no board demo in a faster autocomplete — but it is the one with the highest ratio of IP exposure to governance, and the one where the open ecosystem has matured enough that self-hosting is a lateral move in experience and a decisive move in control. Closing it is a few containers, two config files, and a default-deny egress rule. The result is an assistant your engineers like using and a statement you can defend: the code never left.