Own the Registry: Harbor and Zot for Air-Gapped Images

Almost every Kubernetes platform we review has the same unexamined dependency. The cluster is redundant, the storage replicated, the control plane spread across three failure domains — and every image it runs is fetched at pull time from an account somebody else operates. This article is the afterwards install tutorials skip: the semantics that decide whether an owned registry is an asset or a new outage.

The Registry Is a Control Point, Not a Bucket

Start with the blast radius. If the registry goes unreachable, running pods keep running — their images are already on the node. What stops is everything needing a fresh pull: scale-up, node replacement, reschedule, rollout, recovery. The registry is not on your product's request path but on the request path of every change to it.

Then the rate limit. Docker Hub applies a six-hour pull budget to unauthenticated and Personal users: as its usage documentation states at the time of writing, 100 pulls per six hours per IPv4 address or IPv6 /64 subnet unauthenticated, 200 for a Personal account. The addressing detail hurts clusters: a node pool behind one NAT gateway is one IPv4 address, so forty nodes share one budget and a rolling DaemonSet restart can exhaust it. Treat the figure as dated, the mechanism as permanent.

The third exposure matters most at disconnected and edge sites: with no local registry the WAN link becomes a scheduling dependency, and a site whose uplink is down cannot replace a failed pod however healthy its rack.

What a Registry Must Do Before You Call It Yours

A feature matrix is the wrong instrument. Hold any candidate — including the one you run — against a requirements list.

• OCI Distribution conformance, including the referrers API. The distribution specification defines GET /v2/<name>/referrers/<digest> — endpoint end-12a, current in v1.1.1 (29 January 2025). It is how signatures and SBOMs live beside the image they describe; without it your supply-chain attestations break.
• Authorisation with real scopes — project-level separation, and machine identities that are not a shared admin password.
• Replication you can direct — push and pull, scheduled and event-driven, with a bandwidth ceiling and its own trust material.
• Retention, garbage collection and quota as three separate jobs — teams collapse them into one wrong mental model.
• Scanning that works without egress, or an air-gapped scanner reports clean because it has nothing to compare against.
• Immutability for release tags — enforced at push, not advisory.
• An exit that is a copy, not a migration project.

Harbor and Zot: Two Honest Answers to One Spec

We argued the head-to-head elsewhere, when leaving GitHub for a self-hosted forge and CI. In an estate it is an operational-cost decision about tier, not a feature contest.

Harbor is a CNCF graduated project — the eleventh to graduate, on 23 June 2020 — and a multi-service application, not a daemon: PostgreSQL, Redis, a job service and a scanner adapter alongside the registry. Its installation prerequisites put the floor at 2 CPU, 4 GB RAM and 40 GB disk, the recommendation at 4 CPU, 8 GB and 160 GB. That buys project RBAC, replication, retention, quotas, immutability and scanning as first-class API objects.

zot is a CNCF Sandbox project, accepted 13 December 2022, and its stated design goal is a production-ready, vendor-neutral registry built purely on OCI standards — OCI formats on the wire and on disk — as a single binary. Deduplication and garbage collection are in that binary; storage sits on a filesystem, S3-compatible object storage or GCS. There is no database to back up.

The tiering follows. Harbor is the core of record, where governance, scanning and replication policy live. zot goes at a site — one binary, one file, a filesystem. Both speak the same specification, which is the only reason the split is safe. Current releases: Harbor v2.15.2, 2 July 2026, and zot v2.1.18, 24 June 2026.

Click to zoom

Pull-Through Cache: Get Docker Hub Out of the Hot Path

A proxy cache is the cheapest sovereignty win available: it changes no manifest. In Harbor it is a project bound to a registered upstream endpoint, and three behaviours in the proxy cache documentation are load-bearing. You cannot push to one, so nobody makes it a source of truth. Since v2.1.1 Harbor checks freshness with HEAD requests, which do not trigger Docker Hub's rate limiter, so only real pulls spend budget. And if the upstream is unreachable, the project serves the cached image — the failure mode you bought it for.

Harbor 2.15.0 added a per-project `max_upstream_conn` capping upstream connections, so one busy cache cannot saturate a shared link.

Then the step teams skip: pulls route through the cache only if the runtime is told to. containerd reads per-registry configuration from the directory named by config_path — and if config_path is absent, every hosts.toml on the node is ignored in silence.

# containerd 2.x. On 1.x the table is
# [plugins."io.containerd.grpc.v1.cri".registry] with version = 2.
version = 3

[plugins."io.containerd.cri.v1.images".registry]
  config_path = "/etc/containerd/certs.d"

# This file is read only at daemon start, so:
#   systemctl restart containerd
# Later edits under certs.d need no restart.

server = "https://registry-1.docker.io"

[host."https://harbor.dmz.internal.example/v2/dockerhub-proxy"]
  capabilities = ["pull", "resolve"]
  ca = "/etc/containerd/certs.d/harbor-internal-ca.crt"
  override_path = true
  dial_timeout = "2s"

Two deliberate choices. capabilities omits push, matching what the cache permits. server keeps Docker Hub as a fallback — right in a DMZ, wrong inside an air gap, where dropping that line makes a cache miss fail loudly rather than time out. zot gets there differently: sync with onDemand true fetches an image the first time a client asks.

Replication Topologies for Air Gap and Edge

Harbor's replication rules support push- and pull-based modes with manual, scheduled and event-based triggers, plus a per-task bandwidth ceiling in kilobytes per second, where -1 means unlimited. The topology decision is which direction the connection opens — a security decision first.

Prefer pull-based replication initiated inside the restricted zone. Pushing into an air gap means the DMZ holds credentials for the protected registry plus an inbound path to it; pulling needs neither. Use a cron trigger rather than event-based, so each crossing is scheduled and reviewable, and cap the bandwidth so one large base image cannot starve the link. Harbor 2.15.0's per-endpoint CA certificates matter here: private PKI is the norm, and pinning trust per endpoint beats an estate-wide CA.

At the site, zot mirrors from the core on a schedule. Its mirroring documentation is explicit that onDemand: false plus a pollInterval and at least one content entry enable periodic mode — and equally explicit about where not to use it: because Docker Hub rate-limits pulls and does not support catalog listing, do not use polled mirroring against it. Point the poll at your own core.

One coupling is enforced at startup, not warned about at runtime. preserveDigest: true keeps a digest pinned in a Git-tracked manifest resolvable after the mirror hop; without it, OCI conversion can change the digest and the pinned reference stops matching. It requires http.compat: ["docker2s2"] — that exact string — and zot refuses to start otherwise. Sync reads upstream credentials from a credentialsFile, which any core worth running demands.

{
  "storage": {
    "rootDirectory": "/var/lib/zot",
    "dedupe": true,
    "gc": true,
    "gcDelay": "1h",
    "gcInterval": "24h"
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000",
    "compat": ["docker2s2"],
    "tls": {
      "cert": "/etc/zot/tls/site.crt",
      "key": "/etc/zot/tls/site.key"
    },
    "auth": { "htpasswd": { "path": "/etc/zot/htpasswd" } }
  },
  "log": { "level": "info" },
  "extensions": {
    "sync": {
      "credentialsFile": "/etc/zot/sync-credentials.json",
      "registries": [
        {
          "urls": ["https://harbor.core.internal.example"],
          "onDemand": false,
          "pollInterval": "6h",
          "tlsVerify": true,
          "certDir": "/etc/zot/core-ca",
          "maxRetries": 5,
          "retryDelay": "5m",
          "preserveDigest": true,
          "content": [{ "prefix": "/platform/**" }]
        }
      ]
    }
  }
}

When no wire crosses the boundary, the transfer path is still supported rather than a hack. skopeo sync copies every image found at a source to a destination, with docker, dir and yaml source transports and docker or dir destinations — which makes a directory on removable media a legitimate link in the chain.

#!/usr/bin/env bash
# DMZ -> removable media -> air-gapped core.
set -euo pipefail

MEDIA=/media/transfer/2026-07-30
CORE=harbor.core.internal.example
: "${DMZ_ROBOT:?}" "${DMZ_SECRET:?}" "${CORE_ROBOT:?}" "${CORE_SECRET:?}"
mkdir -p "$MEDIA"

# 1. Pin what crosses. Both projects are private, so the sync
#    authenticates; the file holding the secret stays off the media.
SYNC_YAML=$(mktemp)
trap 'rm -f "$SYNC_YAML"' EXIT
cat >"$SYNC_YAML" <<YAML
harbor.dmz.internal.example:
  credentials:
    username: ${DMZ_ROBOT}
    password: ${DMZ_SECRET}
  tls-verify: true
  images:
    dockerhub-proxy/library/nginx:
      - "1.29.4"
    platform/kube-state-metrics:
      - "v2.17.0"
  images-by-semver:
    platform/cert-manager-controller: ">= 1.19.0"
YAML

# 2. --scoped prevents basename collisions on the media.
skopeo sync --src yaml --dest dir --scoped --all \
  "$SYNC_YAML" "$MEDIA/blobs"

# 3. Record what left, to prove what arrived.
( cd "$MEDIA/blobs" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) \
  >"$MEDIA/MANIFEST.sha256"

# --- courier crosses; below runs inside the air gap ---

# 4. Verify before any blob reaches the registry.
( cd "$MEDIA/blobs" && sha256sum --check --quiet "$MEDIA/MANIFEST.sha256" )

# 5. One login; --dest-creds would expose the secret in ps output.
skopeo login --username "$CORE_ROBOT" --password-stdin "$CORE" <<<"$CORE_SECRET"

# 6. Scoped layout is <source-host>/<repo path>:<tag>. Harbor rejects a
#    push into a missing project, so platform and dockerhub-proxy must
#    already exist on the core.
find "$MEDIA/blobs" -type d -name '*:*' -print0 |
  while IFS= read -r -d '' dir; do
    ref=${dir#"$MEDIA/blobs/"}
    repo=${ref%:*}
    skopeo copy --all --dest-cert-dir /etc/pki/harbor \
      "dir:$dir" "docker://$CORE/${repo#*/}:${ref##*:}"
  done

One project to watch rather than deploy: Harbor Satellite aims at exactly this problem; its latest release is v0.0.4, 20 July 2026. That number is the assessment — a pre-1.0 dependency in a disconnected site's pull path is not a trade we would make yet.

Storage Arithmetic: Retention, Collection, Dedupe

This is where an owned registry most often becomes an incident. Harbor's garbage collection documentation states the first half plainly: when you delete images from Harbor, space is not automatically freed up, and you must run garbage collection to remove blobs no longer referenced by a manifest. Deleting a tag deletes a pointer; the bytes stay and keep counting against quota.

The second half is more reassuring: on the current release line GC does not interrupt push, pull or delete, so there is no read-only window to schedule around. But GC reserves a two-hour window protecting recent uploads, so a nightly job will not reclaim the last hours' churn, and manual runs are capped at once per minute — a throttle on retriggering by hand, not on your schedule.

Retention surprises people most. Harbor's tag retention rules only ever identify which tags to retain — you do not define rules to explicitly remove tags — and rules combine with an OR algorithm, so the retained set is their union. OR keeps more, never less. Accidental deletion comes from a keep set that is too narrow: anything matched by no rule is a removal candidate. Read the rules back adversarially: which artifact matches no rule at all?

zot expresses the same arithmetic with fewer parts. Its admin configuration documents deduplication — one copy of specific content on disk, referenced by many manifests — with gc, gcDelay (tuned to client network speeds and blob sizes) and gcInterval. Dedupe changes the sums in your favour, unpredictably: shared base layers make quota and real disk diverge, so plan on measured disk.

Click to zoom

Audit-Grade: Immutability, Offline CVE Data, Scoped Identity

An auditor asks three things: can a release artifact change after the fact, what can write to it, and is your vulnerability data current.

Harbor's tag immutability rules block a push whose tags match existing tags in scope, and the docs go further than most teams realise: an immutable tagged artifact cannot be deleted, and cannot be altered in any way such as through re-pushing, re-tagging, or replication. That last word is the valuable one — this is not a convention another site can overwrite.

For identity, robot accounts are how automation authenticates: a secret rather than a human credential, a configurable expiration or an explicit never-expire, and a refreshable secret so rotation does not mean recreating every reference. One documented constraint shapes the design — push permission must be assigned together with pull, never alone — so puller and pusher are two identities with two lifetimes, the pusher getting the short expiry plus an admission policy watching what it produces.

#!/usr/bin/env bash
# Endpoint, cache project, release project, immutability rule, robot.
set -euo pipefail

H=https://harbor.dmz.internal.example
A=(-sS --fail-with-body -u "admin:${HARBOR_ADMIN_PASSWORD:?set this}"
   -H 'Content-Type: application/json')

# 1. Authenticated: 200 pulls per six hours on Personal, unlimited on paid.
curl "${A[@]}" -X POST "$H/api/v2.0/registries" -d @- <<JSON
{ "name": "docker-hub", "type": "docker-hub",
  "url": "https://hub.docker.com",
  "credential": { "type": "basic",
                  "access_key": "${HUB_USER:?}", "access_secret": "${HUB_TOKEN:?}" } }
JSON

REG_ID=$(curl "${A[@]}" "$H/api/v2.0/registries?q=name%3Ddocker-hub" | jq -r '.[0].id')

# 2. registry_id makes it a cache: push-forbidden from birth.
curl "${A[@]}" -X POST "$H/api/v2.0/projects" -d @- <<JSON
{ "project_name": "dockerhub-proxy", "registry_id": $REG_ID,
  "metadata": { "public": "true" } }
JSON

# 3. The release project steps 4-5 govern. No registry_id, so a
#    normal project rather than a cache.
curl "${A[@]}" -X POST "$H/api/v2.0/projects" -d @- <<'JSON'
{ "project_name": "platform", "metadata": { "public": "false" } }
JSON

# 4. Matching pushes rejected; the artifact then cannot be deleted,
#    re-tagged, or altered by replication.
curl "${A[@]}" -X POST "$H/api/v2.0/projects/platform/immutabletagrules" -d @- <<'JSON'
{ "disabled": false, "action": "immutable",
  "template": "immutable_template", "priority": 0,
  "scope_selectors": { "repository": [
    { "decoration": "repoMatches", "kind": "doublestar", "pattern": "**" } ] },
  "tag_selectors": [
    { "decoration": "matches", "kind": "doublestar", "pattern": "v*" } ] }
JSON

# 5. Pull-only. Push cannot be granted without pull, so the pusher
#    is a separate, shorter-lived identity; its secret is returned once.
umask 077
curl "${A[@]}" -X POST "$H/api/v2.0/robots" -d @- >node-puller.json <<'JSON'
{ "name": "node-puller", "duration": 90, "level": "project", "disable": false,
  "permissions": [ { "kind": "project", "namespace": "platform",
    "access": [ { "resource": "repository", "action": "pull" } ] } ] }
JSON

Then the scanner, where air-gapped installations most often ship a false negative. Harbor's Trivy adapter exposes `SCANNER_TRIVY_SKIP_UPDATE`, default false, to disable database downloads — but skipping the update alone is not the procedure. Harbor's harbor.yml template is specific: with skip_update you must download trivy-offline.tar.gz, extract trivy.db and metadata.json, and mount them at /home/scanner/.cache/trivy/db; with skip_java_db_update, mount trivy-java.db under /home/scanner/.cache/trivy/java-db/; and it says outright that an air gap needs offline_scan as well as skip-update.

hostname: harbor.core.internal.example

https:
  port: 443
  certificate: /etc/pki/harbor/core.crt
  private_key: /etc/pki/harbor/core.key

# ...

trivy:
  # Mount trivy.db + metadata.json under /home/scanner/.cache/trivy/db.
  skip_update: true
  # Likewise trivy-java.db under .cache/trivy/java-db/.
  skip_java_db_update: true
  # Stops Trivy reaching out mid-scan. Required alongside skip_update.
  offline_scan: true
  security_check: vuln

# ...

_version: 2.15.0

Failure Modes We Design For

• Registry unreachable. Mitigate structurally: a site-local zot, and registry storage that does not live in the cluster it serves.
• A stale offline vulnerability database. Scans pass because the data is months old and nothing alerts. Make the import a dated step in the same courier run as the images, and alert on the age of metadata.json.
• Immutability colliding with replication. Harbor documents that an immutable artifact cannot be altered by replication, and separately that a replication rule can override at the destination. Our reading: the overriding rule fails rather than silently wins. We have not seen the interaction spelled out — verify it.
• Quota exhaustion mid-push. The push fails partway and the obvious remedy — deleting something — frees nothing until GC runs. Alarm on quota headroom, not disk.
• Losing Harbor's PostgreSQL. The blobs are fine. Projects, RBAC, robot accounts, retention and replication rules, scan history are not — a first-class backup target whose restore belongs in your multi-cluster recovery drill.

Exit Ramp: The Registry You Can Walk Away From

Registries are an unusual optionality problem: the data is genuinely portable, the configuration is not. Blobs and manifests are OCI content, and skopeo sync or oras will move an estate of them between any two conformant registries.

What does not travel is everything built around them: projects and their RBAC, retention and immutability rules, robot accounts, replication rules, quota, webhooks, scan history. Express those as code now — the API script above is an exit artifact as much as a provisioning one — and keep the estimate honest: images are hours, governance days.

Rehearse it cheaply. Sync one project to a bare zot instance, repoint a non-production cluster's hosts.toml at it, and run a deployment — then you know the real cost and hold a fallback you have executed. A layer down, Harbor's storage on an S3-compatible backend makes that tier independently replaceable too.

The Long Game

One test applies to a registry decision: unplug the internet, then rebuild every cluster you operate from nothing. If that works, you own your image supply. If not, you have a hosted dependency with a local cache in front of it — a fine thing to have, and a different thing entirely.

This is a durable investment rather than a lock-in trade because the interface is a specification, not a product. Distribution v1.1.1 will still describe how a client fetches a manifest long after any particular registry has been replaced, and both projects implement it rather than extend it. Betting on the spec makes the registry the least interesting component in the platform — small, boring, replaceable — which is what infrastructure built to run for a decade looks like.