Air-Gapped Patching with Katello: Mirrors and Evidence

Search for air-gapped patching and you find transport: how to export, what to put on the media, which command ingests it. All documented, none of it the part that fails an audit. That question is narrower: on a named host, on a named date, was a named security fix installed — and what artifact, made on the isolated side, says so?

The half of air-gapped patching nobody writes down

A patch programme has two halves. Supply: getting signed content from a vendor archive into a repository your isolated hosts can reach. Proof: turning the state of those hosts into a record someone outside the team can check. Connected estates get proof almost free: the tooling that supplies packages also reports on them. Sever the network and supply survives as a courier problem; proof does not, because the network was the mechanism that produced it.

It bites hardest where air gaps are chosen: an estate isolated for export control or classified work carries the heaviest evidence obligations, and isolation removes the easiest ways to meet them. The mirror is necessary, not sufficient.

What the patch SLA actually obliges you to prove

Start with the control text; it is more specific than folklore. NIST SP 800-53r5 control SI-2 sets no patch deadline. It requires you to install security-relevant updates "within [Assignment: organization-defined time period] of the release of the updates" — the number is yours. Its discussion is explicit that the period varies with the system's security category, the update's criticality, risk tolerance, mission and threat environment. A universal thirty days is a habit, not a control.

What it does demand is a determination. SI-2(2) asks you to "[d]etermine if system components have applicable security-relevant software and firmware updates installed" using automated mechanisms at a defined frequency. That is why an air-gapped estate needs more than a mirror: a mirror moves updates and determines nothing about a host.

RA-5 adds the other half. Remediation timing is again organization-defined, but RA-5(f) requires "vulnerability monitoring tools that include the capability to readily update the vulnerabilities to be scanned", and RA-5(2) requires that set be updated on a defined frequency, before a new scan, or when new vulnerabilities are reported. Behind a gap that capability is a courier and a schedule, and your scan data's age becomes an auditable fact.

NIST SP 800-40r4 supplies the framing: patching is "preventive maintenance for computing technologies — a cost of doing business", not a discretionary security activity. It recommends phased deployment, a small subset of assets taking a patch first as a canary, and it is unsentimental about exceptions — an asset that cannot or should not be patched means "there is one less option" among your risk responses. Four artifacts fall out, and a defensible claim needs every one: the advisory's identity, its issue date, the host's install date, and the scope of hosts covered.

The two-server architecture across the gap

Two Foreman/Katello servers: one connected, synchronising from vendor archives; one isolated, serving your hosts. Content moves one way, as export archives on removable media. Nothing else crosses.

The decision inside that shape is the export format, and it is the hardest to reverse. Foreman produces an importable export by default; --format=syncable is opt-in. Syncable carries custom file type, Kickstart and Yum content — the guide is unambiguous about the rest: "You cannot export Ansible collections, Deb content, or Docker content in the syncable format." Only the default importable export lists Deb.

So the format choice is an operating-system choice. On an Enterprise Linux estate syncable is attractive because it is boring: a plain repository tree any web server can publish. The moment a Debian or Ubuntu host is in scope, the plain complete/incremental library export is the procedure. Pick before you write the runbook.

Click to zoom

Configuring the export so it can actually be exported

Five things stop most first attempts, all prerequisites rather than bugs. The download policy is the most confusing: nothing looks wrong until the export produces almost nothing. Under On Demand, Foreman "downloads only the metadata during synchronization", fetching packages later when clients ask — nothing is on disk to export. The prerequisite: policy Immediate for every repository in the Library environment you export.

Second, exporting is its own privilege: the exporting user holds the Content Exporter role, the importing user the Content Importer role. Third, both ends must run the same major Foreman version — the guide requires it for synchronising content by export and import, and a mixed pair fails in a way no other prerequisite explains. Fourth, the import path must be absolute; the guide says relative paths do not work. Fifth is the format trap above.

#!/usr/bin/env bash
# Connected Foreman/Katello server.
set -euo pipefail

ORG="Example_Org"
DEST="dc2-disconnected"

# These defaults reach NEW repositories only, so retrofit every existing
# one -- not just the repository you tested the export with.
hammer settings set --name default_download_policy --value immediate
hammer settings set --name default_redhat_download_policy --value immediate
hammer --no-headers --csv repository list --organization "$ORG" --fields id |
  while read -r id; do
    hammer repository update --id "$id" --download-policy immediate
  done

# First export: complete, default importable.
hammer content-export complete library \
  --organization "$ORG" --destination-server "$DEST" --chunk-size-gb=2

# Every export after the first.
hammer content-export incremental library \
  --organization "$ORG" --destination-server "$DEST" --chunk-size-gb=2

#!/usr/bin/env bash
# Isolated Foreman/Katello server, once the media is mounted.
set -euo pipefail

ORG="Example_Org"
MEDIA="/mnt/transfer"
STAMP="2026-08-31T09-00-00-00-00"
LANDING="/var/lib/pulp/imports/$STAMP"

# Katello does not sign the archive; this checksum is the operator's
# own, carried apart from the media.
sha256sum -c "$MEDIA/export-$STAMP.sha256"

# All three or the import fails: archive, toc.json, metadata.json.
mkdir -p "$LANDING"
cp "$MEDIA"/*.tar.gz "$MEDIA"/*toc.json "$MEDIA"/*metadata.json "$LANDING/"

# Pulp must own what it reads.
chown -R pulp:pulp "$LANDING"

# Absolute. Relative paths do not work.
hammer content-import library --organization "$ORG" --path="$LANDING"

# Syncable variant only: source directory must be in
# ALLOWED_IMPORT_PATHS in /etc/pulp/settings.py.

Keeping signature verification intact when the wire is a USB stick

Mirroring changes who serves your packages. It does not change who signed them: Pulp stores the vendor's RPMs as received and does not resign them, so that signature crosses the gap intact. Repository metadata is the artefact that changes hands, and treating the two as one chain is the fastest route to a client with verification switched off entirely.

DNF's own documentation gives the upstream default for gpgcheck, repo_gpgcheck and localpkg_gpgcheck alike as False. Distributions ship a /etc/dnf/dnf.conf and vendor repository files that set them, which is why the defaults rarely bite — a repository file you author for the mirror inherits none of that.

Which you can honestly switch on depends on who signed what. gpgcheck verifies the package, which Pulp never touched, so the vendor release key stays right behind the gap. repo_gpgcheck verifies a signature over repomd.xml — a different artefact, a separate key store — and Katello republishes repodata on every content view publish, so the vendor never signed the metadata your clients fetch. Pulp can sign its own output, but conditionally: metadata signing "is enabled for all repositories that have a signing service associated with them", and clients then verify against the key Pulp publishes beside the repository. Aim repo_gpgcheck=1 at a Pulp URL while gpgkey names the vendor key and dnf refuses the repository — the moment a tired operator turns checking off entirely.

# /etc/yum.repos.d/example-disconnected.repo

[example-el9-baseos]
name=Example EL9 BaseOS (disconnected Katello mirror)
baseurl=https://katello.dc2.example.internal/pulp/content/Example_Org/production/EL9_Base/custom/example/el9-baseos/
enabled=1

# Packages. Upstream default False. Pulp does not resign RPMs, so the
# vendor release key still verifies them.
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-example-release

# METADATA has a different signer. Katello republishes repodata on every
# content view publish, so the key above cannot verify repomd.xml --
# enabling this against it breaks the repository.
repo_gpgcheck=0

# To verify metadata too, configure a Pulp signing service and add the
# key it publishes; gpgkey takes a list:
#
#   repo_gpgcheck=1
#   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-example-release file:///etc/pki/rpm-gpg/RPM-GPG-KEY-pulp-metadata

# localpkg_gpgcheck is a [main] option, for /etc/dnf/dnf.conf. It governs
# the path nobody writes a runbook for: a hand-carried RPM.
#
#   [main]
#   localpkg_gpgcheck=1

APT inverts that balance. Its chain runs from per-package checksums in the Packages file, to the Release file carrying those checksums, to a signature over Release by the archive key: apt-secure "does not review signatures at a package level". All of APT's trust lives in the metadata signature — the artefact Katello reissues.

Pin every source with Signed-By, which requires a repository to verify against a named set of keys rather than every key APT trusts. Whose key belongs in that set is decided by what serves the Release file. A static vendor-format tree still serves the vendor's signed Release, so the upstream archive key is right there — the exit-ramp case below. A Katello-published Deb repository is not that case: Pulp recreates APT metadata when content management operations run, and a Foreman community thread describes Katello signing it with its own key — so those hosts pin the Deb signing key from your own instance. Reaching for the upstream key is how a disconnected host reports a Release file it cannot verify — the discipline of self-built base images and signed container content, one layer down.

Content promotion is your patch ring

Most estates implement 800-40r4's phased deployment as a spreadsheet of maintenance windows. Katello lets you implement it as content: a lifecycle environment path, filters controlling what a content view publishes, promotion between them. The ring becomes a property of the content server, not a convention people follow.

One flag deserves care, because naming the right date field is this article's argument. The documented CLI example passes --date-type updated, which tests the erratum's updated date — not the issued date your SLA clock runs from. Nor are those the only two: dnf5 advisory list prints an Issued column while its --json output carries buildtime, and nothing states they are the same instant. Name the field your clock uses, and check hammer content-view filter rule create --help for the values your version accepts.

#!/usr/bin/env bash
# Isolated server. The canary ring, as content.
set -euo pipefail

ORG="Example_Org"
CV="EL9_Base"

# Library -> canary -> production.
hammer lifecycle-environment create --organization "$ORG" \
  --name "canary" --prior "Library"
hammer lifecycle-environment create --organization "$ORG" \
  --name "production" --prior "canary"

# EXCLUDE filter. Security errata are absent from --types below, so they
# keep flowing while bugfix and enhancement churn freezes.
hammer content-view filter create --organization "$ORG" \
  --content-view "$CV" --name "Errata Filter" \
  --type erratum --inclusion false

# --date-type updated tests the erratum's UPDATED date, not the issued
# date the SLA clock runs from. Choose the field on purpose.
hammer content-view filter rule create --organization "$ORG" \
  --content-view "$CV" --content-view-filter "Errata Filter" \
  --start-date "2026-08-01" --types enhancement,bugfix --date-type updated

# Publishing makes an applicable erratum installable.
hammer content-view publish --organization "$ORG" --name "$CV"

# Canary first; production only after it soaks.
hammer content-view version promote --organization "$ORG" \
  --content-view "$CV" --version 1 --to-lifecycle-environment "canary"

Producing evidence when the host cannot phone home

Katello's erratum states are the most useful thing it gives an auditor, and the most commonly collapsed into one number. An applicable erratum "has updates for packages present on the host" but is not ready to install. An installable one is "available to a host from its associated content view environments, but [is] not yet installed", and is always a subset of applicable.

So there are two gaps, with different owners. Applicable-minus-installable is a content-server problem: a content view not published or promoted, or an errata filter excluding the fix — Foreman is explicit that you publish incremental content views to make applicable errata installable. Installable-minus-installed is a host problem. Report them as one "outstanding" column and you have hidden which half of the organisation owes the work.

That model is RPM-side. Table 25 says the RPM filter "filters non-modular RPM packages and errata" while the Deb filter matches "packages based on their name", and the errata-import paragraph names Red Hat Enterprise Linux, openSUSE/SUSE Linux Enterprise Server, AlmaLinux and Rocky Linux. Section 15.11 sends Deb readers wanting errata to the RPM procedure — a fork in the documentation, not a denial that Debian has advisories. The operational result is the same: Debian and Ubuntu hosts behind the mirror get packages, not advisories.

Their evidence comes from OVAL, from each distribution's own feed. Canonical publishes per-release USN data as com.ubuntu.<release>.usn.oval.xml.bz2; Debian publishes its OVAL separately, one oval-definitions-<codename>.xml.bz2 per release at www.debian.org/security/oval/. Two publishers, two index pages — not one URL with the codename swapped. Both carry a per-file modification column and both move constantly, so the feed's own date belongs in the evidence record. Foreman's custom file type content is documented as covering OVAL files, so the isolated server can serve the carried feed as a file repository — transport for the data, not errata support for Deb.

Click to zoom

#!/usr/bin/env bash
# Per-host patch evidence, collected on the isolated side.
set -euo pipefail

. /etc/os-release
OUT="/var/lib/patch-evidence/$(hostname -f)/$(date -u +%Y%m%dT%H%M%SZ)"
# FEEDS holds DECOMPRESSED OVAL XML. Both publishers ship .xml.bz2, so
# staging runs once per courier cycle:
#   cp -p /media/courier/oval/*.xml.bz2 "$FEEDS"/   # -p keeps publisher mtime
#   bunzip2 --keep --force "$FEEDS"/*.xml.bz2
FEEDS="/var/lib/patch-evidence/oval"
mkdir -p "$OUT"

case "$ID" in
  rhel | almalinux | rocky | centos | ol | fedora)
    # Pick the tool BEFORE redirecting: "cmd > f || fallback" truncates f
    # first, so a missing dnf5 leaves empty JSON reading as zero advisories.
    # --json: name, type, severity, nevra, buildtime (UNIX time).
    if command -v dnf5 > /dev/null; then
      dnf5 advisory list --json > "$OUT/advisories-available.json"
      dnf5 advisory list --installed --json > "$OUT/advisories-applied.json"
    else
      dnf updateinfo -v list > "$OUT/advisories-available.txt"
      dnf updateinfo -v list --installed > "$OUT/advisories-applied.txt"
    fi
    rpm -qa --qf '%{NAME} %{VERSION}-%{RELEASE} %{INSTALLTIME}\n' \
      > "$OUT/packages.txt"
    # rpm -Va exits non-zero on any difference, which is a finding.
    rpm -Va > "$OUT/file-integrity.txt" || true
    echo "local rpm database" > "$OUT/data-source.txt"
    ;;
  ubuntu | debian)
    if [ "$ID" = "ubuntu" ]; then
      feed="$FEEDS/com.ubuntu.$VERSION_CODENAME.usn.oval.xml"
    else
      feed="$FEEDS/oval-definitions-$VERSION_CODENAME.xml"
    fi
    # A missing or still-compressed feed is a collection failure, not a
    # clean scan. Fail loudly, do not record an uninformed result.
    [ -s "$feed" ] || { echo "no decompressed OVAL feed at $feed" >&2; exit 1; }
    # oscap oval eval returns 0 on success, 1 on error -- unlike xccdf
    # eval, no "findings" exit code to swallow. Confirm on your build:
    # oscap(8)'s general EXIT STATUS also documents 2 for incompliance.
    oscap oval eval --results "$OUT/oval-results.xml" \
      --report "$OUT/oval-report.html" "$feed"
    dpkg-query -W --showformat='${Package} ${Version} ${Status}\n' \
      > "$OUT/packages.txt"
    # RA-5(f): the data's age is part of the result. Publisher mtime,
    # and only because staging used cp -p.
    date -u -r "$feed" +%Y-%m-%dT%H:%M:%SZ > "$OUT/data-source.txt"
    ;;
  *)
    # Without this arm an unmatched ID collects nothing and exits 0.
    echo "unsupported os-release ID=$ID" >&2
    exit 1
    ;;
esac

Failure modes that only show up after the gap

• Ubuntu phased updates quietly withhold content the mirror already holds. Foreman is blunt: "Phased updates obstruct Foreman from managing content on hosts", and with them enabled it "cannot ensure updates and errata are deployed as intended" to Ubuntu hosts. Remedy: set APT::Get::Always-Include-Phased-Updates "true" via a provisioning template, or apply the Disable Phased Updates - SSH Default template. Signal: hosts behind on a package the repository serves, with nothing in Katello's content state to explain it.
• A stale OVAL feed produces a clean scan that means nothing. The scanner reports against the data it has, which behind a gap is as old as the last courier run — the point of RA-5(f). Signal: no feed date in the evidence record, so nobody can tell a clean result from an uninformed one.
• A hand-carried RPM bypasses the mirror and its verification. When the courier cycle is slow someone carries an urgent package by hand, and localpkg_gpgcheck defaults to False upstream. Signal: packages on hosts with no matching content in any content view version.

Exit ramps and the ten-year view

Air gaps outlive tools. The estate will still be isolated long after Foreman's version numbering has moved on, so the honest test is what survives Katello's removal.

For Yum, file and Kickstart content the syncable export is already the exit: a plain repository tree any web server can publish, no Katello process required. For Deb content — which that format cannot carry — the exit is the mirrored archive and its signed Release file, back under the vendor's own key. An apt client pointed at a static copy behaves identically. The OVAL feeds are public files you hold; the evidence records are your own JSON and XML. What you lose is the promotion machinery and the errata state model — name that as the thing you are buying.

The durable asset is the evidence pipeline, not the mirror. Mirrors are replaceable; a per-host record naming an advisory, a date, an install time and the data it was judged against is what an auditor will still recognise in a decade, and what lets you map controls onto infrastructure you own rather than onto a vendor's attestation. It rests on two deliberate things: trustworthy time on every host, because an install date from a drifting clock is not evidence, and a feed-freshness field. Get those right and the OS package layer stops being the part of the disconnected estate nobody can speak for — as container images cross the same gap above it.