Automated Penetration Testing: What a Script Can Prove

Consider a hypothetical. A penetration test finds that a node pool's kubelets answer GET /pods to anyone who asks. The team fixes it, the tester's retest confirms the fix, and the finding closes. Months later the pool is rebuilt from an older template and the kubelets answer again. Nothing alerts, because the retest was an event, not a control.

What a Passing Automated Run Actually Asserts

Every automated check needs three inputs: a known condition (port 10250 open, a ServiceAccount able to read Secrets), a vantage point (a runner outside the perimeter, a pod in a namespace, an API reader), and an expected answer. A pass means: from here, at this time, the answer matched.

That makes a CI scan white-box by construction. Black-box testing means testing without insider knowledge; NIST SP 800-115, in its application-testing section, uses the term for techniques performed without source code knowledge. A baseline file listing which ports must stay closed is insider knowledge. Run the scan from the internet and it becomes external testing — which the same publication defines as testing from outside the organisation's security perimeter — but never black box penetration testing.

SP 800-115 dates from September 2008. It describes penetration testing as mimicking real-world attacks, and says most penetration tests look for combinations of vulnerabilities that yield more access than any single one. Network vulnerability scanning, by contrast, "generally covers surface vulnerabilities, and is unable to address the overall risk level of a scanned network", with a high false-positive rate. Penetration testing is labor-intensive and "requires great expertise to minimize the risk to targeted systems."

The UK NCSC frames it from the other side: a penetration test is a way to gain assurance in your vulnerability assessment and management processes, not a primary method for identifying vulnerabilities. The OWASP Web Security Testing Guide v4.2 names the limit plainly: automated tools "do not think creatively." Automation belongs to the process the test assures. It cannot be the test.

Your Tool List Has a Half-Life

Tools for penetration testing age faster than the findings they check. kube-hunter's README says it "is not under active development anymore" and recommends Trivy's Kubernetes misconfiguration scanning and KBOM vulnerability scanning instead. Its latest release is v0.6.8, published on 18 May 2022.

trivy k8s connects to the cluster and scans it for security issues — it reads configuration through the API rather than probing the network — and its documentation marks the feature EXPERIMENTAL. kube-hunter also offered "active hunting", in which it would exploit vulnerabilities it found to explore for further ones, whereas a normal hunt never changes the state of the cluster. That is a different vantage and a different risk class, not a newer version of the same check.

Penetration testing with Kali has the same property in miniature. Kali's official Docker images are updated once a week; kali-rolling tracks the continuously updated repository, while kali-last-release tracks the last versioned release and gets no update until the next one. As an unpinned CI base, the tool under your regression suite changes between runs, and a changed result no longer says whether the system or the scanner moved.

Reachability Regression With nmap and ndiff

The first assertion type is reachability from outside. Nmap's TCP connect scan, -sT, asks the operating system to open each connection through the connect system call; it is the default when the user lacks raw-packet privileges, so it runs in a non-root CI container. -Pn skips host discovery and scans every listed address. Ndiff then compares two Nmap XML files — host states, port states, service versions from -sV, OS matches and script output — and its exit code says whether they differ: 0 for the same, 1 for different, 2 for a runtime error.

#!/usr/bin/env bash
# reachability-regression.sh: ports closed by past findings stay closed.
# Baseline encodes PT-2026-003 (kubelet 10250), PT-2026-007 (etcd 2379-2380),
# PT-2026-009 (NodePort range on node addresses).
set -euo pipefail

NMAP_PIN="${NMAP_PIN:-7.991}"
TARGETS="${TARGETS:-targets.txt}" # one address per line, committed
BASELINE="${BASELINE:-baseline.xml}" # changes only via reviewed commit
PORTS="${PORTS:-22,80,443,2379,2380,6443,10250,10255,30000-32767}"
CONTROL_HOST="${CONTROL_HOST:-203.0.113.10}" # the ingress, must answer
CONTROL_PORT="${CONTROL_PORT:-443}"
OUT="${OUT:-run}"

have="$(nmap --version | awk '/^Nmap version/ {print $3}')"
if [[ "$have" != "$NMAP_PIN" ]]; then
  echo "nmap $have found, suite pinned to $NMAP_PIN" >&2
  exit 2
fi

mkdir -p "$OUT"
# Connect scan: no root, no ping, no DNS. TCP only.
nmap -sT -Pn -n -p "$PORTS" -iL "$TARGETS" -oX "$OUT/current.xml" \
  -oG "$OUT/current.gnmap" >/dev/null

# Positive control: a blind runner fails as broken, not as a pass.
if ! grep "^Host: ${CONTROL_HOST} " "$OUT/current.gnmap" |
  grep -q "[[:space:]]${CONTROL_PORT}/open/tcp/"; then
  echo "BROKEN: control ${CONTROL_HOST}:${CONTROL_PORT} not open from this vantage" >&2
  exit 2
fi

rc=0
ndiff "$BASELINE" "$OUT/current.xml" >"$OUT/ndiff.txt" || rc=$?
case "$rc" in
  0) echo "PASS: host and port states match $BASELINE" ;;
  1)
    echo "REGRESSION: reachability differs from $BASELINE" >&2
    cat "$OUT/ndiff.txt" >&2
    exit 1
    ;;
  *)
    echo "ERROR: ndiff exited $rc" >&2
    exit 2
    ;;
esac

The positive control is the load-bearing line. A runner with no route to the targets sees every port as unreachable; if the baseline was recorded the same way, ndiff reports no difference and the job goes green. Requiring one port that must be open — the ingress — turns that silence into a failure. Pin the scanner too: the current stable Nmap is 7.991, released on 6 August 2026, and Ndiff moved to Python 3 in Nmap 7.94, which matters on older base images.

As written it is TCP only. Without -sV, ndiff compares host and port state, so a different daemon on an already-open port is invisible. For the in-cluster equivalent — which pod can reach which — the continuous segmentation prober in the network policy audit is the right tool; this script watches the perimeter.

RBAC Regression Without Handing CI the Impersonate Verb

The obvious RBAC check is kubectl auth can-i --as=…, and the kubectl reference notes that --as requires the caller to be allowed to use impersonation. Kubernetes' RBAC good-practices page says the impersonate verb "allows users to impersonate and gain the rights of other users in the cluster." A CI job that tests permissions this way can exercise every permission it tests.

A SubjectAccessReview asks the same question without acting as anyone: it is an access review "for any user, not only the current one", and the runner needs only create on subjectaccessreviews:

# rbac-regression-identity.yaml: asks "can X do Y?"; cannot act as X.
apiVersion: v1
kind: Namespace
metadata:
  name: security-regression
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rbac-regression
  namespace: security-regression
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rbac-regression-reviewer
rules:
  - apiGroups: ["authorization.k8s.io"]
    resources: ["subjectaccessreviews"]
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: rbac-regression-reviewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rbac-regression-reviewer
subjects:
  - kind: ServiceAccount
    name: rbac-regression
    namespace: security-regression

That role is narrow, not inert: its holder can ask what any identity in the cluster may do, which is reconnaissance an attacker would value. Treat the token as sensitive.

The trap is groups. The API reference describes the groups field as "the groups you're testing for": a review that names a user but no groups asks what that user could do if it belonged to no groups at all. Real ServiceAccounts authenticate as system:serviceaccount:<namespace>:<name> and are assigned system:serviceaccounts and system:serviceaccounts:<namespace>; a successful authentication also adds system:authenticated. Any permission granted to one of those groups is invisible to a review that leaves them out — and a deny row then passes for the wrong reason.

#!/usr/bin/env bash
# rbac-regression.sh: one SubjectAccessReview per row; control rows stay allowed.
set -euo pipefail

fail=0
while IFS=$'\t' read -r finding ns sa verb resource sub target expect; do
  [[ -z "$finding" || "$finding" == \#* ]] && continue
  group=""
  if [[ "$resource" == *.* ]]; then # e.g. subjectaccessreviews.authorization.k8s.io
    group="${resource#*.}"
    resource="${resource%%.*}"
  fi
  [[ "$sub" == "-" ]] && sub=""
  [[ "$target" == "-" ]] && target="" # cluster-scoped
  # Authentication adds these groups; a review does not.
  allowed="$(kubectl create --validate=false -o jsonpath='{.status.allowed}' -f - <<SAR
apiVersion: authorization.k8s.io/v1
kind: SubjectAccessReview
spec:
  user: "system:serviceaccount:${ns}:${sa}"
  groups:
    - "system:serviceaccounts"
    - "system:serviceaccounts:${ns}"
    - "system:authenticated"
  resourceAttributes:
    group: "${group}"
    resource: "${resource}"
    subresource: "${sub}"
    verb: "${verb}"
    namespace: "${target}"
SAR
)"
  got="deny"
  [[ "$allowed" == "true" ]] && got="allow"
  status="PASS"
  [[ "$got" == "$expect" ]] || { status="REGRESSION"; fail=1; }
  printf '%-10s  %s  %s:%s %s %s%s  expect=%s got=%s\n' "$status" "$finding" \
    "$ns" "$sa" "$verb" "$resource" "${sub:+/$sub}" "$expect" "$got"
done <<'TSV'
# finding	sa_namespace	sa_name	verb	resource[.group]	subresource	target_ns	expect
PT-2026-014	monitoring	node-exporter	get	nodes	proxy	-	deny
PT-2026-014-ctl	monitoring	node-exporter	get	nodes	metrics	-	allow
PT-2026-021	ci	deployer	create	pods	exec	payments	deny
PT-2026-022	ci	deployer	impersonate	serviceaccounts	-	-	deny
TSV
exit "$fail"

The first row carries the finding worth encoding. The good-practices page says access to the nodes/proxy subresource grants rights to the kubelet API that allow command execution in every pod on the node, and that this access bypasses audit logging and admission control: get on nodes/proxy "is not a read-only permission." If a metrics exporter once held it, the suite should re-prove on every deploy that it no longer does, whether the grant returns directly or through a group.

One Finding, One Template

Exposure findings — an endpoint that answered when it should not have — fit nuclei's format. A template is YAML: a unique ID that contains no spaces, and an info block whose metadata is where the original finding ID belongs, so every match names the finding it reopens.

By the kubelet's flag defaults, a request to its HTTPS endpoint that no other authentication method rejects is treated as system:anonymous, and the default authorization mode is AlwaysAllow; starting the kubelet with --anonymous-auth=false makes unauthenticated requests receive 401 Unauthorized. Those are the kubelet's own defaults, not a claim about how any distribution ships it. The template encodes that one past finding: a 200 carrying a PodList is a regression; a 401 or 403 is the fixed state.

id: kubelet-anonymous-pods

info:
  name: Kubelet lists pods to an unauthenticated caller
  author: platform-security
  severity: high
  description: |
    Regression check for PT-2026-003: an unauthenticated GET /pods returned
    the node's PodList. The fixed state, 401 or 403, does not match.
  reference:
    - https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/
  tags: kubernetes,kubelet,regression
  metadata:
    finding-id: PT-2026-003
    finding-source: human-led external test

http:
  - method: GET
    path:
      - "{{BaseURL}}/pods"
    matchers-condition: and
    matchers:
      - type: status
        status:
          - 200
      - type: word
        part: body
        words:
          - "PodList"

An empty result has two causes: every kubelet refused, or the scan never reached one. So, as with nmap, the gate needs a positive control — a second template, kept in controls/, matching any 200, 401 or 403 on the same path. It proves the request arrived, whatever the answer:

id: kubelet-answers-control

info:
  name: Kubelet answers over HTTPS from this vantage
  author: platform-security
  severity: info
  description: Positive control. Any 200, 401 or 403 proves the scan arrived.
  tags: kubernetes,kubelet,control

http:
  - method: GET
    path:
      - "{{BaseURL}}/pods"
    matchers:
      - type: status
        status:
          - 200
          - 401
          - 403

Run both directories with a pinned binary — v3.11.1 here; -duc disables the automatic nuclei and template update check. Targets are kubelet HTTPS URLs, https://<node>:10250, never a bare host:port. Nuclei's documentation does not specify an exit code for matches, so the gate reads the output file, and counts an empty one as a pass only after the control has matched every target:

#!/usr/bin/env bash
# exposure-regression.sh: every template is one closed finding; any match is a regression.
set -euo pipefail

TARGETS="${TARGETS:-exposure-targets.txt}" # https://<node>:10250, one per line
CONTROL="kubelet-answers-control"           # in controls/, matches 200/401/403

nuclei -version 2>&1 | grep -q 'Engine Version: v3.11.1' || {
  echo "nuclei is not v3.11.1" >&2
  exit 2
}
if grep -vq '^https://' "$TARGETS"; then
  echo "BROKEN: $TARGETS lines must be https:// URLs" >&2
  exit 2
fi
: >findings.jsonl
# -or/-ot: keep raw responses (a real PodList carries env vars) out of the record.
nuclei -duc -silent -or -ot -t templates/ -t controls/ -l "$TARGETS" \
  -jsonl -o findings.jsonl </dev/null >/dev/null || { # targets from -l, never stdin
  echo "ERROR: nuclei exited $?" >&2
  exit 2
}

# Positive control: every target must have answered.
missing="$(jq -r --arg id "$CONTROL" 'select(."template-id" == $id) | ."matched-at"' \
  findings.jsonl | sed 's|/pods$||' | grep -vxF -f - "$TARGETS" || true)"
if [[ -n "$missing" ]]; then
  printf 'BROKEN: no HTTP answer from:\n%s\n' "$missing" >&2
  exit 2
fi

hits="$(jq -r --arg id "$CONTROL" 'select(."template-id" != $id) |
  [.info.metadata."finding-id", ."template-id", ."matched-at"] | @tsv' findings.jsonl)"
if [[ -n "$hits" ]]; then
  printf 'REGRESSION:\n%s\n' "$hits" >&2
  exit 1
fi
echo "PASS: every target answered; no template in templates/ matched"

Detection can regress too. Falco's event-generator has a test command that runs suspect actions against a running Falco; its README warns that some actions modify files under /bin and /etc, and recommends Docker. That makes kernel-time detection an assertion rather than a hope. MITRE Caldera goes further, into automated adversary emulation — and its own README says it is built to "assist manual red-teams", not replace them.

Click to zoom

Where the Script Stops

Chaining is the first boundary. The CVSS v4.0 User Guide says CVSS is designed to rate individual vulnerabilities; its treatment of chained vulnerabilities "is not a formal metric" but guidance, and identifying which vulnerabilities combine is "the responsibility of the analyst."

An illustrative chain, not drawn from any engagement: a metrics exporter's ServiceAccount holds get on nodes/proxy, and the exporter has a remote-code-execution bug. Alone, each looks minor. Together they are command execution in every pod on the node, outside the audit log. A script can assert either half forever and never see the whole, because seeing the whole is discovery.

When the kubelet delegates authorization to the API server with --authorization-mode=Webhook, its default path table maps /metrics/* to nodes/metrics and /stats/* to nodes/stats, and everything it does not list to nodes/proxy. An exporter that only reads metrics and stats can be granted those two subresources instead — exactly what the control row above asserts.

Intent is the second boundary. Whether an unusual binding is a deliberate exception or a mistake is a decision, and Kubernetes' guidance is that it is "vital to periodically review" RBAC for redundant entries and possible privilege escalations — a human act. What automation can do is enforce the decision once made. Keep accepted exceptions in a register with an owner, a rationale and an expiry, and fail the suite when one lapses:

#!/usr/bin/env bash
# exceptions-check.sh: an accepted exception is a decision with a shelf life.
set -euo pipefail
# exceptions.tsv columns: finding_id, owner, expires (YYYY-MM-DD), rationale
awk -F'\t' -v today="$(date -u +%F)" '
  /^#/ || NF == 0 { next }
  $3 < today { printf "EXPIRED %s (owner %s, %s): re-decide\n", $1, $2, $3; bad = 1 }
  END { exit bad }
' exceptions.tsv

Reading a vulnerability assessment and penetration testing report covers the human-led test and its retest; this suite starts where that retest ends.

Click to zoom

Reporting an Automated Run Honestly

Penetration testing reporting for an automated run is short. A record that survives an auditor carries:

• tool versions, and the commit of the templates, baseline and expectations table;
• the vantage point — which runner, from which network;
• the target list, as committed;
• each assertion ID mapped to its finding ID, with its result;
• the timestamp of the run.

Phrase the result as what it is: "these N known conditions were absent from this vantage at time T." Never "no vulnerabilities found" — the run did not look for any it did not already know. A regression reopens the original finding in the register rather than opening a new ticket, so its history stays in one place; running the vulnerability management life cycle yourself covers that register.

Failure Modes That Make the Suite Lie

• A runner with no route. Everything looks closed. Control: a must-be-open port, a must-answer template.
• A baseline refreshed to make CI green. The regression becomes normal. Control: reviewed commits only.
• A review without groups. Group grants vanish. Control: pass all three groups; keep the allow row.
• A self-updating template set. Yesterday's suite is gone. Control: -duc and a pinned directory in git.
• Silent scope gaps. UDP and IPv6 are out of scope as written; say so in the run record.
• Testing what you are not authorised to test. kube-hunter's README says not to run it on a cluster you do not own; that holds for every tool here. Check your hosting provider's terms before scanning from outside.
• Production side effects. A nightly scan can trip rate limits or intrusion detection. Tell whoever watches those systems.

Exit Ramps and the Long Game

The durable asset is the assertion set — a TSV of expectations, YAML templates and XML baselines in git — not the tools that read it. Nuclei is MIT-licensed, Trivy and event-generator are Apache-2.0, and any of them can be swapped without losing a single finding. Nmap ships under the Nmap Public Source License, which aims to "prohibit redistribution and use of Nmap within proprietary hardware and software products" and funds the project through an OEM licence. That matters only if the suite ever becomes part of something you sell; this is not legal advice.

Regression detects what admission policy with Kyverno failed to prevent, and both run best on CI you operate yourself. Over a decade, each periodic human test adds findings and each finding adds an assertion. The suite becomes the institutional memory of every mistake the organisation has already paid to find — and the tester's time goes to what is new.