Catalogue Audit: 49 Articles, 342 Defects — What We Found

On 2026-07-29 we rebuilt the daily blog pipeline. The previous path was a single model call that wrote the article and then audited itself in the same context. The new path is plan, independent plan audit, write, deterministic verifier, independent article review. The first article under that regime was held twice rather than published — once because the review found four blocking defects, once because CI was red.

That raised an obvious question about the 49 articles already live. They were written with no plan, no sourced claims, a self-audit inside the author's own context, and no verifier. So we audited the whole catalogue.

What the audit was

Stage 1 was mechanical. A deterministic verifier ran every parseable snippet and every real citation across all 49 posts. Result: zero blocking defects. The catalogue looked clean.

Stage 2 was judgment. A separate model, with no plan and no author context, read each article as a reader would and asked one question the mechanical sweep cannot answer: does this command actually do what the prose says it does? That produced 321 findings — 205 wave-1 (paste-breaking or load-bearing wrong), 116 wave-2.

Stage 3 was independent verification. A second model, on a separate stack, had to try to refute each finding against primary documentation and fail. It confirmed 313, refuted 8 (refute rate 2.5%), and challenged the auditor's framing on 177 of the confirmed findings. Severity was overridden 13 times. The verifier also found 29 defects the auditor had missed. That is 342 items on the fix list. Articles with at least one wave-1 finding: 47 of 49. Articles clean: 0.

Those numbers are not a scorecard. They are a failure inventory.

Defect classes

• other-technical — 88
• contradicts-itself — 58
• regulation-detail-wrong — 50
• version-or-release-wrong — 46
• default-stated-wrong — 36
• unsourced-claim — 19
• flag-or-command-does-not-exist — 16

The worst single article had 16 wave-1 findings (18 total). The densest by total findings had 21. Density is not the point. Class is.

What wave-1 looked like in practice

Five defects, each checked against the audit artifacts and primary sources.

kube-apiserver audit policy logging Secrets at RequestResponse. In the CERT-In Directions article, the first rule logged secrets and configmaps at full request/response body level, then shipped that stream to an in-India, object-locked, 180-day WORM store. Every get/list/watch of a Secret therefore wrote credential material into an immutable regulator-facing archive. Upstream Kubernetes' own example policy logs those resources at Metadata for exactly this reason. We published the opposite as a compliance recommendation.

Chrony NTP hostnames that do not resolve. Same article, section on time-sync as a compliance primitive. The config pointed at samay1.nplindia.org and samay2.nplindia.org — NXDOMAIN on the day of the audit. CERT-In FAQ Q43 names samay1.nic.in, samay2.nic.in, and time.nplindia.org. A reader pasting our block got two dead sources and no line traceable to NPL.

OpenSearch ISM rollover that never fires. Same article again. The hot state ran a rollover action without plugins.index_state_management.rollover_alias and without the index name form ^.*-\d+$ that rollover requires. On a real cluster the action fails, the managed index stops, and hot → warm → delete never runs. The caption sold a 200-day retention window the policy does not deliver.

Vault Transit rotate presented as crypto-shredding. In the DPDP Act residency article, prose claimed that rotating the KEK makes prior offshore ciphertext permanently unreadable. Vault Transit rotation adds a new key version; old ciphertext stays decryptable via the keyring until you delete the key (after deletion_allowed=true) or raise min_decryption_version. A reader who "shredded" by rotating had only rolled forward.

A restore drill that passes bash -n and shellcheck and still cannot emit its own evidence. The first held article under the new pipeline included a restore-drill script that was syntactically clean and shellcheck-clean, yet set -euo pipefail aborted before the audit artifact the prose promised was written. The same failure class is in a published post: the disaster-recovery restore drill uses velero backup get -o name, which is not a valid output format (only table/json/yaml), so the command substitution dies under errexit and the drill never restores or records a result. Mechanical syntax gates passed. Runtime behaviour did not match the claim.

These are not style nits. They are the difference between a reader building a real control and a reader building a control that looks right and fails closed, fails open, or fails silently.

The audit tooling was wrong too

Saying this is the point, not a footnote.

On the same day the catalogue work started, tooling built for this site reported:

• 96 diagram contrast failures — all of them <mask> stencil geometry, not real contrast defects
• a text-fit heuristic with 117 candidates against a true 108
• a citation checker with 11 dead links, all 11 false (it harvested configuration URLs, OIDC issuers, and ${SHELL_VARIABLE} placeholders as if they were references)

An auditor that cries wolf across 49 articles is worse than no auditor: the findings stop being read. That is why nothing was fixed on one model's word. A second, independent model had to attempt refutation against primary sources. It refuted 8 findings. It left 313 standing. It challenged how 177 of those were framed — not because the defects were imaginary, but because the write-up overstated, understated, or misplaced the load-bearing detail.

The same class of system that wrote the bad articles also produced the audit. Independence and primary-source checks are mitigations, not proof of omniscience. Eight confirmed false findings remain eight false findings.

Fixes that introduced new errors

When fixes began, independent review of the fix branches did not rubber-stamp them. Of 25 articles that reached independent fix review: 12 merge, 13 fix-first, 0 reject. Roughly half failed on first pass. Many failed because the fix introduced a new error. Review recorded 25 blocking problems across those runs.

Two examples, both confirmed on the fix branches:

• Trustee KBS Rego. The rewrite fixed parse and input paths, then renamed the decision rule to allowed. Trustee evaluates data.policy.allow. Missing allow logs a warning and treats the result as false — permanent deny. The original broken snippet at least named the rule correctly; the "fixed" one parsed and never released secrets.
• Hash-chain integrity query. A HIPAA audit-log fix used id > 1 as a genesis test. After a DELETE, Postgres BIGSERIAL does not restart. A clean chain whose first live id is not 1 is reported broken. Operators learn to ignore the nightly pass — the failure mode the original defect warned against.

The pattern matches the original writing failure: a model that "fixed" something in its own context, without a hostile second pass against primary behaviour, can replace one wrong claim with another that looks more polished.

What the pipeline does now

New articles do not publish unless they clear:

1. Plan — structure and claims before prose
2. Independent plan audit — a second model attacks the plan
3. Write — constrained to the audited plan
4. Deterministic verifier — parse, citations, diagram gates (the citation harvester no longer treats config URLs and placeholders as references; that was one of the false-positive paths)
5. Independent article review — a second model tries to break the article against sources

A quality hold publishes nothing rather than publish something wrong. The first article under this regime was held. That is a hold doing its job. It does not unpublish the 49.

This process rejects a class of defects that previously shipped. It does not guarantee the absence of every defect a hostile reader can find.

What remains unfixed or uncertain

State this plainly.

• Zero of 49 articles audited clean. The back catalogue was not a good corpus with a few bad posts in it. 47 of 49 carried at least one wave-1 defect.
• 342 defects reached the fix queue — 313 auditor findings that survived adversarial verification, plus 29 the verifier found that the auditor had missed entirely.
• All 49 articles are corrected and live. Each passed an independent reviewer that never saw the fixer's reasoning, then seven CI checks. The last three took the most rounds, and for the same reason: the prose had been corrected while a *diagram* still taught the old, wrong thing. One of them — the Gaia-X post — went through five rounds because every attempt to replace a false project-activity figure produced a different wrong figure. It was resolved by deleting the claim rather than sourcing it again: the argument never needed a commit count, and a number a reader cannot verify is decoration, not evidence.
• About half the first-pass fixes failed independent review, most often because the fix was applied to only part of a multi-part defect, or because the fix introduced a *new* error. Those went back for repair, some four times. A fix is not a closed incident.
• Eight findings were wrong. A 2.5% refutation rate is low, not zero. Framing was challenged on 177 of the confirmed findings and severity was overridden 13 times. Treat the inventory as a work queue that survived one adversarial pass, not as revealed truth.
• Mechanical cleanliness is not correctness. Stage 1 found zero blocking defects across all 49 articles. Stage 2 found 321. Anything in a language no parser handles — SQL, HCL, Rego, ini, toml, nginx, PromQL — and anything that depends on whether a command does what the prose *says* it does, still rests on judgment checked against primary sources.
• This is not a clean bill of health. The corrections are recent and were produced by the same class of system that produced the defects. If you are about to paste one of our blocks into a production terminal, check it against upstream documentation. That is the standard we should have applied before publishing it the first time.

We are not asking you to trust the remaining posts. We are stating what we measured, what concrete failures look like, how far remediation has actually gotten, and what still cannot be caught by a green mechanical sweep.

The thesis of this site is evidence discipline and engineering sovereignty. Publishing 49 articles with 205 wave-1 defects under a self-auditing writer was a failure of that thesis. Publishing the failure, the method, the residual risk, and the gates that now hold bad work is the minimum corrective action. It is not a redemption arc. It is the work that should have existed before the first of those 49 went live.