Workflow Orchestration You Host: Temporal vs Airflow 3

Search for workflow orchestration and you get comparison grids: durable execution here, a scheduler and a provider ecosystem there. Accurate, and they decide nothing.

Temporal and Airflow are two different state machines, and the state each persists is the whole decision — what a third party would hold, what you back up, what breaks when someone deploys on a Tuesday. This article is about that state: what each engine writes down, what encryption provably cannot remove from it, and how you get it back out.

Two engines, two state machines

Temporal is a durable-execution engine. Your Workflow code is ordinary code that happens to be replayable: the engine records an append-only Event History per execution and reconstructs in-flight state by replaying it. Hence the determinism rule — you must "ensure that any time your Workflow code is executed it makes the same Workflow API calls in the same sequence, given the same input". Non-deterministic work belongs in an Activity, whose result is recorded as an event rather than recomputed.

Airflow is a DAG scheduler. Its metadata database records which task instances exist, their state, when they ran, and the small hand-off values passed between them as XComs — which the documentation says "are only designed for small amounts of data; do not use them to pass around large values, like dataframes". The work happens in operators talking to systems outside Airflow, so the database is a ledger, not a replayable program.

That difference cascades. Because Temporal replays, editing a running Workflow Definition is a correctness hazard: "if a generated Command doesn't match what it needs to in the existing Event History, then the Workflow Execution returns a non-deterministic error." Because Airflow schedules, the same edit is a versioning question — in Airflow 3, "a DAG will run through to completion based on the version at start, even if a new version has been uploaded while this DAG was being run."

So the question is not which is better, but whether the engine should hold the business process itself or a schedule and a ledger. A payment saga that must survive a six-hour third-party outage without losing its place wants the first. A nightly warehouse transformation wants the second — often paired with a self-managed event backbone rather than making the scheduler carry data.

Click to zoom

What the engine actually knows about your business

This is what a residency review should ask and usually does not: which artefacts of a running business process land in the engine's store, and in what form.

Temporal's data-encryption page enumerates what the Event History persists and can encrypt: Workflow, Activity and Child Workflow inputs and outputs; Signal inputs; Memos; headers; Query inputs and results; Local Activity and Side Effect results; application errors and failures. With a Payload Codec, "data exists unencrypted only on the Client and the Worker process, on hosts that you control".

That is a strong boundary, and it is not the whole store. Temporal documents a second surface directly: Search Attribute values "are stored unencrypted in the Visibility store and are not processed by a custom Payload Codec", because the Server "must be able to read these values in plain text to support filtering and ordering, so encryption is not possible without breaking search functionality". The attached instruction is equally direct — no sensitive data, secrets or PII in Search Attribute names or values.

One more documented leak, and it bites during an incident: failure messages and call stacks "are not encoded as codec-capable Payloads by default". Stack traces are exactly where a payload fragment ends up quoted verbatim.

Airflow's version is smaller in payload terms and sharper in credential terms. The metadata database holds task instance rows, DAG run state, XCom values — and your secrets: "Airflow uses Fernet to encrypt passwords in the connection configuration and the variable configuration." Real encryption, with a matching hazard, since "changing the key will cause decryption of existing credentials to fail". The Fernet key is not a config value; it is a restore dependency.

Click to zoom

Self-hosting Temporal: four services and one number you cannot change

A Temporal Service is four independently scalable roles plus persistence: Frontend, Matching, History and an internal Worker Service. The Frontend is "a stateless gateway service that exposes a strongly typed Proto API" with "no sharding or partitioning" — the easy part. Matching is "responsible for hosting user-facing Task Queues for Task dispatching" and "can scale internally by having multiple instances". History "is responsible for persisting Workflow Execution state to the Event History", sized from what Temporal "recommends starting at a ratio of": "1 History Service process for every 500 History Shards".

Which brings us to the number. History Shards are chosen once: "after the Shard count is configured and the database integrated, the total number of History Shards for the Temporal Service cannot be changed." Pick against the throughput you expect in three years, and record the reasoning beside the value.

Persistence is a PostgreSQL, MySQL or Cassandra cluster you now own — an HA-Postgres problem before it is a workflow problem, which CloudNativePG on Kubernetes keeps from becoming a bespoke one. Visibility is a second store, and need no longer be Elasticsearch: the compatibility matrix carries Advanced Visibility on PostgreSQL v12+ and MySQL v8.0.17+ from Temporal Server v1.20, and OpenSearch 2+ from v1.30.1.

Read that as one dependency becoming optional, not disappearing. The same matrix still marks Elasticsearch "Recommended for any setup that spawns more than a few Workflow Executions". SQL-backed visibility removes it as a hard prerequisite for custom Search Attributes at low to moderate volume; it does not tell a team sizing a busy cluster to skip it. Check those floors against what you deploy — v1.31.2 was Latest on 26 August 2026, but the floors matter, not that number.

Self-managing Airflow 3: control plane, data plane, and the upgrade you already missed

Airflow 3's Task Execution Interface (AIP-72) "enables the evolution of Airflow into a client-server architecture, which represents one of the most significant architectural shifts in Airflow's history" — and its security consequence is concrete enough to hand a reviewer. From the ASF security model: workers communicate with the API server "exclusively through the Execution API", and "do not receive database credentials and genuinely cannot access the metadata database directly".

Be precise about the scope, because this is the part people overstate. Workers are genuinely cut off; the Dag File Processor and Triggerer are not. Airflow "implements software guards that prevent accidental direct database access from Dag author code" — against accident, not intent. If DAG authors sit outside your trust boundary, the processor is where that matters.

Scheduler HA needs less ceremony than its reputation suggests. Airflow coordinates schedulers with "database row-level locks (using SELECT ... FOR UPDATE)", and "users of PostgreSQL 12+ or MySQL 8.0+ are all ready to go... there is no further set up or config options needed".

The official Helm chart supports LocalExecutor, CeleryExecutor and KubernetesExecutor across all Airflow versions. One setting in it is not a preference: the chart's own Production Guide advises an external database, says the embedded Postgres "lacks stability, monitoring and persistence features that you need for a production database", and warns that "you might experience data loss when you are using it". Disable it on the chart's authority, not on anyone's taste:

# helm upgrade --install airflow apache-airflow/airflow -f values.yaml
executor: KubernetesExecutor

# Production Guide: "Embedded Postgres lacks stability, monitoring and
# persistence features that you need for a production database."
postgresql:
  enabled: false

data:
  # kubectl create secret generic airflow-metadata --from-literal=\
  #   connection=postgresql://airflow:PASSWORD@pg-rw:5432/airflow
  metadataSecretName: airflow-metadata

# Lose this key and existing credentials cannot be decrypted: it belongs
# in the restore runbook, not only in the cluster.
fernetKeySecretName: airflow-fernet-key
jwtSecretName: airflow-jwt
apiSecretKeySecretName: airflow-api-secret

# HA needs nothing else on a supported backend.

scheduler:
  replicas: 2
triggerer:
  replicas: 2

Then the upgrade tax, learnable from someone else's calendar. Airflow 2 entered limited maintenance on 22 October 2025 and reached EOL on 22 April 2026; Airflow 3 first released 22 April 2025. Getting to 3 requires being on 2.7 or later first, and the removals are real work: SubDAGs replaced by TaskGroups and asset-driven scheduling, SLAs replaced by Deadline Alerts, and "task code can no longer directly import and use Airflow database sessions or models".

Failure modes that only appear under load

Three failures only show up under load, and all three are invisible in a proof of concept: functions of volume and time, not correctness.

Event History growth hits a hard ceiling. A Temporal Workflow that loops — polling, retrying, draining a queue forever — accumulates events until the Service stops it. Temporal warns at 10 MB of Event History and errors at 50 MB. There is a matching count ceiling, and both are dynamic-config keys you can set:

# The Temporal Service's dynamic config file. Values are the documented
# defaults, made explicit so a change shows up as a diff.
limit.historyCount.warn:
  - value: 10240
    constraints: {}
limit.historyCount.error:
  - value: 51200
    constraints: {}

limit.historySize.warn:
  - value: 10485760      # 10 MB
    constraints: {}
limit.historySize.error:
  - value: 52428800      # 50 MB
    constraints: {}

# Single-Payload ceiling. The warn threshold is deliberately absent -
# Temporal's two pages disagree on it (see the prose).
limit.blobSize.error:
  - value: 2097152       # 2 MB
    constraints: {}

A note on that omission, which you will find yourself: the self-hosted defaults page says Temporal warns at 256 KB for a single Payload blob, while the dynamic-configuration reference gives limit.blobSize.warn as 512 KB. Both agree the 2 MB error ceiling. Design so neither number is load-bearing — pass a reference to an object store, not the object. The fix for history growth is the same shape: Continue-As-New starts a fresh execution with a fresh history. Watch the default limit, as of v1.21, of 2,000 pending Activities, Child Workflows, Signals or Workflow cancellation requests per execution too.

A deploy breaks replay determinism. Editing a Workflow Definition while executions are in flight is the classic Temporal outage: replayed Commands stop matching the recorded history and the execution fails. Temporal's answer is Worker Versioning — "a Pinned Workflow is guaranteed to complete on a single Worker Deployment Version", while an Auto-Upgrade Workflow "will automatically move to a new code version as you roll it out". It needs Temporal Server v1.29.1 or later: an argument for keeping a self-hosted cluster current rather than parked.

The metadata database becomes the bottleneck. Oversized XCom values and high-frequency scheduling land on the same Postgres, so Airflow's warning against passing dataframes through XComs is a scaling instruction as much as a style one. Instrument the database first — connection saturation, lock waits, scheduler loop duration — because self-hosted observability turns all three of these into a graph you watched climb. If the worker pool spikes, event-driven autoscaling with KEDA belongs in front of it.

The security boundary you actually control

A Payload Codec is a pair of functions running in your process: encode before data leaves, decode after it arrives. The Service stores ciphertext and never holds the key, which is what makes the earlier claim — plaintext only on hosts you control — true rather than aspirational.

import dataclasses
import os
from typing import Iterable, List

from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from temporalio.api.common.v1 import Payload
from temporalio.client import Client
from temporalio.converter import PayloadCodec, default

# Key material comes from your KMS, never from the Service.
_AEAD = AESGCM(bytes.fromhex(os.environ["TEMPORAL_PAYLOAD_KEY"]))
_ENCODING = b"binary/aes-256-gcm"
_KEY_ID = b"2026-08"


class AesGcmCodec(PayloadCodec):
    """Encodes whole Payloads, so metadata is sealed with the body."""

    async def encode(self, payloads: Iterable[Payload]) -> List[Payload]:
        sealed = []
        for p in payloads:
            nonce = os.urandom(12)
            ct = _AEAD.encrypt(nonce, p.SerializeToString(), None)
            sealed.append(
                Payload(
                    metadata={"encoding": _ENCODING, "key-id": _KEY_ID},
                    data=nonce + ct,
                )
            )
        return sealed

    async def decode(self, payloads: Iterable[Payload]) -> List[Payload]:
        opened = []
        for p in payloads:
            if p.metadata.get("encoding") != _ENCODING:
                opened.append(p)  # not ours
                continue
            inner = Payload()
            inner.ParseFromString(_AEAD.decrypt(p.data[:12], p.data[12:], None))
            opened.append(inner)
        return opened


async def connect() -> Client:
    return await Client.connect(
        "temporal-frontend.temporal.svc.cluster.local:7233",
        namespace="payments",
        data_converter=dataclasses.replace(default(), payload_codec=AesGcmCodec()),
    )

Two things it does not do. It does not encrypt Search Attribute values, for the documented reason that the Server must read them to filter and order on them. And it does not hide attribute names, which are additionally visible in Namespace configuration, query expressions and the Temporal UI — so a rigorously empty value still leaks the schema of what you index on.

The consequence is a naming rule, enforced in code review rather than by the engine. Workflow Ids should be opaque — a UUID with a type prefix, never a customer reference or case identifier — and custom Search Attributes should index shape, not subject: a tenant surrogate key, a status enum.

Then decide where the decrypt path lives, because that is the real boundary. A Codec Server is "an HTTP server that uses your custom Codec logic to decode your data remotely", so the Web UI can show readable payloads. Wherever you host it, and whoever can reach it, is the real answer to who can read your workflow data — however good the cipher.

Exit ramps: getting workflow state back out

An orchestrator accumulates gravity fast, because business processes get written against its primitives. The exit test is not "can we export" but "have we exported, recently, and did it contain anything".

Start with retention, the setting most likely to be inherited rather than chosen. A Namespace Retention Period governs how long the Service keeps data for closed Workflow Executions, and Temporal's two creation paths differ on purpose: a Namespace created with temporal operator namespace create and no explicit value "defaults to 3 days", while the Register Namespace Request API refuses to guess and returns an error. Three days is fine for a queue and poor for anything a regulator might ask about.

#!/usr/bin/env bash
# exit-drill.sh <namespace> <output-dir>
set -euo pipefail

NS="$1"
OUT="$2"

# Retention is a decision, not an inheritance: the CLI defaults to 3 days.
# Set it once, out of band; this drill only reports it.
temporal operator namespace describe --namespace "$NS" | grep -i retention

mkdir -p "$OUT"

# Closed executions only - what retention counts down on.
temporal workflow list --namespace "$NS" \
  --query 'ExecutionStatus != "Running"' \
  --limit 200 --output json > "$OUT/closed.json"

# -o json emits a bracketed array - guard the shape.
sel='if type=="array" then . else .executions end'
count=$(jq "$sel | length" "$OUT/closed.json")
[ "$count" -gt 0 ] || { echo "FAIL: nothing closed to export"; exit 1; }

# Pull one full Event History; prove it parses.
wid=$(jq -r "$sel | .[0].execution.workflowId" "$OUT/closed.json")
rid=$(jq -r "$sel | .[0].execution.runId" "$OUT/closed.json")

temporal workflow show --namespace "$NS" \
  --workflow-id "$wid" --run-id "$rid" \
  --output json > "$OUT/history.json"

events=$(jq '[.. | objects | select(has("eventId"))] | length' "$OUT/history.json")
[ "$events" -gt 0 ] || { echo "FAIL: empty history for $wid"; exit 1; }

echo "OK: $count closed, $events events exported for $wid"

The managed tier has its own documented exit: Temporal Cloud's Workflow History Export writes closed Workflow Histories to an S3 or GCS bucket in proto format. It also reframes retention as a bill — Temporal Cloud publishes Active Storage at $0.042 per GB-hour and Retained Storage at $0.00105 per GB-hour, rates read on 26 August 2026 — which is exactly the sort of number that belongs in an exit-cost model.

For Airflow the export is duller and the trap sharper: the metadata database dumps like any Postgres, and the dump is useless without the Fernet key that decrypts the credentials inside it. Rehearse the pair together, as with any Kubernetes restore drill. A backup nobody has restored is a hypothesis.

The long game: which engine is still here in ten years

Answer it from two artefacts checkable today, not from roadmaps.

The first is the licence. Temporal's server is published under the MIT License — a permissive licence on the server itself. That is the strongest structural guarantee a single-vendor project can offer: whatever the company does later, the code you run today stays runnable and forkable.

The second is the support record. Airflow publishes a supported-versions table with specific dates: Airflow 2 first released 17 December 2020, limited maintenance 22 October 2025, EOL 22 April 2026, with 3.3.1 the current patch line off a 22 April 2025 first release. Published in advance and adhered to, that is what foundation governance buys: a schedule you can plan against.

So the ten-year question resolves into two answerable ones. Which state model matches the work? And which upgrade tax can your team pay — Temporal's determinism discipline and keeping a cluster current enough for Worker Versioning, or Airflow's scheduled major transitions with named removals?

Whichever you pick, the sovereignty work is the same and smaller than a migration. Write down which artefacts land in the engine's store. Name Workflow Ids so the plaintext ones say nothing. Set retention deliberately. Run the export drill on a schedule and read its output. Cheap now, unbuyable during an incident — and none of it depends on being right about the next decade.