Dagster vs Airflow 3: Asset Graph or Asset-Aware Tasks

Most Dagster-versus-Airflow pages still repeat an older line: Airflow schedules tasks, Dagster models the data those tasks produce. Airflow 3 made that line stale. What remains is a difference in the unit you author, the unit you test, the unit you backfill, and what you have to bolt on before an auditor will accept either one running on your own hardware.

Versions are as of 19 October 2026: Apache Airflow 3.3.2 is the current stable release (a 3.4.0 beta exists above it), and Dagster is 1.13.26. Both are Apache-2.0. Every snippet below is written against those two versions.

The short answer

Choose Dagster when the thing your team argues about is the data: which table is stale, which partition is wrong, what a change upstream will break. Its asset definition is the unit you write, call in a unit test, backfill by partition and see on one lineage graph.

Choose Airflow when the thing you orchestrate is work, much of it not producing tables at all: file transfers, API calls, ML jobs, infrastructure chores. Its Dag remains the unit, with assets as a scheduling signal between Dags, and its control plane is the more complete one to self-host: multiple schedulers, a pluggable auth layer, foundation governance.

Both are open source you can run entirely on infrastructure you own. The decision is which operating gaps you would rather fill.

'Tasks vs assets' is a pre-Airflow 3 framing: what Airflow 3.3 actually does

Airflow 3.0.0 shipped on 2025-04-22. Its release notes say Datasets were renamed to Assets and that an @asset decorator enables "clearer, asset-centric DAG definitions". Airflow 3.2.0 (2026-04-07) made asset partitioning its headline feature: downstream processing can be scheduled on specific partitions of data rather than on a whole asset. Airflow 3.3.0 (2026-07-06) added partition mappers (RollupMapper, FanOutMapper, and FixedKeyMapper with SegmentWindow) that compose with day-to-year time windows and a wait_policy of WaitForAll or MinimumCount(n).

So Airflow can now say "run this when the 2026-10-01 partition of orders lands". What it has not changed is the authored unit. Airflow's asset docs say declaring an @asset automatically creates an Asset, a Dag with the function's name, and a task inside it whose outlet is that asset. The asset is a label on a task's output. And per the asset scheduling docs, Airflow marks an asset as updated only if the task completes successfully: if it fails or is skipped, no update occurs.

import pendulum
from airflow.sdk import Asset, dag, task

orders = Asset(name="warehouse_orders", uri="file:///srv/warehouse/orders")


@dag(
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
)
def load_orders():
    # The asset event is emitted only if this task succeeds;
    # a failed or skipped run updates nothing downstream.
    @task(outlets=[orders])
    def load(data_interval_start=None, data_interval_end=None):
        print(f"loading orders from {data_interval_start} to {data_interval_end}")

    load()


@dag(schedule=[orders], start_date=pendulum.datetime(2026, 1, 1, tz="UTC"))
def build_revenue():
    @task
    def aggregate():
        print("recomputing daily revenue from warehouse_orders")

    aggregate()


load_orders_dag = load_orders()
build_revenue_dag = build_revenue()

if __name__ == "__main__":
    load_orders_dag.test()

In Dagster the relationship runs the other way. Its asset guide defines an asset as an object in persistent storage, such as a table, file or persisted model, and says the Python function inside an asset is an op, with one crucial distinction: asset definitions know their dependencies, ops do not. The graph is built from the assets; the computation hangs off them.

Two control planes, side by side

Airflow 3's architecture overview names a Dag processor, a scheduler, an API server and a metadata database, usually PostgreSQL or MySQL. The API server is also how Task SDK tasks report state back: since 3.0 task code can no longer touch the metadata database directly. Airflow supports more than one scheduler at once, for performance and resiliency, coordinated through that database.

Dagster's OSS deployment architecture needs three long-running services. The webserver can run one or more replicas. The daemon, which operates schedules, sensors and run queuing, lists replicas as not supported. Code location servers can be many, but each code location gets one replica. On the Helm chart, each materialization starts a Kubernetes Job. An unconfigured instance stores runs in SQLite (SqliteRunStorage); the Helm path deploys PostgreSQL, and production should point at one you operate.

Click to zoom

A minimal Dagster release against an external database, with keys taken from the dagster/dagster chart's own values.yaml. If you already run Postgres through an operator, as in CloudNativePG on Kubernetes, point postgresqlHost at its read-write service.

global:
  postgresqlSecretName: dagster-pg-credentials
generatePostgresqlPasswordSecret: false

postgresql:
  enabled: false
  postgresqlHost: dagster-pg-rw.data.svc.cluster.local
  postgresqlUsername: dagster
  postgresqlDatabase: dagster
  service:
    port: 5432

dagsterWebserver:
  replicaCount: 2

dagsterDaemon:
  enabled: true

runLauncher:
  type: K8sRunLauncher

dagster-user-deployments:
  enabled: true
  deployments:
    - name: warehouse
      image:
        repository: registry.example.com/data/warehouse-code
        tag: "2026.10.19"
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/opt/dagster/app/assets.py"
      port: 3030
      replicaCount: 1

Two details in the chart's templates decide whether this release comes up. With postgresql.enabled: false the chart uses postgresqlHost rather than deploying its own database, and the password is read from the postgresql-password key of the Secret named in global.postgresqlSecretName, so create that Secret before the release. The daemon's Deployment hardcodes one replica: the replicaCount knob that scales the webserver has no daemon equivalent, which is the architecture table's "not supported" expressed in YAML.

Testing: a function call versus a Dag run

Here is the Dagster side of the same pipeline: two daily-partitioned assets in SQLite and an asset check. Both assets rewrite their whole time window, so re-running a partition is idempotent.

import os
import sqlite3
from contextlib import closing

import dagster as dg

daily = dg.DailyPartitionsDefinition(start_date="2026-01-01")
FMT = "%Y-%m-%d %H:%M:%S"


def connect() -> sqlite3.Connection:
    conn = sqlite3.connect(os.environ.get("WAREHOUSE_DB", "warehouse.db"))
    conn.executescript(
        """
        CREATE TABLE IF NOT EXISTS source_orders (
            id INTEGER PRIMARY KEY, customer_id INTEGER,
            amount REAL NOT NULL, ordered_at TEXT NOT NULL);
        CREATE TABLE IF NOT EXISTS orders (
            id INTEGER PRIMARY KEY, customer_id INTEGER,
            amount REAL NOT NULL, ordered_at TEXT NOT NULL);
        CREATE TABLE IF NOT EXISTS daily_revenue (
            day TEXT PRIMARY KEY, revenue REAL NOT NULL);
        """
    )
    return conn


@dg.asset(partitions_def=daily, backfill_policy=dg.BackfillPolicy.single_run())
def orders(context: dg.AssetExecutionContext) -> dg.MaterializeResult:
    # One run may cover one day or a 90-day range: read the window, not the key.
    start, end = context.partition_time_window
    lo, hi = start.strftime(FMT), end.strftime(FMT)
    with closing(connect()) as conn, conn:
        conn.execute("DELETE FROM orders WHERE ordered_at >= ? AND ordered_at < ?", (lo, hi))
        cur = conn.execute(
            "INSERT INTO orders SELECT id, customer_id, amount, ordered_at "
            "FROM source_orders WHERE ordered_at >= ? AND ordered_at < ?",
            (lo, hi),
        )
    return dg.MaterializeResult(metadata={"rows": cur.rowcount, "window": f"{lo} .. {hi}"})


@dg.asset(
    partitions_def=daily,
    backfill_policy=dg.BackfillPolicy.single_run(),
    deps=[orders],
)
def daily_revenue(context: dg.AssetExecutionContext) -> dg.MaterializeResult:
    start, end = context.partition_time_window
    lo, hi = start.strftime(FMT), end.strftime(FMT)
    with closing(connect()) as conn, conn:
        conn.execute("DELETE FROM daily_revenue WHERE day >= ? AND day < ?", (lo[:10], hi[:10]))
        cur = conn.execute(
            "INSERT INTO daily_revenue SELECT substr(ordered_at, 1, 10), SUM(amount) "
            "FROM orders WHERE ordered_at >= ? AND ordered_at < ? "
            "GROUP BY substr(ordered_at, 1, 10)",
            (lo, hi),
        )
    return dg.MaterializeResult(metadata={"days": cur.rowcount})


@dg.asset_check(asset=orders)
def orders_have_customer() -> dg.AssetCheckResult:
    with closing(connect()) as conn:
        (missing,) = conn.execute(
            "SELECT COUNT(*) FROM orders WHERE customer_id IS NULL"
        ).fetchone()
    return dg.AssetCheckResult(passed=missing == 0, metadata={"missing": missing})


defs = dg.Definitions(assets=[orders, daily_revenue], asset_checks=[orders_have_customer])

Dagster's unit-testing guide says assets can be tested by invoking the definition directly, passing upstream values where needed. With build_asset_context you supply a partition key and call the asset like a function. No webserver, no daemon, no Dagster database:

import sqlite3

import dagster as dg
import pytest

from assets import daily_revenue, orders, orders_have_customer


@pytest.fixture
def warehouse(tmp_path, monkeypatch):
    path = tmp_path / "warehouse.db"
    monkeypatch.setenv("WAREHOUSE_DB", str(path))
    orders_have_customer()  # connect() creates the schema
    with sqlite3.connect(path) as conn:
        conn.executemany(
            "INSERT INTO source_orders VALUES (?, ?, ?, ?)",
            [
                (1, 10, 40.0, "2026-10-01 09:00:00"),
                (2, 11, 60.0, "2026-10-01 17:30:00"),
                (3, None, 25.0, "2026-10-02 08:15:00"),
            ],
        )
    return path


def test_one_partition_loads_only_its_day(warehouse):
    result = orders(dg.build_asset_context(partition_key="2026-10-01"))
    assert result.metadata["rows"] == 2


def test_revenue_is_summed_per_day(warehouse):
    ctx = dg.build_asset_context(partition_key="2026-10-01")
    orders(ctx)
    daily_revenue(ctx)
    with sqlite3.connect(warehouse) as conn:
        rows = conn.execute("SELECT day, revenue FROM daily_revenue").fetchall()
    assert rows == [("2026-10-01", 100.0)]


def test_check_fails_on_missing_customer(warehouse):
    orders(dg.build_asset_context(partition_key="2026-10-02"))
    assert orders_have_customer().passed is False

The asset check does double duty: it is a test here, and in production every job targeting orders runs it by default, so the same assertion guards each materialization.

Airflow's equivalent is dag.test(). Its debugging docs are explicit that it executes a real Dag run, with two prerequisites: an initialized metadata database (airflow db migrate) and a Dag defined in a file inside a configured Dag bundle. That is a fast integration test, not a unit test. The sound Airflow habit is to keep transformation logic in plain modules tested on their own, and use dag.test() for wiring. Dagster's model gives you the unit test for free; Airflow's asks you to impose the separation yourself.

Backfills and partitions

Take a 90-day backfill. In Dagster, per its backfill guide, a backfill over N partitions launches N separate runs by default, one per partition: 90 runs, each a Kubernetes Job on the chart above. BackfillPolicy.single_run() collapses that to one run, under two conditions. The backfill must be launched from the asset graph or asset page, or the assets must belong to an asset job that shares the same backfill policy across every included asset. And the code must be range-aware: if it reads context.partition_key, it has to switch to partition_time_window, partition_key_range or partition_keys. That is why the assets above read the window: launched over 2026-10-01 to 2026-10-03, they execute as one run, one DELETE/INSERT per asset over the whole window, and Dagster records a materialization against each of the three partitions. Note that Dagster 1.13.26 still marks backfill_policy as a beta parameter, so pin the version and re-read the changelog on upgrade.

The window arithmetic is where range-aware code goes wrong, so it is worth stating exactly. For a single-run backfill, Dagster 1.13.26 builds the time window from the start of the first selected partition to the end of the last. Daily partitions 2026-10-01 through 2026-10-03 therefore arrive as one window that opens at 2026-10-01 00:00 and closes at 2026-10-04 00:00, end exclusive. That is why both assets compare >= on the start and < on the end: write <= and the run also deletes and reloads the first rows of a fourth day it was never asked to touch. The docs' instruction to stop reading context.partition_key follows from the same fact, since a run that covers a range has no single key to give you.

The guide says you launch it with the Materialize button on the asset details page or the Global asset lineage page. The backfill is something you do to data you can see.

Airflow 3.0 moved backfills into the scheduler, with UI and REST API support, instead of a separate command-line job. The backfill docs give each backfill its own max_active_runs and a reprocess behaviour of Missing Runs, Missing and Errored Runs, or All Runs, and say backfill does not make sense for Dags without a time-based schedule. In 3.3.2 the CLI help adds that for partitioned Dags the date range is read as a partition-date range.

Those three UI labels are the CLI's --reprocess-behavior tokens none, failed and completed, in that order. none creates runs only for logical dates that have no run yet; failed also re-runs dates whose earlier run failed; completed re-runs every date in the range. Under none of the three does Airflow duplicate a run that is currently running or queued. Both --from-date and --to-date are inclusive, so the script below covers 90 logical dates, --max-active-runs 3 caps how many of those Dag runs execute concurrently, and --dry-run lists the runs it would create without creating them, which is the cheap way to check the date arithmetic before the work lands on your executors.

#!/usr/bin/env bash
set -euo pipefail

dag_id="load_orders"
from_date="2026-07-01"
to_date="2026-09-28"

# Show which runs would be created before creating any.
airflow backfill create --dag-id "$dag_id" \
  --from-date "$from_date" --to-date "$to_date" \
  --reprocess-behavior failed --max-active-runs 3 --dry-run

airflow backfill create --dag-id "$dag_id" \
  --from-date "$from_date" --to-date "$to_date" \
  --reprocess-behavior failed --max-active-runs 3

The unit of an Airflow backfill is a Dag run per logical date; the assets update as a consequence. Dagster's unit is the asset partition, and whether it costs one run or ninety is a property you set on the asset.

Lineage and what an auditor is shown

Dagster's lineage is its primary interface: the Global asset lineage page is derived from the same definitions you test, so it cannot drift from the code. Airflow offers three separate things that are easy to conflate. Asset views show which Dags produce and consume which assets. Hook lineage, which the lineage docs open by calling "very experimental and subject to change", collects nothing until a HookLineageReader is registered in a plugin; without one, a no-op collector is used. And the OpenLineage provider emits events to an open lineage framework, which is the portable choice if lineage has to outlive your orchestrator. It needs a configured transport to know where to send those events, so the lineage backend is a separate service you choose and operate, not a feature of the orchestrator.

Lineage is half of the audit story. The other half is who can trigger a backfill. Airflow 3's default Simple auth manager is intended for development and testing; the docs say that in production access must be controlled through other means. Auth managers are pluggable, and the FAB auth manager provides users, roles and permissions stored in the Airflow database. Its default roles (Admin, User, Op, Viewer and Public) give an auditor a role model to review, with permissions attached to roles rather than scattered across users.

Dagster OSS is thinner here. Issue #2219, "Support Auth & RBAC in Dagit", was opened on 2020-02-27 and is still open. Dagster Labs' own blog, a vendor source, positions fine-grained RBAC, audit logs and SSO as Dagster+ features, and says running OSS means owning the webserver, daemon and database. Put an authenticating proxy or IdP in front of the webserver, and plan as if anyone past it can launch and backfill anything the UI offers. If that does not satisfy your data classification rules, that is a finding, not a configuration task.

Failure modes that only show up self-hosted

• The Dagster daemon is one process. Schedules, sensors and run queuing all require it, and replicas are not supported. A Kubernetes Deployment restarts it, but while it is down nothing automated fires. Alert on its heartbeat, not only on pod status.
• One replica per code location. With no second replica to fail over to, a crashed or redeploying code location server is an outage for that location's assets. Split large repositories into several code locations so one bad import does not blind the whole graph.
• Declarative automation lives in the daemon too. Dagster's AutomationCondition (on_cron, eager, on_missing) is evaluated there; a stopped daemon means assets quietly go stale.
• Airflow's default login is a development tool. The Simple auth manager is the default, which makes it easy to ship to production by accident. Pick a provider auth manager before the first real user.
• Silent absence of asset events. A skipped producer task emits no event, so the consumer Dag never runs and nothing errors. Monitor expected asset updates, not just failed tasks.

None of these is exotic. They are the cost of running a control plane yourself, and the chart and docs of each project say so plainly. For the Airflow side of the deployment (Helm values, external Postgres, the Airflow 2 end-of-life path), see Temporal vs Airflow 3.

Exit ramps: Airflow to Dagster, and back

Dagster's Airlift toolkit (dagster-airlift) observes Airflow instances from Dagster and accelerates migrating Dags to assets. It depends on Airflow's REST API, and its compatibility note lists OSS Airflow as "Stable as of 2.00", naming no Airflow 3 version.

This is a narrow, dated finding. It says nothing about Dagster's plans, which the project has not announced, and a later Airlift release could move to /api/v2. Before planning around Airlift, read the base URL in the exact release you would pin. If it still says /api/v1, budget for the rewrite. The mechanical half is turning each producer task into an asset definition that wraps the same module function. The slower half is re-expressing schedules, sensors and asset-triggered Dags as Dagster schedules or automation conditions, then running both orchestrators side by side until the asset graph and the Airflow run history agree on what is fresh.

What does not port in either direction is the model. Dagster asset definitions, IO managers and partition definitions have no Airflow counterpart; Airflow operators, hooks and connections have none in Dagster. Keep business logic in plain Python modules and SQL, as the assets above do, so an orchestrator change is a rewrite of thin wrappers. The same discipline applies to storage: the ClickHouse, Iceberg and DuckLake lakehouse piece makes the case for open table formats, and schema migration tooling for keeping table changes out of either orchestrator.

The long game

Both are Apache-2.0, so neither can be taken from you. The difference is who decides what lands next. Apache Airflow is governed by the Apache Software Foundation. Dagster has a single commercial owner, and on 13 July 2026 that owner announced that Prefect is acquiring Dagster Labs. Its open-source project continues under its existing licence, will be actively developed, not just maintained, and Dagster+ continues under the Dagster+ brand; after close, the combined company was expected to operate under the Prefect name from August 2026.

That is a stated commitment, not a guarantee, and the OSS access-control gap has been open for more than six years under the previous owner. Over a decade, a foundation's roadmap and a vendor's roadmap fail differently. Choose the asset model if it fits your data, but keep the logic portable, the lineage in an open format and the database yours.