Self-Hosted Grafana vs Datadog: Kubernetes Observability

Datadog and similar SaaS observability platforms bill on a model that compounds quietly: an agent per node, a charge per custom metric ingested for SLO tracking, log retention beyond the default window sold as a paid add-on, APM billed per instrumented service, and a per-seat charge for every engineer with dashboard access. None of those line items shrinks on its own — cardinality grows as services multiply, retention needs grow as compliance requirements tighten, and headcount grows with the team. For a regulated organization on that pricing model, the question that eventually has to be answered is not whether to negotiate the next renewal but whether to own the telemetry pipeline outright. This article is the architecture for that: OpenTelemetry, Prometheus, Loki, and Tempo, trading per-metric and per-GB licensing for storage and compute you already operate — with retention set by your own policy rather than bought back from a SaaS tier.

The self-hosted observability stack in 2026 has a clear center of gravity: OpenTelemetry as the instrumentation and collection layer, Prometheus for metrics (with Thanos or Cortex — both CNCF incubating — for long-term storage), Grafana Loki for logs, Grafana Tempo for distributed traces, and Grafana as the unified query and dashboard layer. Maturity is mixed: OpenTelemetry and Prometheus are CNCF graduated; Thanos and Cortex are incubating; Grafana, Loki, and Tempo are Grafana Labs OSS (AGPLv3), not CNCF projects. None of them has proprietary wire formats that trap your data. All run on Kubernetes via mature Helm charts. The OTLP protocol and the traces and metrics SDKs are stable across the major languages; the Collector remains on a 0.x line with per-component stability, and the logs SDKs are stable in some languages and 0.x in others — still a viable vendor-neutral layer over whatever backends you choose.

This is not a simple-is-better argument. A self-hosted observability stack requires more operational attention than a SaaS product, and that trade-off deserves honest examination. We will cover the architecture in enough detail to deploy it production-grade, the specific operational investments required, a named trade-off (the cardinality management problem that Prometheus inherits from its data model), and the decision boundary at which the self-hosted stack makes sense versus when SaaS is the right answer. The sovereignty thesis is not "always self-host" — it is "understand what you are renting and what you own, and make that decision deliberately."

Why Telemetry Data Belongs Inside Your Audit Perimeter

Observability data is not just operational — it is forensic evidence. When a regulated fintech's production cluster has an incident at 2 AM, the traces, logs, and metric time series from that window become the audit trail for the post-incident review, the evidence for any regulatory inquiry, and the data source for any internal investigation into whether the incident constituted a breach. Under NIS2, DORA's 4-hour incident-reporting clock, sector-specific regulations like PCI DSS v4.0, and SOC 2's CC7 criterion for retained, monitored system-operations evidence, the organization is responsible for the integrity and availability of that evidence — including knowing where it is stored, who has access to it, and that it has not been tampered with.

SaaS observability platforms hold that data in their infrastructure, subject to their data retention policies, their access controls, their geo-replication decisions, and — critically — their data export APIs. If the vendor changes their export API, introduces rate limits on historical data retrieval, or simply discontinues a product tier, your forensic evidence is at risk. The alternative is not necessarily harder: a Grafana Loki cluster backed by S3-compatible object storage with server-side encryption at rest, a 13-month retention policy enforced by Loki's compactor (compactor.retention_enabled: true, delete_request_store, and retention_period) or S3 lifecycle rules, and immutable object versioning turned on — that is a forensic-grade log store you control, auditable by any third party with read access to the bucket. The audit-grade rigor pillar rests on this: controls that can be independently verified produce better evidence than controls that must be trusted because they are opaque.

The OpenTelemetry Collector: One Agent to Rule Them All

The OpenTelemetry Collector is the key architectural decision in a self-hosted stack. It functions as a vendor-neutral telemetry pipeline: it receives signals in dozens of formats (OTLP, Jaeger, Zipkin, Prometheus scrape, Fluent Bit, statsd, and more), processes them (batching, filtering, attribute enrichment, sampling), and exports them to any number of backends. On Kubernetes, the canonical deployment is a DaemonSet for per-node collection (node metrics, host logs, container logs via the filelog receiver) plus a Gateway Deployment for cluster-level aggregation (receiving OTLP from all pods, applying global tail-sampling for traces, and fan-out to all backends). The separation matters: DaemonSet receivers stay lightweight; the Gateway can run tail-sampling only when every span of a trace reaches the same sampling pod — a single replica, or a loadbalancing exporter routing by traceID.

# OpenTelemetry Collector — Gateway Deployment (cluster aggregation layer)
# Deployed via the opentelemetry-operator or directly via Helm (open-telemetry/opentelemetry-collector).
# This config covers the core production pipeline: metrics, logs, and traces.
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: Deployment   # Gateway mode — receives from DaemonSet collectors
  # replicas: 1 while tail_sampling runs here — multi-replica Gateway behind a
  # normal Service splits one trace across pods. Scale sampling with a front-tier
  # loadbalancing exporter (routing_key: traceID) to a headless Service of this tier.
  replicas: 1
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317    # Receives from DaemonSet and instrumented apps
          http:
            endpoint: 0.0.0.0:4318

    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 512          # Hard limit — prevents OOM on burst ingestion
        spike_limit_mib: 128
      batch:
        send_batch_size: 8192
        timeout: 5s             # Balance latency vs efficiency
      resource:
        attributes:
          - action: upsert
            key: k8s.cluster.name
            value: "production"
      # Tail-based sampling: keep 100% of error traces, 5% of successful traces.
      # Requires the Gateway to see full traces — incompatible with DaemonSet-only mode.
      tail_sampling:
        decision_wait: 10s      # Wait up to 10s for all spans before sampling decision
        policies:
          - name: keep-errors
            type: status_code
            status_code: {status_codes: [ERROR]}
          - name: probabilistic-sample
            type: probabilistic
            probabilistic: {sampling_percentage: 5}

    exporters:
      prometheusremotewrite:
        endpoint: "http://prometheus-operated.observability.svc:9090/api/v1/write"
        tls:
          insecure: true   # mTLS via Cilium / WireGuard at node level
      otlphttp/loki:
        endpoint: http://loki-gateway.observability.svc/otlp
      otlp/tempo:
        endpoint: "http://tempo-distributor.observability.svc:4317"
        tls:
          insecure: true

    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, resource, batch]
          exporters: [prometheusremotewrite]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, resource, batch]
          exporters: [otlphttp/loki]
        traces:
          receivers: [otlp]
          processors: [memory_limiter, resource, tail_sampling, batch]
          exporters: [otlp/tempo]

The instrumentation side is mature enough for production. For Go services, the go.opentelemetry.io/otel SDK at v1.44.0 (May 2026) provides stable APIs for traces and metrics; the logs SDK remains a 0.x module (v0.20.0) and its API can still change. For Java (Spring Boot 3.x), the OpenTelemetry Java Agent — a single -javaagent JVM flag — auto-instruments JDBC, HTTP clients, Kafka, gRPC, and Micrometer with zero application code changes. For Python, opentelemetry-sdk with FastAPI or Django instrumentation covers the common cases. Because all three signals flow through OTLP to the Collector, switching backends is a Collector config change — not an application rewrite. Instrumenting a WebAssembly component still needs more manual wiring; Wasm's tracing ecosystem is younger than Go's, Java's, or Python's. The fourth signal, profiles, takes a different route entirely — an eBPF profiler needs no SDK at all.

Click to zoom

Prometheus at Production Scale: The Cardinality Problem and How to Manage It

Prometheus is the gold standard for Kubernetes metrics — it integrates natively with the Kubernetes API via the kube-state-metrics exporter and cAdvisor, and the ServiceMonitor / PodMonitor CRDs from the kube-prometheus-stack Helm chart have become the canonical way to configure scrape targets. But Prometheus has a structural limitation that every team running it at scale encounters: high cardinality labels. Prometheus stores metrics as time series identified by a metric name plus a set of label key-value pairs. Each unique combination is a separate series. If a single metric has a user_id label with a million possible values, Prometheus must store and query a million time series for that metric. Memory consumption scales linearly with active series count; query performance degrades as the series count grows.

This is the named trade-off. Datadog and New Relic solve this with custom indexing infrastructure that can handle billions of metric series at cloud scale. A single Prometheus is comfortable in the low millions of active series; beyond roughly 10M you shard by scrape target or move to Thanos Receive / Mimir. The operational discipline required: never use user-generated values as Prometheus labels. user_id, request_id, session_token, URL path (without normalization), IP address — these are cardinality bombs. Use them as trace attributes (where Tempo stores them as fields per span rather than as series dimensions) and as structured log fields in Loki. Rule of thumb: if a label can exceed 1,000 values, it does not belong in Prometheus.

# kube-prometheus-stack Helm values — production-grade Prometheus configuration.
# Deploys Prometheus Operator + Prometheus + Alertmanager + Grafana + kube-state-metrics.
prometheus:
  prometheusSpec:
    replicas: 2                    # HA pair — Thanos sidecar handles deduplication
    retention: 2d                  # Keep only 2 days local; Thanos sidecar uploads to S3
    retentionSize: "50GB"          # Circuit breaker: evict oldest blocks if disk fills
    # Required for OTel Collector prometheusremotewrite → Prometheus /api/v1/write
    # (chart default enableRemoteWriteReceiver: false).
    enableRemoteWriteReceiver: true

    # Thanos sidecar — uploads Prometheus TSDB blocks to S3 for long-term retention.
    # Grafana queries Thanos Query for unified local + remote data.
    thanos:
      image: quay.io/thanos/thanos:v0.36.1
      objectStorageConfig:
        secret:
          type: S3
          config:
            bucket: "prod-metrics-longterm"
            endpoint: "s3.eu-central-1.amazonaws.com"
            region: "eu-central-1"
            access_key: ""       # Use IRSA / workload identity — never static keys
            secret_key: ""

    # Resource limits — size based on expected series count.
    # 1M active series ~ 8-10 GB RAM at steady state (Prometheus 2.x).
    resources:
      requests:
        cpu: "1"
        memory: "8Gi"
      limits:
        cpu: "4"
        memory: "12Gi"

    # Cardinality guardrail: scrape interval 60s for most targets (not 15s).
    # 15s intervals 4x ingestion rate with minimal operational gain for most SLOs.
    scrapeInterval: "60s"
    evaluationInterval: "60s"

alertmanager:
  alertmanagerSpec:
    replicas: 3   # match HA section below; 2 works, 3 eases rolling drains
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: "fast-ssd"
          resources:
            requests:
              storage: 10Gi

Grafana Loki: Log Aggregation Without the Elasticsearch Tax

Loki is architecturally different from Elasticsearch-based log stacks in a way that has significant operational consequences. Rather than full-text indexing every log field at ingestion time, Loki indexes only the stream labels — the same label model as Prometheus — and stores the raw log content as compressed chunks in object storage. Queries (LogQL) filter by labels first to identify the relevant chunk set, then scan those chunks. This means: (1) ingestion is cheap — no per-field analysis at write time, just label extraction and compression; (2) storage is cheap — compressed log chunks in S3 at object-storage prices rather than Elasticsearch's hot-warm-cold node tiers; (3) query performance on high-cardinality label sets is poor — the same cardinality constraint from Prometheus applies. The design principle: Loki labels should identify the source (pod, namespace, application, environment), not the content of individual log lines.

# Loki Helm values — grafana/loki chart (not the deprecated loki-distributed chart).
# Microservices: Distributor, Ingester, Querier, Compactor, Index Gateway.

deploymentMode: Distributed   # Object-store backed (S3) microservices mode

loki:
  auth_enabled: false      # Single-tenant; enable for multi-tenant with X-Scope-OrgID

  commonConfig:
    replication_factor: 3  # 3 ingesters hold a write quorum

  storage:
    type: s3
    # Chart requires chunks + ruler when ruler.enabled (default true).
    # admin is only for Grafana Enterprise Logs (GEL) — omit for OSS Loki.
    bucketNames:
      chunks: prod-loki-chunks
      ruler: prod-loki-ruler
    s3:
      region: eu-central-1
      endpoint: ""         # Default AWS endpoint; override for MinIO / Ceph

  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb         # TSDB index — required with schema v13 / structured metadata
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

  limits_config:
    retention_period: 9504h    # ~13 months (13 × 730.15h) — set per-tenant for multi-tenant
    ingestion_rate_mb: 32
    ingestion_burst_size_mb: 64
    max_streams_per_user: 10000
    max_label_names_per_series: 15   # Cardinality guardrail: reject streams with > 15 labels

  compactor:
    working_directory: /data/compactor
    retention_enabled: true       # Enables per-tenant retention enforcement
    # delete_request_store is required when retention_enabled (Loki 3.x); it is
    # not replaced by storage.bucketNames above — those hold chunks/ruler (and
    # admin for GEL), while this store holds delete-request objects the
    # compactor tracks for retention.
    delete_request_store: s3

# Ship container logs via OTel Collector filelog receiver (preferred over Promtail)
# exporting OTLP to Loki's native /otlp endpoint — backend-agnostic vs a Loki-only agent.

Log shipping in 2026 is predominantly via the OTel Collector's filelog receiver (DaemonSet mode), which tails /var/log/containers/*.log, parses the container log format, and emits structured log records as OTLP. The OTel Collector ships logs to Loki's native OTLP endpoint (Loki 3.x); Loki maps Grafana's default OTLP resource-attribute set (about 17 labels) to index labels — including service.name, k8s.namespace.name, k8s.pod.name, k8s.container.name, other k8s.* owners, and cloud region/zone — with remaining attributes stored as structured metadata. Because Loki's default max index labels is 15 and attributes like k8s.pod.name are high-cardinality, production clusters should set an explicit reduced label set via distributor.otlp_config.default_resource_attributes_as_index_labels or per-tenant limits_config.otlp_config rather than accepting the full default list. The advantage over Promtail — Loki's native log shipper — is backend agnosticism: adding Elasticsearch later is another Collector exporter, not a new agent on every node.

Grafana Tempo: Distributed Tracing at Object-Store Prices

Grafana Tempo stores distributed traces as blocks in object storage — S3, GCS, Azure Blob, or MinIO — with no local disk dependency beyond a small write-ahead log for in-flight traces. S3 Standard is billed on stored bytes (~$0.023/GB/month in us-east-1 for the first 50 TB; other regions differ), so monthly storage cost is roughly ingress × retention window, not a single month of ingress alone. At 10,000 rps with 5% sampling you get ~500 sampled traces/s; at ~10 spans each and ~150 bytes/span compressed that is roughly 2 TB/month of ingress. Holding that volume for a typical regulated 90-day trace window yields ~6 TB steady-state (~$140/month S3 Standard storage, before request and transfer charges). That figure is traces at 90 days — distinct from the ~13-month audit windows often used for logs (Loki) and metrics (Thanos) in this article; keeping the same trace volume for 13 months is ~26 TB steady-state (~$600/month at the same unit price), still usually far below Datadog APM at comparable throughput. Many teams deliberately retain logs and metrics longer than traces; set each signal's policy explicitly. Actual volume will vary with span size and attribute cardinality.

Tempo's query language is TraceQL, which shipped with Tempo 2.0 (January 2023) and gained structural operators in Tempo 2.2 (July 2023). A TraceQL query that finds all traces where the api-server service had spans exceeding 500ms and downstream calls to a database service looks like this:

# TraceQL — find traces where api-server has slow database spans
{
  resource.service.name = "api-server" &&
  span.db.system = "postgresql" &&
  duration > 500ms
}

# Find all error traces across the entire platform in the last hour
{
  status = error
}

# Find traces where a specific user-facing endpoint exceeded SLO (2s)
{
  resource.service.name = "api-server" &&
  name = "POST /api/v1/orders" &&
  duration > 2s
}

# Structural query: find traces where api-server called payments-service
# and that downstream span had an error
{ resource.service.name = "api-server" } >> { resource.service.name = "payments-service" && status = error }

The integration between Tempo and Prometheus via exemplars is the sharpest workflow in the self-hosted stack. When a Go service records a histogram observation (for example, HTTP request duration), it can attach the current TraceID as an exemplar to the histogram sample. Prometheus stores exemplars alongside metrics. In Grafana, a spike in the http_request_duration_seconds_bucket panel surfaces exemplar dots on the graph; clicking a dot jumps directly to the Tempo trace for that request — the one that was actually slow. This three-signal correlation path (dashboard alert -> metric spike -> exemplar -> trace -> log context) is the investigation workflow that replaces both the Datadog APM flame graph and the Datadog log search in a single Grafana Explore session.

Click to zoom

Grafana: Unified Query Without Proprietary Dashboards

Grafana 12 (released May 2025) solidified the unified observability experience that earlier versions promised but partially delivered. The key features for a self-hosted stack: Grafana Scenes for parameterized, drillable dashboard composition; Explore as a multi-signal, multi-datasource investigation workspace; and Alert rules backed by PromQL, LogQL, or TraceQL (unified alerting routes to Alertmanager, PagerDuty, Slack, or any webhook). Grafana OnCall OSS was archived on 2026-03-24 — for self-hosted paging use Alertmanager routing plus an external paging provider, or accept Grafana Cloud IRM as SaaS.

The dashboard portability argument in favor of self-hosting Grafana is concrete: Grafana dashboard JSON is a version-controlled artifact. A dashboard created today for kube-prometheus-stack metrics will work in three years on the next major Prometheus version, because the underlying PromQL queries are portable across any Prometheus-compatible backend (including Thanos, Mimir, and Victoria Metrics). Datadog dashboards are JSON too, but they reference Datadog's proprietary metric naming scheme, DDL filter syntax, and widget types that have no equivalent elsewhere. The exit cost compounds with every dashboard built in the proprietary format.

# Import the kube-prometheus-stack dashboards and the Kubernetes / USE Method dashboards
# into Grafana. These ship as ConfigMaps via the Helm chart and are auto-provisioned.

# Grafana dashboard provisioning — sidecar approach (grafana/grafana Helm chart default).
# Place dashboard JSON files in a ConfigMap labeled with:
#   grafana_dashboard: "1"
# The Grafana sidecar scrapes these ConfigMaps and imports them automatically.

kubectl create configmap grafana-slo-dashboard \
  --from-file=slo-dashboard.json \
  --namespace observability

kubectl label configmap grafana-slo-dashboard \
  grafana_dashboard=1 \
  --namespace observability

# Verify the dashboard was imported
kubectl logs -n observability -l app.kubernetes.io/name=grafana \
  -c grafana-sc-dashboard | grep "slo-dashboard"

# Force a Grafana reload without restart (Grafana 9+ hot-reload API)
kubectl exec -n observability -it \
  $(kubectl get pod -n observability -l app.kubernetes.io/name=grafana -o name | head -1) \
  -- curl -s -X POST http://localhost:3000/api/admin/provisioning/dashboards/reload \
  -H "Authorization: Bearer $GRAFANA_SERVICE_ACCOUNT_TOKEN"

The multi-cluster GitOps pattern applies cleanly here: the observability stack (Helm releases for kube-prometheus-stack, Loki, Tempo, and Grafana) is a set of FluxCD HelmRelease objects in a dedicated platform/observability/ directory in your fleet repository. When a new cluster is provisioned, Flux reconciles the observability stack automatically — the platform team does not have to manually install anything. Alerting rules and Grafana dashboards are ConfigMaps in the same directory. The entire observability stack is version-controlled, reviewed, and deployed identically across all clusters.

The SaaS vs Self-Hosted Decision: Where Each Side Wins

The honest framing: self-hosted observability is the right choice for a specific set of requirements, not universally. Understanding the decision boundary prevents both the reflexive "always SaaS" bias that inflates costs and the reflexive "always self-host" bias that wastes engineering time.

Click to zoom

Self-hosted wins when: (1) Data sovereignty is a hard requirement — regulated industries (fintech, healthcare, defense) where data leaving your control requires legal review or is prohibited outright. (2) Scale has made SaaS costs non-linear — at approximately 5–10 million active Prometheus series, SaaS pricing commonly exceeds the TCO of a self-hosted stack operated by a half-FTE. (3) Retention periods exceed 30 days — SaaS default retention is 15–30 days; extending it to 90 days or 13 months on most platforms costs a multiple of the base contract. (4) Optionality is a strategic requirement — when the organization's plan includes potential migration across cloud providers, the proprietary SaaS query language and dashboard format become an exit barrier.

SaaS wins when: (1) The team has fewer than five engineers and cannot afford a half-FTE to operate and tune the observability stack. (2) Time-to-first-insight is the primary metric — SaaS platforms ship with pre-built APM, infrastructure dashboards, and anomaly detection that a self-hosted stack requires weeks to configure equivalently. (3) The workload is ephemeral — short-lived experiments, hackathons, or MVPs where the platform will be shut down before long-term retention matters. The optionality pillar does not mean building everything yourself; it means preserving the ability to change your choice without catastrophic exit costs. For observability, that means using OpenTelemetry for instrumentation even if you ship to Datadog today — because the instrumentation is vendor-neutral, switching backends later is a configuration change, not a code change.

Alerting That Survives Incident Conditions

An alerting stack that goes silent during the incident it should be alerting on is worse than no alerting at all. The classic failure mode in self-hosted stacks: Prometheus is the source of truth for cluster health, but Prometheus depends on the cluster. When etcd has a split-brain event or a node running a Prometheus pod fails, Prometheus itself may be degraded at the moment you most need it. The mitigation requires two patterns working together: HA Prometheus pairing with deduplication at the Alertmanager level, and Alertmanager clustering with a gossip mesh so that alert routing does not go through a single SPOF.

# Alertmanager HA clustering — deployed via kube-prometheus-stack
# Alertmanager replicas form a gossip mesh; duplicate alerts from the HA Prometheus
# pair are deduplicated before routing to the notification channels.
alertmanager:
  alertmanagerSpec:
    replicas: 3   # gossip mesh via memberlist — no quorum/odd-count requirement; 3 gives headroom for a rolling node drain
    clusterAdvertiseAddress: ""   # Auto-detected from Pod IP
    clusterGossipInterval: "200ms"
    clusterPushpullInterval: "1m"

    # Route critical alerts to PagerDuty (high-urgency) and Slack (all severities).
    # Alertmanager config is managed as a Kubernetes Secret.
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: "fast-ssd"
          resources:
            requests:
              storage: 2Gi   # Alertmanager state is small; persistence prevents
                             # alert storm on pod restart after an incident

---
# Example PrometheusRule — SLO multi-window burn rate (Google SRE workbook)
# 14.4x on 1h + 5m windows ≈ 2% budget in the first hour (page).
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-server-slo
  namespace: observability
spec:
  groups:
    - name: slo.api-server
      interval: 30s
      rules:
        - alert: APIServerSLOFastBurn
          expr: |
            (
              rate(http_requests_total{service="api-server",code=~"5.."}[1h])
                /
              rate(http_requests_total{service="api-server"}[1h])
            ) > (14.4 * (1 - 0.999))
            and
            (
              rate(http_requests_total{service="api-server",code=~"5.."}[5m])
                /
              rate(http_requests_total{service="api-server"}[5m])
            ) > (14.4 * (1 - 0.999))
          for: 2m
          labels:
            severity: critical
            slo: api-server-availability
          annotations:
            summary: "API server fast burn rate exceeding SLO budget"
            runbook_url: "https://runbooks.internal/api-server-slo"

The Prometheus Operator's PrometheusRule CRD makes alert rule management part of the same GitOps workflow as everything else. Alert rules live in version control, reviewed in pull requests, and applied via ArgoCD or Flux. This eliminates the "who changed that alert last week" problem endemic to SaaS alerting UIs where rule history is limited or non-existent. For regulated environments, having a complete audit trail of every alert rule change — who made it, when, what the prior rule was, and what PR it was part of — is a material compliance capability, not a nice-to-have.

Audit-Grade Retention: Designing for the Incident You Have Not Had Yet

Long-term telemetry retention is not primarily an operational concern — it is a forensics and compliance concern. The incident you need 90-day-old traces to investigate is, by definition, one you did not anticipate. Designing the retention policy after an incident is too late. The architecture for audit-grade retention has three components: immutable object storage (S3 Object Lock or equivalent), retention period enforcement at the backend level (not just at the Collector), and encryption at rest with key management outside the cluster.

For Loki: enable S3 Object Lock in Compliance mode on the Loki chunks bucket (Object Lock requires S3 Versioning). With retention enabled (retention_enabled plus the required delete_request_store), Loki's compactor will generate S3 delete markers when retention expires — Object Lock prevents those from taking effect until the retain-until date. Compliance mode admits no bypass — not even the account root user can delete a locked version early; the only escape is deleting the AWS account. If you need a break-glass path for a mis-set retention, use Governance mode, which can be overridden by a principal holding s3:BypassGovernanceRetention (logged in CloudTrail). For Tempo: the same Object Lock pattern applies to the block store bucket. For Prometheus/Thanos: Object Lock is incompatible with Thanos's block lifecycle by default — copy blocks to a second bucket via aws s3 sync --storage-class GLACIER_IR on a daily schedule with independent retention.

Encryption: Loki, Tempo, and Thanos store data in S3. Enabling SSE-KMS with a KMS key whose policy allows access only to the observability stack's IAM role (via IRSA/workload identity) means that even if an attacker obtains the S3 bucket name and object keys, they cannot decrypt the data without the KMS key. Key rotation should be annual at minimum; the KMS key policy should be reviewed as part of the regular OSS supply chain audit cycle, since the IRSA role binding is a security boundary.

Putting It Together: The Self-Hosted Stack Deployment Order

For a platform team migrating from SaaS or building a new observability stack, the recommended deployment sequence minimizes blast radius at each step. (1) Deploy kube-prometheus-stack with 2-day local retention, enableRemoteWriteReceiver: true, and default ServiceMonitor selectors — cluster metrics within an hour. (2) Deploy Loki (grafana/loki, Distributed mode) with S3 buckets for chunks and ruler so a log backend exists before shippers point at it. (3) Deploy the OTel Collector DaemonSet with a filelog receiver and otlphttp/loki exporter at Loki's /otlp — structured container logs. (4) Deploy Tempo with S3 and connect the traces pipeline. (5) Migrate instrumentation service by service to OTel SDKs, dual-exporting SaaS and self-hosted during the window. (6) Enable Thanos sidecar on Prometheus for long-term S3 retention. (7) Enable exemplars and the spanmetrics connector on the Gateway — three-signal correlation comes online.

The result: a policy-as-code compliant, GitOps-managed stack where every signal flows through open standards (OTLP, PromQL, LogQL, TraceQL), components are a mix of CNCF graduated (OpenTelemetry, Prometheus), CNCF incubating (Thanos), and Grafana Labs OSS (Grafana, Loki, Tempo), and all data lives in infrastructure you control with audit-grade retention at the storage layer. Switching any backend is a Collector configuration change — not an instrumentation rewrite across the fleet.

The long-game property of this architecture is exactly the property that SaaS observability cannot provide: your telemetry data outlasts your vendor relationship. When you switch cloud providers, when your SaaS contract lapses, when the vendor is acquired — the traces and logs still inside the retention windows you set (for example 90 days for traces, ~13 months for logs and metrics) remain in your S3 bucket, queryable by any Tempo or Loki instance you stand up. Engineers investigating an incident months later can open the same Grafana Explore interface with the same TraceQL query against data you never handed to a vendor. That continuity of evidence is what engineering sovereignty means in an observability context.