Own Your Telemetry: Self-Hosted Observability on Kubernetes with OpenTelemetry, Prometheus, Loki, and Tempo

A regulated fintech we worked with was spending more on Datadog per month than on the Kubernetes infrastructure running their platform. The bill had grown silently: an agent on every node, custom metrics for SLO tracking, 30-day log retention purchased as an add-on, APM for half a dozen services, and a per-seat charge for every engineer who needed a dashboard. When the renewal notice arrived at four times the original contract value, the question the platform team brought to us was not "can we negotiate a discount" but "can we own this ourselves?" The answer was yes, with a specific architecture — and the migration, executed over six weeks, cut monthly observability spend by 80% while extending retention from 30 days to 13 months and moving all telemetry data into infrastructure the team controlled. This article is that architecture.

The self-hosted observability stack in 2026 has a clear center of gravity: OpenTelemetry as the instrumentation and collection layer, Prometheus for metrics (with Thanos or Cortex for long-term storage), Grafana Loki for logs, Grafana Tempo for distributed traces, and Grafana as the unified query and dashboard layer. Each component is a CNCF graduated project. None of them has proprietary wire formats that trap your data. All of them run comfortably on Kubernetes, deployed via mature Helm charts that are actively maintained. The ecosystem matured enough by 2025 that OpenTelemetry's collector and SDKs reached stable status across all three signal types — metrics, logs, and traces — removing the last credible technical objection to using OTel as a vendor-neutral layer over whatever backends you choose.

This is not a simple-is-better argument. A self-hosted observability stack requires more operational attention than a SaaS product, and that trade-off deserves honest examination. We will cover the architecture in enough detail to deploy it production-grade, the specific operational investments required, a named trade-off (the cardinality management problem that Prometheus inherits from its data model), and the decision boundary at which the self-hosted stack makes sense versus when SaaS is the right answer. The sovereignty thesis is not "always self-host" — it is "understand what you are renting and what you own, and make that decision deliberately."

Why Telemetry Data Belongs Inside Your Audit Perimeter

Observability data is not just operational — it is forensic evidence. When a regulated fintech's production cluster has an incident at 2 AM, the traces, logs, and metric time series from that window become the audit trail for the post-incident review, the evidence for any regulatory inquiry, and the data source for any internal investigation into whether the incident constituted a breach. Under NIS2, DORA, and sector-specific regulations like PCI DSS v4.0, the organization is responsible for the integrity and availability of that evidence — including knowing where it is stored, who has access to it, and that it has not been tampered with.

SaaS observability platforms hold that data in their infrastructure, subject to their data retention policies, their access controls, their geo-replication decisions, and — critically — their data export APIs. If the vendor changes their export API, introduces rate limits on historical data retrieval, or simply discontinues a product tier, your forensic evidence is at risk. The alternative is not necessarily harder: a Grafana Loki cluster backed by S3-compatible object storage with server-side encryption at rest, a 13-month retention policy enforced by Loki's table_manager or S3 lifecycle rules, and immutable object versioning turned on — that is a forensic-grade log store you control, auditable by any third party with read access to the bucket. The audit-grade rigor pillar in our practice is grounded in this: controls that can be independently verified produce better evidence than controls that must be trusted because they are opaque.

The OpenTelemetry Collector: One Agent to Rule Them All

The OpenTelemetry Collector is the key architectural decision in a self-hosted stack. It functions as a vendor-neutral telemetry pipeline: it receives signals in dozens of formats (OTLP, Jaeger, Zipkin, Prometheus scrape, Fluent Bit, statsd, and more), processes them (batching, filtering, attribute enrichment, sampling), and exports them to any number of backends. On Kubernetes, the canonical deployment is a DaemonSet for per-node collection (node metrics, host logs, container logs via the filelog receiver) plus a Gateway Deployment for cluster-level aggregation (receiving OTLP from all pods, applying global tail-sampling for traces, and fan-out to all backends). The separation matters: DaemonSet receivers run on every node and must be lightweight; the Gateway can apply more expensive processing like tail-sampling because it sees the full trace.

# OpenTelemetry Collector — Gateway Deployment (cluster aggregation layer)
# Deployed via the opentelemetry-operator or directly via Helm (open-telemetry/opentelemetry-collector).
# This config covers the core production pipeline: metrics, logs, and traces.
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: Deployment   # Gateway mode — receives from DaemonSet collectors
  replicas: 2        # HA: 2 replicas with HPA for traffic spikes
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317    # Receives from DaemonSet and instrumented apps
          http:
            endpoint: 0.0.0.0:4318

    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 512          # Hard limit — prevents OOM on burst ingestion
        spike_limit_mib: 128
      batch:
        send_batch_size: 8192
        timeout: 5s             # Balance latency vs efficiency
      resource:
        attributes:
          - action: upsert
            key: k8s.cluster.name
            value: "production"
      # Tail-based sampling: keep 100% of error traces, 5% of successful traces.
      # Requires the Gateway to see full traces — incompatible with DaemonSet-only mode.
      tail_sampling:
        decision_wait: 10s      # Wait up to 10s for all spans before sampling decision
        policies:
          - name: keep-errors
            type: status_code
            status_code: {status_codes: [ERROR]}
          - name: probabilistic-sample
            type: probabilistic
            probabilistic: {sampling_percentage: 5}

    exporters:
      prometheusremotewrite:
        endpoint: "http://prometheus-operated.observability.svc:9090/api/v1/write"
        tls:
          insecure: true   # mTLS via Cilium / WireGuard at node level
      loki:
        endpoint: "http://loki-gateway.observability.svc/loki/api/v1/push"
      otlp/tempo:
        endpoint: "http://tempo-distributor.observability.svc:4317"
        tls:
          insecure: true

    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, resource, batch]
          exporters: [prometheusremotewrite]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, resource, batch]
          exporters: [loki]
        traces:
          receivers: [otlp]
          processors: [memory_limiter, resource, tail_sampling, batch]
          exporters: [otlp/tempo]

The instrumentation side of the OTel equation is equally mature. For Go services, the go.opentelemetry.io/otel SDK at v1.35.0 (May 2026) provides stable APIs for traces, metrics, and logs. For Java (Spring Boot 3.x), the OpenTelemetry Java Agent — a single -javaagent JVM flag — auto-instruments JDBC, HTTP clients, Kafka consumers, gRPC, and frameworks like Micrometer with zero application code changes. For Python services, opentelemetry-sdk with opentelemetry-instrumentation-fastapi or opentelemetry-instrumentation-django covers the common cases. The key operational property: because all three signals flow through OTLP to the OTel Collector, the backend you ship them to is a Collector configuration change — not an application code change. Switching from Tempo to a different tracing backend, or adding Jaeger alongside Tempo for a migration period, is a Helm values file edit.

Click to zoom

Prometheus at Production Scale: The Cardinality Problem and How to Manage It

Prometheus is the gold standard for Kubernetes metrics — it integrates natively with the Kubernetes API via the kube-state-metrics exporter and cAdvisor, and the ServiceMonitor / PodMonitor CRDs from the kube-prometheus-stack Helm chart have become the canonical way to configure scrape targets. But Prometheus has a structural limitation that every team running it at scale encounters: high cardinality labels. Prometheus stores metrics as time series identified by a metric name plus a set of label key-value pairs. Each unique combination is a separate series. If a single metric has a user_id label with a million possible values, Prometheus must store and query a million time series for that metric. Memory consumption scales linearly with active series count; query performance degrades as the series count grows.

This is the named trade-off. Datadog and New Relic solve this with custom indexing infrastructure that can handle billions of metric series at cloud scale. Prometheus's TSDB is optimized for hundreds of millions of series, and in practice most Kubernetes clusters stay comfortably within that bound if instrumentation is done with discipline. The operational discipline required: never use user-generated values as Prometheus labels. user_id, request_id, session_token, URL path (without normalization), IP address — these are cardinality bombs. Use them as trace attributes (where Tempo stores them as indexed fields per span rather than as series dimensions) and as structured log fields in Loki. The rule of thumb: if the number of possible values for a label exceeds 1,000, it does not belong in Prometheus labels.

# kube-prometheus-stack Helm values — production-grade Prometheus configuration.
# Deploys Prometheus Operator + Prometheus + Alertmanager + Grafana + kube-state-metrics.
prometheus:
  prometheusSpec:
    replicas: 2                    # HA pair — Thanos sidecar handles deduplication
    retention: 2d                  # Keep only 2 days local; Thanos sidecar uploads to S3
    retentionSize: "50GB"          # Circuit breaker: evict oldest blocks if disk fills

    # Thanos sidecar — uploads Prometheus TSDB blocks to S3 for long-term retention.
    # Grafana queries Thanos Query for unified local + remote data.
    thanos:
      image: quay.io/thanos/thanos:v0.36.1
      objectStorageConfig:
        secret:
          type: S3
          config:
            bucket: "prod-metrics-longterm"
            endpoint: "s3.eu-central-1.amazonaws.com"
            region: "eu-central-1"
            access_key: ""       # Use IRSA / workload identity — never static keys
            secret_key: ""

    # Resource limits — size based on expected series count.
    # 1M active series ~ 8-10 GB RAM at steady state (Prometheus 2.x).
    resources:
      requests:
        cpu: "1"
        memory: "8Gi"
      limits:
        cpu: "4"
        memory: "12Gi"

    # Cardinality guardrail: scrape interval 60s for most targets (not 15s).
    # 15s intervals 4x ingestion rate with minimal operational gain for most SLOs.
    scrapeInterval: "60s"
    evaluationInterval: "60s"

alertmanager:
  alertmanagerSpec:
    replicas: 2
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: "fast-ssd"
          resources:
            requests:
              storage: 10Gi

Grafana Loki: Log Aggregation Without the Elasticsearch Tax

Loki is architecturally different from Elasticsearch-based log stacks in a way that has significant operational consequences. Rather than full-text indexing every log field at ingestion time, Loki indexes only the stream labels — the same label model as Prometheus — and stores the raw log content as compressed chunks in object storage. Queries (LogQL) filter by labels first to identify the relevant chunk set, then scan those chunks. This means: (1) ingestion is cheap — no per-field analysis at write time, just label extraction and compression; (2) storage is cheap — compressed log chunks in S3 at object-storage prices rather than Elasticsearch's hot-warm-cold node tiers; (3) query performance on high-cardinality label sets is poor — the same cardinality constraint from Prometheus applies. The design principle: Loki labels should identify the source (pod, namespace, application, environment), not the content of individual log lines.

# Loki Helm values — production scalable mode (microservices).
# loki-distributed chart: separate Distributor, Ingester, Querier, Compactor, Index Gateway.
# Object-store backed (S3) — no local disk dependency for data.

loki:
  auth_enabled: false      # Single-tenant; enable for multi-tenant with X-Scope-OrgID

  commonConfig:
    replication_factor: 3  # 3 ingesters hold a write quorum

  storage:
    type: s3
    s3:
      region: eu-central-1
      bucketnames: "prod-loki-chunks"
      endpoint: ""         # Default AWS endpoint; override for MinIO / Ceph

  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb         # TSDB index (Loki >= 2.8) — more efficient than boltdb-shipper
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

  limits_config:
    retention_period: 8760h    # 365 days — set per-tenant for multi-tenant
    ingestion_rate_mb: 32
    ingestion_burst_size_mb: 64
    max_streams_per_user: 10000
    max_label_names_per_series: 15   # Cardinality guardrail: reject streams with > 15 labels

  compactor:
    working_directory: /data/compactor
    shared_store: s3
    retention_enabled: true       # Enables per-tenant retention enforcement

# Ship container logs via OTel Collector filelog receiver (preferred over Promtail in 2026)
# or via Promtail DaemonSet for teams that prefer a Loki-native agent.
# OTel filelog receiver is preferred: it emits OTLP and can send to any backend,
# avoiding a Loki-only agent dependency.

Log shipping in 2026 is predominantly via the OTel Collector's filelog receiver (DaemonSet mode), which tails /var/log/containers/*.log, parses the container log format, and emits structured log records as OTLP. The OTel Collector then exports to Loki via the loki exporter, which derives Loki stream labels from OTLP resource attributes (k8s.namespace.name, k8s.pod.name, k8s.container.name). The advantage over Promtail — Loki's native log shipper — is backend agnosticism: if you add Elasticsearch to the stack later (for full-text search on security logs, for example), the OTel Collector adds an Elasticsearch exporter without changing the shipping agent on every node.

Grafana Tempo: Distributed Tracing at Object-Store Prices

Grafana Tempo stores distributed traces as blocks in object storage — S3, GCS, Azure Blob, or MinIO — with no local disk dependency beyond a small write-ahead log for in-flight traces. This makes Tempo's storage cost a direct function of trace volume and retention period at object-storage pricing ($0.02–0.023/GB/month on AWS S3 standard, ~$0.004/GB/month on S3 Glacier Instant Retrieval). A typical regulated-environment retention of 90 days of traces at 5% sampling of all requests produces roughly 200–400 GB of Tempo storage per month for a platform handling 10,000 requests per second — at approximately $4–9/month in S3 costs. For comparison, the Datadog APM price for the same throughput at comparable retention is measured in thousands of dollars per month.

Tempo's query language is TraceQL, which reached GA in Tempo 2.3 (2024) and is now stable and expressive enough for production investigation. A TraceQL query that finds all traces where the api-server service had spans exceeding 500ms and downstream calls to a database service looks like this:

# TraceQL — find traces where api-server has slow database spans
{
  resource.service.name = "api-server" &&
  span.db.system = "postgresql" &&
  duration > 500ms
}

# Find all error traces across the entire platform in the last hour
{
  status = error
}

# Find traces where a specific user-facing endpoint exceeded SLO (2s)
{
  resource.service.name = "api-server" &&
  name = "POST /api/v1/orders" &&
  duration > 2s
}

# Structural query: find traces where api-server called payments-service
# and that downstream span had an error
{
  resource.service.name = "api-server" &&
  .span.child[].resource.service.name = "payments-service" &&
  .span.child[].status = error
}

The integration between Tempo and Prometheus via exemplars is the sharpest workflow in the self-hosted stack. When a Go service records a histogram observation (for example, HTTP request duration), it can attach the current TraceID as an exemplar to the histogram sample. Prometheus stores exemplars alongside metrics. In Grafana, a spike in the http_request_duration_seconds_bucket panel surfaces exemplar dots on the graph; clicking a dot jumps directly to the Tempo trace for that request — the one that was actually slow. This three-signal correlation path (dashboard alert -> metric spike -> exemplar -> trace -> log context) is the investigation workflow that replaces both the Datadog APM flame graph and the Datadog log search in a single Grafana Explore session.

Click to zoom

Grafana: Unified Query Without Proprietary Dashboards

Grafana 12 (released April 2026) solidified the unified observability experience that earlier versions promised but partially delivered. The key features for a self-hosted stack: Grafana Scenes for parameterized, drillable dashboard composition; Explore as a multi-signal, multi-datasource investigation workspace; Alert rules backed by PromQL, LogQL, or TraceQL (with a unified alerting engine that routes to Alertmanager, PagerDuty, Slack, or any webhook target); and Grafana OnCall for schedule management and escalation policies if you want a fully self-hosted incident response stack.

The dashboard portability argument in favor of self-hosting Grafana is concrete: Grafana dashboard JSON is a version-controlled artifact. A dashboard created today for kube-prometheus-stack metrics will work in three years on the next major Prometheus version, because the underlying PromQL queries are portable across any Prometheus-compatible backend (including Thanos, Mimir, and Victoria Metrics). Datadog dashboards are JSON too, but they reference Datadog's proprietary metric naming scheme, DDL filter syntax, and widget types that have no equivalent elsewhere. The exit cost compounds with every dashboard built in the proprietary format.

# Import the kube-prometheus-stack dashboards and the Kubernetes / USE Method dashboards
# into Grafana. These ship as ConfigMaps via the Helm chart and are auto-provisioned.

# Grafana dashboard provisioning — sidecar approach (grafana/grafana Helm chart default).
# Place dashboard JSON files in a ConfigMap labeled with:
#   grafana_dashboard: "1"
# The Grafana sidecar scrapes these ConfigMaps and imports them automatically.

kubectl create configmap grafana-slo-dashboard \
  --from-file=slo-dashboard.json \
  --namespace observability

kubectl label configmap grafana-slo-dashboard \
  grafana_dashboard=1 \
  --namespace observability

# Verify the dashboard was imported
kubectl logs -n observability -l app.kubernetes.io/name=grafana \
  -c grafana-sc-dashboard | grep "slo-dashboard"

# Force a Grafana reload without restart (Grafana 9+ hot-reload API)
kubectl exec -n observability -it \
  $(kubectl get pod -n observability -l app.kubernetes.io/name=grafana -o name | head -1) \
  -- curl -s -X POST http://localhost:3000/api/admin/provisioning/dashboards/reload \
  -H "Authorization: Bearer $GRAFANA_SERVICE_ACCOUNT_TOKEN"

The multi-cluster GitOps pattern applies cleanly here: the observability stack (Helm releases for kube-prometheus-stack, Loki, Tempo, and Grafana) is a set of FluxCD HelmRelease objects in a dedicated platform/observability/ directory in your fleet repository. When a new cluster is provisioned, Flux reconciles the observability stack automatically — the platform team does not have to manually install anything. Alerting rules and Grafana dashboards are ConfigMaps in the same directory. The entire observability stack is version-controlled, reviewed, and deployed identically across all clusters.

The SaaS vs Self-Hosted Decision: Where Each Side Wins

The honest framing: self-hosted observability is the right choice for a specific set of requirements, not universally. Understanding the decision boundary prevents both the reflexive "always SaaS" bias that inflates costs and the reflexive "always self-host" bias that wastes engineering time.

Click to zoom

Self-hosted wins when: (1) Data sovereignty is a hard requirement — regulated industries (fintech, healthcare, defense) where data leaving your control requires legal review or is prohibited outright. (2) Scale has made SaaS costs non-linear — at approximately 5–10 million active Prometheus series, SaaS pricing commonly exceeds the TCO of a self-hosted stack operated by a half-FTE. (3) Retention periods exceed 30 days — SaaS default retention is 15–30 days; extending it to 90 days or 13 months on most platforms costs a multiple of the base contract. (4) Optionality is a strategic requirement — when the organization's plan includes potential migration across cloud providers, the proprietary SaaS query language and dashboard format become an exit barrier.

SaaS wins when: (1) The team has fewer than five engineers and cannot afford a half-FTE to operate and tune the observability stack. (2) Time-to-first-insight is the primary metric — SaaS platforms ship with pre-built APM, infrastructure dashboards, and anomaly detection that a self-hosted stack requires weeks to configure equivalently. (3) The workload is ephemeral — short-lived experiments, hackathons, or MVPs where the platform will be shut down before long-term retention matters. The optionality pillar does not mean building everything yourself; it means preserving the ability to change your choice without catastrophic exit costs. For observability, that means using OpenTelemetry for instrumentation even if you ship to Datadog today — because the instrumentation is vendor-neutral, switching backends later is a configuration change, not a code change.

Alerting That Survives Incident Conditions

An alerting stack that goes silent during the incident it should be alerting on is worse than no alerting at all. The classic failure mode in self-hosted stacks: Prometheus is the source of truth for cluster health, but Prometheus depends on the cluster. When etcd has a split-brain event or a node running a Prometheus pod fails, Prometheus itself may be degraded at the moment you most need it. The mitigation requires two patterns working together: HA Prometheus pairing with deduplication at the Alertmanager level, and Alertmanager clustering with a gossip mesh so that alert routing does not go through a single SPOF.

# Alertmanager HA clustering — deployed via kube-prometheus-stack
# Alertmanager replicas form a gossip mesh; duplicate alerts from the HA Prometheus
# pair are deduplicated before routing to the notification channels.
alertmanager:
  alertmanagerSpec:
    replicas: 3   # Odd number for quorum; minimum 3 for split-brain resilience
    clusterAdvertiseAddress: ""   # Auto-detected from Pod IP
    clusterGossipInterval: "200ms"
    clusterPushpullInterval: "1m"
    clusterSettleTimeout: "1m"

    # Route critical alerts to PagerDuty (high-urgency) and Slack (all severities).
    # Alertmanager config is managed as a Kubernetes Secret.
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: "fast-ssd"
          resources:
            requests:
              storage: 2Gi   # Alertmanager state is small; persistence prevents
                             # alert storm on pod restart after an incident

---
# Example PrometheusRule — SLO burn rate alert using the Google SRE model
# (fast burn rate = high severity, slow burn rate = warning).
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-server-slo
  namespace: observability
spec:
  groups:
    - name: slo.api-server
      interval: 30s
      rules:
        # 1h burn rate > 14.4x budget consumption rate = critical page
        - alert: APIServerSLOFastBurn
          expr: |
            (
              rate(http_requests_total{service="api-server",code=~"5.."}[5m])
                /
              rate(http_requests_total{service="api-server"}[5m])
            ) > (14.4 * (1 - 0.999))
          for: 2m
          labels:
            severity: critical
            slo: api-server-availability
          annotations:
            summary: "API server fast burn rate exceeding SLO budget"
            runbook_url: "https://runbooks.internal/api-server-slo"

The Prometheus Operator's PrometheusRule CRD makes alert rule management part of the same GitOps workflow as everything else. Alert rules live in version control, reviewed in pull requests, and applied via ArgoCD or Flux. This eliminates the "who changed that alert last week" problem endemic to SaaS alerting UIs where rule history is limited or non-existent. For regulated environments, having a complete audit trail of every alert rule change — who made it, when, what the prior rule was, and what PR it was part of — is a material compliance capability, not a nice-to-have.

Audit-Grade Retention: Designing for the Incident You Have Not Had Yet

Long-term telemetry retention is not primarily an operational concern — it is a forensics and compliance concern. The incident you need 90-day-old traces to investigate is, by definition, one you did not anticipate. Designing the retention policy after an incident is too late. The architecture for audit-grade retention has three components: immutable object storage (S3 Object Lock or equivalent), retention period enforcement at the backend level (not just at the Collector), and encryption at rest with key management outside the cluster.

For Loki: enable S3 Object Lock in Compliance mode on the Loki chunks bucket. Loki's compactor will generate S3 delete markers when retention expires — Object Lock prevents those from taking effect until the retention period elapses. This means even a compromised cluster operator cannot delete logs before retention expires without explicit S3 Object Lock bypass — which requires the S3 account's root credentials and leaves an audit trail in CloudTrail. For Tempo: the same Object Lock pattern applies to the Tempo block store bucket. For Prometheus/Thanos: Thanos uses S3-native delete operations to expire old metric blocks — Object Lock is incompatible with Thanos's block lifecycle by default. The alternative: a second S3 bucket (or S3 Glacier vault) that receives Thanos block copies via aws s3 sync --storage-class GLACIER_IR on a daily schedule, with independent retention enforcement.

Encryption: Loki, Tempo, and Thanos store data in S3. Enabling SSE-KMS with a KMS key whose policy allows access only to the observability stack's IAM role (via IRSA/workload identity) means that even if an attacker obtains the S3 bucket name and object keys, they cannot decrypt the data without the KMS key. Key rotation should be annual at minimum; the KMS key policy should be reviewed as part of the regular OSS supply chain audit cycle, since the IRSA role binding is a security boundary.

Putting It Together: The Self-Hosted Stack Deployment Order

For a platform team migrating from SaaS or building a new observability stack, the recommended deployment sequence minimizes blast radius at each step and gives you working signal before the previous layer is perfect. (1) Deploy kube-prometheus-stack with a 2-day local retention and default ServiceMonitor selectors — you have cluster and workload metrics within an hour. (2) Deploy the OTel Collector DaemonSet with a filelog receiver and a Loki exporter — you have structured container logs. (3) Deploy Loki in scalable mode with S3 backend — connect the OTel Collector exporter. (4) Deploy Tempo with S3 backend and connect the OTel Collector's traces pipeline. (5) Migrate instrumentation service by service from vendor agents to OTel SDKs, using the Collector's dual-export capability to run SaaS and self-hosted in parallel during the migration window. (6) Enable Thanos sidecar on Prometheus and configure long-term S3 retention with Object Lock. (7) Enable exemplar scraping in Prometheus and the spanmetrics connector in the OTel Collector Gateway — the three-signal correlation path comes online.

The result at the end of this sequence: a policy-as-code compliant, GitOps-managed observability stack where every signal flows through open standards (OTLP wire format, PromQL, LogQL, TraceQL), every component is a CNCF graduated project, and all data is stored in infrastructure you control with audit-grade retention enforced at the storage layer. The exit cost, if you need to switch any individual component, is a Collector configuration change — not an instrumentation rewrite across your entire application fleet.

The long-game property of this architecture is exactly the property that SaaS observability cannot provide: your telemetry data outlasts your vendor relationship. When you switch cloud providers, when your SaaS contract lapses, when the vendor is acquired — the traces and logs from three years of production operation remain in your S3 bucket, queryable by any Tempo or Loki instance you stand up. The engineers who investigate the incident in 2029 can read the trace from 2026 in the same Grafana Explore interface with the same TraceQL query. That continuity of evidence is what engineering sovereignty means in an observability context.