Observability vs Monitoring When You Run the Backend

Most comparisons stop at known unknowns versus unknown unknowns. That is accurate, and it does not tell an operator what to deploy, what to drop or how long to keep anything. This article draws the line where it costs something.

The distinction everyone draws, and why it tells an operator nothing

The canonical definition of monitoring is still the one in the 2016 Google SRE book: collecting, processing, aggregating and displaying real-time quantitative data about a system — query counts and types, error counts and types, processing times, server lifetimes. The same chapter says dashboards should normally include some form of the four golden signals. Monitoring, in that framing, is a set of questions you decided to ask in advance.

OpenTelemetry's observability primer defines the other side: observability lets you understand a system from the outside by asking questions about it without knowing its inner workings, and lets you troubleshoot novel problems — the “unknown unknowns.” Both definitions are about questions.

The operator's problem is different. A question you have not asked yet can only be answered from data you kept; a pre-aggregated counter cannot say which build or pod produced last Tuesday's slow tail. So the practical question is not “do we want observability?” but how much high-dimensional data to retain, for how long, at what resolution — and who pays for the disk.

Where the line actually falls: what you pay to keep

Classify a signal by its storage behaviour. A monitoring signal is bounded: a fixed series set per service, aggregated at source, cheap to keep, safe to alert on. An observability signal is unbounded by design: it carries identifiers — trace IDs, request paths, user or tenant attributes, build hashes — precisely because you might need to slice by them later.

Prometheus states the mechanism plainly in its naming guidance: every unique combination of key-value label pairs represents a new time series, and labels must not store high-cardinality dimensions such as user IDs, email addresses or other unbounded sets. Loki says the same about streams: its label best practices recommend labels for bounded values like regions, clusters, namespaces and environments, and warn that too many label value combinations create too many streams — a large index and small chunks. Loki's cardinality page lists pod names, IP addresses, user IDs and trace IDs as typical high-cardinality attributes.

SaaS vendors bill on exactly this axis. Datadog's custom-metrics billing page identifies a custom metric as a unique combination of metric name and tag values, including the host tag, and, under its cardinality-based model, bills the monthly average of the hourly count of distinct custom metrics (some contracts use metric-name pricing; Metrics without Limits bills ingested and indexed volumes separately, each as that average); a HISTOGRAM or DISTRIBUTION metric generates five custom metrics per combination by default. Grafana Cloud bills metrics on active series and data points per minute, both measured at the 95th percentile over the billing period. Those pages describe the billing model, not your contract.

The point is not price: the vendor's billing unit is the quantity your own TSDB pays for in memory, index and object storage. Self-hosting moves it from an invoice line to a config file you version, review and set per tenant.

Click to zoom

The bridge between the two paths is the exemplar: a reference, typically a trace ID, attached to a metric sample. Prometheus stores them only behind a feature flag, --enable-feature=exemplar-storage. If your dashboards are meant to jump from a latency spike to the trace behind it, that flag is part of the design, not a default.

Decision one: a cardinality budget you enforce

Why label explosions happen — and how recording rules and relabelling tame them — is covered in the build guide for the OpenTelemetry, Prometheus, Grafana and Loki stack. This section covers what a vendor's invoice used to do: measuring the budget, and making a breach fail loudly.

Measure first

Prometheus exposes cardinality statistics at GET /api/v1/status/tsdb (HTTP API), including seriesCountByMetricName and labelValueCountByLabelName. It returns 10 items per statistic by default; limit raises that to a maximum of 10000. Offline, `promtool tsdb analyze` reports churn, label pair cardinality and compaction efficiency against a data directory; its own --limit flag defaults to 20. Two tools, two different defaults — set both explicitly.

#!/usr/bin/env bash
set -euo pipefail

PROM_URL="${PROM_URL:-http://localhost:9090}"

# Live head block: top 20 metric names by series, top 20 labels by distinct values
curl -fsS "${PROM_URL}/api/v1/status/tsdb?limit=20" \
  | jq '{series: .data.seriesCountByMetricName, labels: .data.labelValueCountByLabelName}'

# Persisted blocks: churn, label-pair cardinality, compaction efficiency
promtool tsdb analyze --limit=20 /prometheus/data

Read the label list first. A label with thousands of values is usually a leaked identifier — a request ID, a raw URL path, a session token. That is the monitoring/observability line drawn in the wrong place: the identifier belongs on a span or a log line, not a metric.

Enforce at scrape time

The scrape configuration gives you two hard limits and one surgical tool. sample_limit caps samples accepted per scrape after metric relabelling; label_limit caps labels per sample. Both default to 0, meaning no limit, and exceeding either causes the entire scrape to be treated as failed. labeldrop removes every label whose name matches a regex.

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: checkout-api
    # Budget, not a measurement: size it from the TSDB status output above.
    sample_limit: 20000
    label_limit: 30
    static_configs:
      - targets: ["checkout-api.example.internal:9464"]
    metric_relabel_configs:
      # Identifiers belong on spans and log lines, never on metric labels.
      - action: labeldrop
        regex: "request_id|session_id|user_email"

The failure mode is deliberate. A team that adds an unbounded label gets not a slowly growing bill but a whole failed scrape, visible as up == 0 for that target. For a monitoring signal that is the right trade — a loud, attributable failure beats a quiet one in shared storage — but the limit needs an alert, an owner and headroom over the measured baseline. Treat labeldrop as a stopgap: if the dropped label distinguished two series, they now collide. Remove the identifier in the instrumentation.

Behind a multi-tenant store, the same budget exists per tenant. Grafana Mimir's `max_global_series_per_user` limits in-memory series per tenant across the cluster, before replication, and defaults to 150000 (0 disables it). That default is a vendor-documented starting point, not a sizing recommendation; set it per tenant through overrides so one team's label leak cannot consume another team's headroom.

Decision two: sampling, and the collector topology it forces

Traces are where observability gets expensive, so sampling is where it gets decided. OpenTelemetry's sampling concepts draw the line cleanly. Head sampling decides as early as possible, without inspecting the whole trace — cheap and stateless, but it cannot guarantee that every trace containing an error is kept. Tail sampling decides by considering all or most of the spans in a trace, so it can keep every error and every slow request. The cost is stated as plainly: tail-sampling components must be stateful and store a lot of data, which depending on traffic can mean dozens or even hundreds of compute nodes.

A single-replica tail-sampling gateway — errors plus a small baseline, with RED metrics derived alongside — is the starting configuration in the self-hosted observability build guide. The next step, when one replica is no longer enough, is where the topology changes.

The tail sampling processor README is explicit: all spans for a given trace must be received by the same collector instance, and the processor keeps spans in memory while it waits to decide. So replicas cannot sit behind a round-robin balancer. The README recommends two layers — one running the load balancing exporter, one running the tail sampling processor — kept separate for failure isolation, and the gateway deployment docs describe the same shape.

Click to zoom

Two placement decisions matter. First, compute RED metrics in tier 1 with the span metrics connector, which aggregates request, error and duration metrics from span data. It counts spans per dimension set, not requests: a trace with client, server and database spans adds three, so a request rate is the server-span series only. Placed before sampling, those counts cover every span; placed after, your error rate would describe only the traces the sampler chose to keep. Second, route tier 1 to tier 2 with the load balancing exporter, whose default routing key for traces is traceID. Its resolvers are static, dns, k8s and aws_cloud_map; the example below uses DNS against a headless Service, which is one option among four.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  # Each tier-1 replica emits its own copy of every RED series. A bounded
  # per-replica label keeps them distinct in the metrics store; it replaces
  # the connector's per-process collector.instance.id UUID (excluded below).
  transform/replica:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["collector_replica"], "${env:POD_NAME}")
  batch: {}

connectors:
  span_metrics:
    histogram:
      explicit:
        buckets: [5ms, 25ms, 100ms, 250ms, 1s, 5s]
    exclude_dimensions: ['collector.instance.id']
    dimensions:
      - name: http.request.method
      - name: http.response.status_code

exporters:
  load_balancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-tail-sampling-headless.observability.svc.cluster.local
        port: 4317
  prometheusremotewrite:
    endpoint: http://mimir-distributor.observability.svc:8080/api/v1/push
    headers:
      X-Scope-OrgID: platform

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [span_metrics, load_balancing]
    metrics/red:
      receivers: [span_metrics]
      processors: [memory_limiter, transform/replica, batch]
      exporters: [prometheusremotewrite]

Two notes on that file. The load balancing exporter was renamed from loadbalancing to load_balancing, and the span metrics connector from spanmetrics to span_metrics; the old names remain deprecated aliases, and an older pinned distribution may only know them. And span_metrics dimensions are cardinality too: every metric always carries service.name, span.name, span.kind, status.code and collector.instance.id, and configured dimensions multiply on top. A span named after a raw URL path explodes the series through span.name even though only method and status code were budgeted. collector.instance.id is a per-process UUID that keeps each replica a single writer; the file excludes it because collector_replica does that job under a queryable name. After Prometheus name translation, a request rate under the default namespace is sum without (collector_replica) (rate(traces_span_metrics_calls_total{span_kind="SPAN_KIND_SERVER"}[5m])) — leave the UUID in and that sum folds nothing. POD_NAME comes from the Kubernetes downward API.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15
  tail_sampling:
    decision_wait: 30s
    num_traces: 50000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1500
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch: {}

exporters:
  # Newer Collector releases also accept the name otlp_grpc.
  otlp/tempo:
    endpoint: tempo-distributor.observability.svc:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]

decision_wait and num_traces are written out at their documented defaults — 30s and 50000 — because they size tier-2 memory, and a default nobody can see is one nobody reviews. The file leaves sampling_strategy at its default. Policies combine as a vote: a trace any policy samples is kept unless another explicitly drops it. The 5% baseline and 1500ms threshold are illustrative; set them from your SLOs. insecure: true assumes a mesh or network policy provides in-cluster transport security — otherwise configure TLS.

Decision three: retention tiers, not one retention number

Every store ships with a retention answer, and none of them is a policy. Prometheus defaults to 15d of local retention when neither --storage.tsdb.retention.time nor --storage.tsdb.retention.size is set, and the size limit is disabled by default. Tempo's compaction `block_retention` defaults to 336h, fourteen days. Thanos is the one that will surprise you: by default it sets no retention at all on object-storage data.

The basic split — short local Prometheus retention, long-term blocks in Thanos, a Loki compactor with retention enabled — is in the build guide. What it treats as one number per store is really several tiers inside each store.

Metrics: retention per resolution

The Thanos compactor creates 5m downsampled data for blocks older than 40 hours and 1h downsampled data for blocks older than 10 days, and sets retention per resolution with --retention.resolution-raw, --retention.resolution-5m and --retention.resolution-1h. Not setting them, or setting them to 0d, means no retention. Raw samples answer questions about recent incidents; hourly aggregates answer trend questions for years.

thanos compact \
  --data-dir=/var/thanos/compact \
  --objstore.config-file=/etc/thanos/objstore.yml \
  --retention.resolution-raw=30d \
  --retention.resolution-5m=180d \
  --retention.resolution-1h=730d \
  --wait

Logs: retention per stream

Loki's retention is set globally with retention_period and per log-stream selector with retention_stream; each retention_stream period must be at least 24h. The larger the priority value, the higher the priority, and if two matching rules tie, Loki applies the shorter period. Retention also only works when the index period is 24h. The compliance floor — which records must be kept, and proving deletion — is in the records-retention post. Here the angle is cost: debug streams are high-volume and rarely queried, audit streams are small and must outlive the incident.

# excerpt: not a complete Loki config (no server, storage or compactor sections)
schema_config:
  configs:
    - from: "2026-01-01"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h   # retention requires a 24h index period
# ...
limits_config:
  retention_period: 744h
  retention_stream:
    # Higher priority wins, so audit debug lines still get the audit period.
    - selector: '{namespace="audit"}'
      priority: 2
      period: 8760h
    - selector: '{level="debug"}'
      priority: 1
      period: 72h
# ...

Note what the selector needs: a level label. Loki's label guidance discourages a dynamic level label in favour of filtering the line, but retention_stream selectors accept only label matchers — so the label is a deliberate cardinality spend, bought for a cheaper retention tier. That is the whole article in one line of YAML.

Cloud, data and AI observability: three different jobs under one word

The word is stretched across disciplines that share little beyond the name. Cloud observability usually means the same telemetry problem on someone else's storage and billing unit; the knobs become pricing tiers rather than config. If you are weighing the move between the two, what leaving Datadog, Splunk, Grafana Cloud or Elastic costs is the companion piece.

Data observability is a different subject: the health, quality and lineage of datasets and pipeline jobs, not request paths through services. OpenLineage, an open framework for data lineage collection, models datasets, jobs and runs — useful, and not something a trace backend answers.

AI observability for model-serving and agent workloads sits closer to service telemetry, but the conventions are still moving: OpenTelemetry's GenAI semantic conventions have moved out of the main semantic-conventions docs into a dedicated repository. Treat prompt, model and token attributes as new cardinality to budget, and treat prompt content as data with its own retention and privacy decision.

Failure modes of running it yourself

• The memory limiter drops data you assumed was queued. The memory_limiter refuses data once memory exceeds its soft limit, and data is permanently lost if the component before it does not retry. SDK exporters and tier-1 senders need retry enabled, and refused-data metrics need an alert.
• The tail sampler runs out of memory, or out of patience. With defaults of num_traces 50000 and decision_wait 30s, a traffic burst fills the buffer, and past num_traces the oldest trace is dropped before any policy runs — error or not — counted in sampling_trace_dropped_too_early. An error span arriving after a trace's decision does not reverse it. Size tier 2 from peak arrival rate times the wait, and alert on that metric.
• Rebalancing splits traces. When a tier-2 replica is added or removed, the trace-ID hash ring changes and in-flight traces can land on two replicas — expect partial traces around scaling events, and avoid autoscaling tier 2 on short intervals.
• Thanos retention silently stops. The docs warn that retention is applied right after compaction and downsampling loops; if those fail, data is never deleted. A halted compactor is a storage-growth and a compliance incident at once, so alert on it.
• Loki retention does nothing. Without retention_enabled: true the compactor only compacts tables, and without a 24h index period retention is unavailable. The config validates; the data stays.

Audit evidence and the exit ramp

Because the three decisions are configuration, they are also evidence. A reviewer asking how long logs from a regulated namespace are kept can be shown the retention_stream rule, its priority, the 24h index period and the compactor alert, from version control with history. The same holds for sampling — the retained fraction, and that error traces are kept when the status_code policy sees the error at decision time — and for per-tenant cardinality limits. For audit-log pipelines with jurisdictional constraints, the Kubernetes audit logs to Wazuh post covers the retention evidence end to end.

The exit ramp is the formats. OTLP instrumentation, Prometheus exposition format and blocks in object storage you control make the backend replaceable: swap Tempo, Mimir or Loki for another OTLP-capable store and the instrumentation, collector tiers and sampling policy survive. Kernel-level signals follow the same rule — eBPF profiling with Pyroscope adds a fourth signal without adding a vendor agent.

The long game: decide what to keep before you need it

Size monitoring for the alerts you run today: bounded series, short raw retention, long downsampled history. Size observability for next year's questions: traces kept by policy rather than luck, identifiers on spans and log lines rather than metric labels, and retention tiers a reviewer can read. The organisations that keep owning their telemetry are the ones that write these three decisions down before the incident that tests them.