Pyroscope and eBPF: Profiling Without a Vendor Agent

Most organisations that repatriated their observability stack did it signal by signal, and stopped at three. Profiles — the fourth signal, the one that says which function burns the CPU you pay for — are still bought as a black-box SaaS agent running on your nodes with more privilege than most of your own workloads.

The agent you never audited

Start with the boundary, where this argument usually goes wrong. A commercial APM product is not one thing: its tracing SDK links into your process and sees request-level data; its profiler only samples stacks. Conflating them turns a sound sovereignty argument into an unfair accusation.

The vendors document the tracing side plainly. Datadog's data-security page states that “the query string could contain sensitive data, so by default Datadog parses it and redacts suspicious-looking values”. Redaction is heuristic, collection is opt-out via DD_HTTP_SERVER_TAG_QUERY_STRING — request-level data leaving the host by default, governed by an environment variable most teams never set.

A whole-system eBPF profiler is a different animal: its raw material is stack addresses and symbol names, function identities rather than payloads. The honest argument is not that profilers exfiltrate query strings. It is narrower and harder to dismiss: profiling requires the most privileged DaemonSet in your estate, reading process memory layout across every container on every node. Who operates it, where its output lands, on whose retention schedule: the questions we worked through for runtime threat detection and the dataplane. Profiles are the third eBPF vertical.

What an eBPF profiler actually samples

The kernel's perf subsystem raises an event at a fixed frequency; an eBPF program attached to it walks the stack of whatever was running and writes the frame addresses into a map. Userspace drains it, resolves addresses to function names, and aggregates identical stacks.

It is statistical, not exhaustive: a function that never runs while the timer fires never appears. That is a feature: cost scales with sampling frequency, not request volume, so overhead is bounded by a number you choose.

It is also whole-system — no per-service opt-in, no SDK, no redeploy. The profiles alpha announcement promises “low-overhead whole-system continuous profiling on Linux with support of the most widely-used language runtimes without any additional instrumentation”, including the processes nobody instrumented.

What it does not give you is a trace: no spans, no parent-child relationships, no request identity. Profiles answer “where did CPU time go on this machine, over this window”, and answer it well. Ask which request was slow and they say nothing.

The kernel floor and the unwinding tax

Two prerequisites decide this, both properties of your nodes, not your applications. First, the kernel. The upstream README is specific: “Commit 8047150e was the last to support kernel version 5.4. Subsequent changes may require a minimal Linux kernel version of 5.10 or greater.” That floor has teeth for anyone still on RHEL 8, which shipped a 4.18 kernel at GA while RHEL 9.0 shipped 5.14. Unsupported is not impossible — Red Hat backports eBPF features, and both the profiler and Alloy expose a no_kernel_version_check escape hatch — but treat a sub-5.10 node as ineligible until tested.

Second, stack unwinding. Without frame pointers or unwind tables the profiler cannot walk past the top frame, and you get flamegraphs two frames deep that look like a bug, not a compilation choice. Distributions have been moving: Fedora added -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer to its default C/C++ flags in Fedora 38, and Ubuntu followed for 64-bit platforms in 24.04 LTS, putting the penalty at “between 1-2% in most cases”. Not a blanket guarantee — where impact is high, “such as the Python interpreter”, Canonical still omits them. The profiler compensates where it can, claiming support for “profiling of system libraries without frame pointers and without debug symbols on the host”. Your own binaries remain your problem.

#!/usr/bin/env bash
# Whole-system eBPF profiling: is this node eligible?
# Exit 0 = eligible; 1 = fails a stated requirement.
set -euo pipefail

MIN_KERNEL=5.10
verdict=0

kver=$(uname -r | cut -d- -f1)
if [ "$(printf '%s\n' "$MIN_KERNEL" "$kver" | sort -V | head -n1)" != "$MIN_KERNEL" ]; then
  echo "FAIL  kernel $kver below the stated minimum $MIN_KERNEL."
  verdict=1
else
  echo "OK    kernel $kver meets the stated $MIN_KERNEL minimum."
fi

# perf_event sampling needs CAP_PERFMON (5.8+) or CAP_SYS_ADMIN,
# or perf_event_paranoid below 1.
paranoid=$(cat /proc/sys/kernel/perf_event_paranoid)
if [ "$paranoid" -ge 1 ]; then
  echo "WARN  perf_event_paranoid=$paranoid — grant the capability instead."
fi

# Path A mounts /sys/kernel/tracing.
if [ -d /sys/kernel/tracing ]; then
  echo "OK    /sys/kernel/tracing present."
else
  echo "FAIL  no tracefs mount point on this node."
  verdict=1
fi

exit "$verdict"

Two paths to the same Pyroscope

There is not one eBPF profiler for Pyroscope but two supported collection paths, documented on separate pages that never cross-reference each other — which is why mixed configurations are the most common failure here. Pyroscope's README names both: profiles reach the server via the SDKs, “Grafana Alloy (pull or push), or OTLP from OpenTelemetry-compatible sources such as the OpenTelemetry eBPF profiler”. They share lineage: Alloy's reference states that “the pyroscope.ebpf component embeds the grafana/opentelemetry-ebpf-profiler which is a fork of open-telemetry/opentelemetry-ebpf-profiler”. Unwinder behaviour and language support are broadly common; nothing else is.

• Agent — Alloy's pyroscope.ebpf component, versus the otel/opentelemetry-collector-ebpf-profiler distribution with a profiling receiver.
• Stability — pyroscope.ebpf is *General availability*. The Collector path rides the profiles signal, Development in the OTLP specification while the other three are Stable, whose alpha announcement says it “should not be used for critical production workloads”.
• Privilege — seven documented capabilities, versus privileged: true in every published example for the Collector image.
• Wire protocol — Pyroscope's HTTP ingest via pyroscope.write, versus OTLP gRPC on port 4040.
• Labelling — Alloy sets service_name in relabeling rules you write; the Collector profiler needs a server-side ingestion rule.
• Feature gate — none on Path A; the Collector needs --feature-gates=+service.profilesSupport or the profiles pipeline does not exist.

Click to zoom

Path A: Alloy, and least privilege instead of privileged: true

Grafana is direct: “you must run Alloy as root and inside the host PID namespace”, that the simplest Kubernetes option is securityContext.privileged: true, and that “users who prefer least-privilege can instead grant the specific capabilities required”.

The set is seven, each documented: BPF loads programs and maps, PERFMON attaches perf events, SYS_PTRACE reads /proc/<pid>/, SYS_RESOURCE raises RLIMIT_MEMLOCK, DAC_READ_SEARCH reads ELF binaries past permission bits, SYSLOG reads the ring buffer for verifier diagnostics. The seventh, CHECKPOINT_RESTORE, follows magic-links in /proc/<pid>/map_files/* for symbol reading — kernel 5.9 or newer, falling back below that to SYS_ADMIN, a materially weaker position. Two host requirements follow: /sys/kernel/tracing mounted read-only, and writable storage at /tmp/symb-cache — the omission that bites first on a read-only root filesystem. Grafana documents that set; your node's LSM has the last word. If probes fail under enforcing SELinux, confirm the cause with privileged: true before assuming the seven are universal.

// PATH A ONLY. No OTLP here; nothing from the Path B collector config
// belongs in this file. Save as config.alloy — the ConfigMap key below.

discovery.kubernetes "local_pods" {
  role = "pod"
  selectors {
    role  = "pod"
    field = "spec.nodeName=" + sys.env("HOSTNAME")
  }
}

discovery.relabel "local_pods" {
  targets = discovery.kubernetes.local_pods.targets

  rule {
    action        = "replace"
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
    separator     = "@"
    regex         = "(.*)@(.*)"
    replacement   = "ebpf/${1}/${2}"
    target_label  = "service_name"
  }
}

pyroscope.ebpf "local_pods" {
  targets    = discovery.relabel.local_pods.output
  forward_to = [pyroscope.write.sovereign.receiver]

  // Explicit, never inherited: component default 19, upstream 20.
  // Pin what you profiled at; an upstream change must not re-rate production.
  sample_rate      = 19
  collect_interval = "15s"

  // Interpreter unwinders default on. Disable runtimes you do not run.
  ruby_enabled   = false
  perl_enabled   = false
  dotnet_enabled = false
}

pyroscope.write "sovereign" {
  endpoint {
    url = "http://pyroscope.observability.svc.cluster.local:4040"
  }
}

# PATH A ONLY — the seven capabilities Grafana enumerates, in place of
# privileged: true. Prereq — create the ConfigMap this DaemonSet mounts:
#   kubectl -n observability create configmap alloy-profiler-config \
#     --from-file=config.alloy=./config.alloy
apiVersion: v1
kind: ServiceAccount
metadata: { name: alloy-profiler, namespace: observability }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: alloy-profiler }
rules:
  # discovery.kubernetes returns an empty target set without this
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: alloy-profiler }
roleRef:
  { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: alloy-profiler }
subjects:
  - { kind: ServiceAccount, name: alloy-profiler, namespace: observability }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: alloy-profiler
  namespace: observability
spec:
  selector:
    matchLabels: { app: alloy-profiler }
  template:
    metadata:
      labels: { app: alloy-profiler }
    spec:
      hostPID: true
      serviceAccountName: alloy-profiler
      containers:
        - name: alloy
          # Pin from the Alloy release list; v1.18.1 current at writing.
          image: grafana/alloy:v1.18.1
          args: ["run", "/etc/alloy/config.alloy"]
          env:
            # REQUIRED: the config selects pods by spec.nodeName=$HOSTNAME.
            # Without it, discovery matches nothing.
            - name: HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
            privileged: false
            capabilities:
              drop: ["ALL"]
              add:
                - BPF
                - PERFMON
                - SYS_PTRACE
                - CHECKPOINT_RESTORE # kernel 5.9+; else SYS_ADMIN
                - SYS_RESOURCE
                - DAC_READ_SEARCH
                - SYSLOG
          volumeMounts:
            - { name: config, mountPath: /etc/alloy }
            - { name: tracing, mountPath: /sys/kernel/tracing, readOnly: true }
            - { name: symb-cache, mountPath: /tmp/symb-cache } # writable
      volumes:
        - name: config
          # must carry key config.alloy
          configMap: { name: alloy-profiler-config }
        - name: tracing
          hostPath: { path: /sys/kernel/tracing }
        - name: symb-cache
          emptyDir: {}

Path B: the Collector profiler, and three version numbers that disagree

The OpenTelemetry path has the longer future and the shorter track record. It “runs as an OpenTelemetry Collector distribution with the profiling receiver”, exports OTLP gRPC to Pyroscope on port 4040, and needs a feature gate before the profiles pipeline exists. Grafana's assessment sits on the same page: “This feature is suitable for development and testing. Evaluate carefully before production use.”

Three things deserve stating. Privilege: every published example for this image runs privileged: true with hostPID, and no least-privilege set is documented. Labelling: “By default, the profiler sets process.executable.name on each profile but does not set service_name, which Pyroscope uses as the primary label.” The fix is server config, not collector config: a labelmap rule under limits.ingestion_relabeling_rules on Pyroscope. Paste the DaemonSet alone and profiles arrive unlabelled, effectively unqueryable.

Versions. The alpha announcement names “OTel Collector (v0.148.0 or newer)”; Grafana's example pins 0.147.0; the current Collector release is v0.158.0, published 4 August 2026. Three numbers, none wrong in context. Pin from the release list yourself — Grafana is candid about why: breaking changes “have occurred and may continue”, compatibility “requires careful version management”.

# PATH B ONLY — OpenTelemetry Collector eBPF profiler. Nothing here is
# interchangeable with Path A: protocol, flags, mounts, privileges all differ.
apiVersion: v1
kind: ServiceAccount
metadata: { name: otel-ebpf-profiler, namespace: observability }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: otel-ebpf-profiler }
rules:
  # k8sattributes needs these to enrich profiles; without them, no labels
  - apiGroups: [""]
    resources: ["pods", "namespaces", "nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: otel-ebpf-profiler }
roleRef:
  { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: otel-ebpf-profiler }
subjects:
  - { kind: ServiceAccount, name: otel-ebpf-profiler, namespace: observability }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-ebpf-profiler-config
  namespace: observability
data:
  config.yaml: |
    receivers:
      profiling:
        # Explicit. Upstream default 20; Grafana's example 97. Choose.
        samples_per_second: 20
    processors:
      k8sattributes/profiles:
        auth_type: serviceAccount
        extract:
          metadata: [k8s.pod.name, k8s.namespace.name, k8s.deployment.name]
        pod_association:
          - sources:
              - from: resource_attribute
                name: container.id
    exporters:
      otlp_grpc:
        endpoint: pyroscope.observability.svc.cluster.local:4040
        tls:
          insecure: true
    service:
      pipelines:
        profiles:
          receivers: [profiling]
          processors: [k8sattributes/profiles]
          exporters: [otlp_grpc]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-ebpf-profiler
  namespace: observability
spec:
  selector:
    matchLabels: { app: otel-ebpf-profiler }
  template:
    metadata:
      labels: { app: otel-ebpf-profiler }
    spec:
      hostPID: true
      serviceAccountName: otel-ebpf-profiler
      containers:
        - name: profiler
          # Current release, above the alpha's floor. Pin yourself.
          image: otel/opentelemetry-collector-ebpf-profiler:0.158.0
          args:
            - "--config=/etc/otel/config.yaml"
            # Without this gate there is no profiles pipeline.
            - "--feature-gates=+service.profilesSupport"
          securityContext:
            # As documented; no least-privilege set is published.
            privileged: true
          volumeMounts:
            - { name: config, mountPath: /etc/otel }
            - { name: sys-kernel, mountPath: /sys/kernel, readOnly: true }
            - { name: tracefs, mountPath: /sys/kernel/tracing, readOnly: true }
            - { name: lib-modules, mountPath: /lib/modules, readOnly: true }
            - { name: proc, mountPath: /proc, readOnly: true }
      volumes:
        - name: config
          configMap: { name: otel-ebpf-profiler-config }
        - name: sys-kernel
          hostPath: { path: /sys/kernel }
        - name: tracefs
          hostPath: { path: /sys/kernel/tracing }
        - name: lib-modules
          hostPath: { path: /lib/modules }
        - name: proc
          hostPath: { path: /proc }

One asymmetry: Grafana's docs-page DaemonSet mounts only /proc and /sys/kernel; the complete kustomize example it links to also mounts /lib/modules and /sys/kernel/tracing. The manifest above follows the complete example, with the RBAC k8sattributes needs.

Overhead you measured versus overhead you were promised

The upstream project states its ceiling: “1% CPU and 250MB memory are our upper limits in testing and the agent typically manages to stay way below that”. Read it as what it is: the project's own testing, not an independent benchmark, not annotated with the sample rate.

The other number you will be quoted is Google's, almost always wrongly. The Google-Wide Profiling paper reports that “the aggregated profiling overhead is negligible—less than 0.01 percent” — fleet-aggregated, achieved by profiling a small subset of machines at a time, at low sample rates, with symbolization moved off-host. The same paper's per-machine figure is two orders of magnitude larger: rates set conservatively “to ensure the overhead is always less than a few percent” — the figure a DaemonSet on every node answers to. Sample rate is the knob, priced from outside the container: the Path B image is built FROM scratch, with no shell to kubectl exec into. The kubelet's Summary API reports a cumulative usageCoreNanoSeconds per container, whatever the image contains.

#!/usr/bin/env bash
# Profiler CPU cost as a percentage of one node's CPU capacity.
# Needs get on nodes/proxy; whole-core node capacity assumed.
set -euo pipefail

POD=${1:?usage: measure-profiler-cost.sh <pod-name> [window-seconds]}
NS=${NAMESPACE:-observability}
CTR=${CONTAINER:-profiler}
WINDOW=${2:-300}

NODE=$(kubectl get pod -n "$NS" "$POD" -o jsonpath='{.spec.nodeName}')
cores=$(kubectl get node "$NODE" -o jsonpath='{.status.capacity.cpu}')

read_ns() {
  kubectl get --raw "/api/v1/nodes/$NODE/proxy/stats/summary" \
    | jq -r --arg p "$POD" --arg n "$NS" --arg c "$CTR" '
        .pods[] | select(.podRef.name == $p and .podRef.namespace == $n)
        | .containers[] | select(.name == $c) | .cpu.usageCoreNanoSeconds'
}

start=$(read_ns)
sleep "$WINDOW"
end=$(read_ns)

awk -v s="$start" -v e="$end" -v w="$WINDOW" -v c="$cores" 'BEGIN {
  cpu = (e - s) / 1000000000
  printf "%.2f CPU-seconds = %.3f%% of one core, %.3f%% of the node\n",
    cpu, 100 * cpu / w, 100 * cpu / (w * c)
}'

What eBPF still cannot do for you

Budget the remaining instrumentation work now, not after someone asks why there are no traces. Distributed tracing is the clearest boundary: zero-code eBPF tracing exists — OpenTelemetry eBPF Instrumentation, the donated Grafana Beyla — but its first-release announcement is explicit that context propagation works well “for Go (HTTP and gRPC), Node.js (HTTP), Python (HTTP), NGINX (HTTP), PHP (HTTP/FPM), while for other programming languages support will vary a lot” with how the application manages threads and connections. Java or .NET-heavy estates should plan for SDKs.

Business metrics are the second gap: no kernel probe knows what a failed settlement is. Symbolisation is the third, an acknowledged rough edge on the Collector path: Grafana warns that “function names may not resolve in flamegraphs for some programs”. Profiles slot in beside the metrics, logs and traces stack you already run; they replace none of it.

Profiles turn “this namespace costs too much” into “this function costs too much” — the missing half of every showback conversation, and what makes scale-to-zero and rightsizing evidence-based.

The exit ramp

Ask the exit question before you deploy; the answer here is unusually good. The collection agent is already shared: Parca's maintainers “voted that Parca-Agent, our beloved profiling tool, is merging its development with the OpenTelemetry-eBPF-Profiler” in August 2024, and Alloy's component embeds a fork of that same profiler. Switching backends changes the exporter, not the instrumentation strategy — what makes a dependency cheap to leave.

The backend is AGPL-3.0 — copyleft, not source-available, not BSL. And the storage layer changed in the current major version: profiles are “written directly to object storage, removing the need for in-memory ingesters and local disks”. Your history sits in a bucket you own — though much surviving blog content still describes the v1 ingester.

The long game

The two paths are not equally ready, and the honest recommendation follows the labels, not the roadmap. Alloy's component is GA today; the Collector path rides a signal its own SIG says should not carry critical production workloads. Run Path A in production and Path B in staging, so when the signal stabilises you migrate with experience.

Neither path is agent-free — both put a highly privileged DaemonSet on every node. What changes is custody: the privilege stays inside a boundary you drew, the data lands in storage you own, retention is a decision, not a plan you are on. Pin the server major version, pin the sample rate, record why. That makes the fourth signal a system you operate for a decade rather than a subscription you renew.