Every observability rollout eventually hits the same wall: the services you most need to see are the ones nobody can instrument. The vendor binary, the legacy monolith whose build nobody trusts, the twelve teams who each have a different quarter in mind for adding an SDK. eBPF observability goes around that wall. Instead of asking every application to report on itself, it watches the work from the Linux kernel, where every request has to pass through a socket regardless of the language it was written in.

This guide explains how that actually works: what eBPF is, which hooks (kprobes, uprobes, tracepoints) produce which signals, how a request and its latency are reconstructed from raw socket reads and writes, and where kernel-level telemetry stops being enough. The short version: eBPF is excellent at the questions every service shares and blind to the ones only your code can answer. Knowing which is which is most of the skill.

What eBPF is, in kernel terms

eBPF lets you load small programs into the running Linux kernel and attach them to events: a function being entered, a system call returning, a packet arriving. Before a program is allowed to run, the kernel's verifier walks every possible path through it and rejects anything that could crash or hang the kernel, such as unbounded loops, out-of-bounds memory access, or calls to functions outside an approved set of helpers. Programs that pass are JIT-compiled to native instructions. They share data with user space through maps (hash tables, arrays, per-CPU counters) and through ring or perf buffers that stream events out to an agent process.

That combination is what makes eBPF usable in production. A kernel module can do anything, including take the machine down. An eBPF program is constrained to be safe before it ever runs, and it can be loaded and unloaded without a reboot and without restarting the processes it observes.

The hooks that matter for eBPF monitoring

An eBPF program is only as useful as the event it is attached to. Four kinds of attach point carry most observability work:

kprobes and kretprobes fire on entry to and return from almost any kernel function. Attaching to tcp_sendmsg and tcp_recvmsg sees every TCP payload on the host. The catch: kernel functions are internal, so their names and signatures can change between kernel versions.
Tracepoints are stable instrumentation points that kernel maintainers commit to keeping, such as sched:sched_switch or syscalls:sys_enter_connect. Prefer them whenever one exists for the event you care about.
uprobes and uretprobes do the same for functions in user-space binaries and shared libraries. This is how a tool reads TLS traffic before it is encrypted: by attaching to SSL_write and SSL_read in OpenSSL.
Network hooks (socket filters, tc and XDP programs) see packets themselves, which suits connection-level and network-path signals.

You can try the first three styles with bpftrace on a recent Linux host. These one-liners need root:

# Tracepoint (stable): count outbound connect() calls per process
bpftrace -e 'tracepoint:syscalls:sys_enter_connect { @connects[comm] = count(); }'

# kprobe (kernel internals): bytes handed to TCP per process.
# arg2 is the size argument of tcp_sendmsg on current kernels.
bpftrace -e 'kprobe:tcp_sendmsg { @bytes[comm] = sum(arg2); }'

# uprobe (user space): TLS writes through OpenSSL. The library path varies by distro.
bpftrace -e 'uprobe:/usr/lib/x86_64-linux-gnu/libssl.so.3:SSL_write { @writes[comm] = count(); }'

Stop each with Ctrl-C and it prints a table per process. Nothing was installed in those processes, nothing restarted, and the language they were written in did not matter. That is the core promise of zero-code instrumentation, in three lines.

From socket bytes to golden signals

Counting connections is not observability. The interesting part is turning raw reads and writes into requests with latency and status codes. Most eBPF monitoring agents follow the same pipeline:

Capture. Probes on the socket send and receive paths record the process ID, the connection (source and destination address and port), a timestamp and the first few hundred bytes of each buffer. Copying is bounded on purpose: the verifier limits how much work a program does per event, and copying whole payloads would cost far more than the signal is worth.
Classify. The agent inspects those first bytes to recognize the protocol. A request line such as GET /orders HTTP/1.1 is HTTP/1.1, a connection preface marks HTTP/2, and PostgreSQL, MySQL and Redis each have recognizable wire formats.
Pair. A request written on a connection is matched with the response read back on the same connection. The time between the two is the latency as the caller experienced it, including time in the kernel network stack that an in-process timer never sees.
Enrich. The process ID is resolved to a container, and the container to its pod, namespace, deployment and node, so the signal arrives labeled like the rest of your telemetry.
Export. Aggregated metrics (rate, errors and latency percentiles per endpoint) and per-request spans leave the node, typically over OTLP.

Because the connection is known on both ends, the same data yields a service map: which workload talks to which, at what rate and latency, built from connections that actually happened rather than from a diagram somebody drew last year. And because the probes sit in the TCP stack, you also get signals that no application SDK can see at all: retransmissions, connection resets, and time spent queued between the socket and the process.

Where kernel telemetry runs out

eBPF is not magic, and its gaps are specific. These are the ones that matter in production.

Encrypted traffic

If a service terminates TLS itself, the bytes crossing the socket are ciphertext. The fix is the uprobe approach above: hook the TLS library's read and write functions to see plaintext. That works well for dynamically linked OpenSSL, but each TLS implementation needs its own probes, and statically linked binaries (most Go services, which use Go's own crypto/tls) need symbol offsets resolved per binary. Go adds a further wrinkle: uretprobes are unsafe with Go's movable goroutine stacks, so tools attach probes at a function's return instructions instead. Before assuming encrypted traffic is covered, ask exactly which TLS libraries an agent handles.

Stateful protocols

HTTP/2 and gRPC compress headers with HPACK, which builds a header table over the life of a connection. An agent that starts watching a connection midway has missed the entries that later headers refer to, so it can see requests without their paths until the connection is replaced. Long-lived gRPC channels make this a real concern in the hours after an agent is first deployed.

Distributed context

An eBPF agent sees each hop, but joining hops into one end-to-end trace needs a trace ID carried in the request. If your services already propagate W3C traceparent headers, an agent that parses headers can attach to that context. If they do not, cross-service joins rely on timing and connection heuristics, which are approximate under concurrency. A trace that crosses a message queue is harder still, because the producer and the consumer never share a connection.

Business meaning

The kernel knows that a POST to /api/checkout took 840 ms and returned a 500. It does not know the customer's plan, the size of the cart, the feature flag that was on, or which of four internal steps was slow. That context lives in your code, and only your code can attach it to a span.

Privileges and platforms

Loading eBPF programs requires kernel privileges: CAP_BPF and CAP_PERFMON on newer kernels, CAP_SYS_ADMIN on older ones, which in Kubernetes usually means a privileged DaemonSet. Some environments will not allow that at all, including serverless container platforms that do not run DaemonSets. Portable agents also rely on BTF (BPF Type Format), the kernel type information that lets one compiled program adapt to different kernel struct layouts. Check that /sys/kernel/btf/vmlinux exists on your nodes before you plan a rollout.

eBPF observability vs SDK instrumentation: use both

Framing this as a choice is the most common mistake. The two approaches answer different questions, and they overlap less than vendor pages suggest.

eBPF gives you breadth on day one. Every workload on the node reports rate, errors and latency for HTTP, gRPC and database calls, including third-party images and services nobody owns. Coverage does not depend on twelve teams finding time.
SDKs give you depth where it pays. Custom spans around business logic, attributes like tenant and plan, timings for code that never touches a socket, and reliable context propagation across queues.
The network view belongs to eBPF. Retransmits, resets and kernel-side queueing are invisible to an SDK by definition.
The business view belongs to the SDK. No probe can tell you which customer had the slow request unless the code says so.

A pattern that works: deploy eBPF cluster-wide first to get a baseline and an accurate service map, then use that map to decide where SDK instrumentation earns its maintenance cost. Services on the revenue path get OpenTelemetry SDKs and custom spans (our OpenTelemetry for Node.js guide shows a production setup); the long tail stays on kernel telemetry. When both exist for a service, each checks the other: a large gap between the latency the socket sees and the latency the handler span reports usually means requests are queueing before your code even runs.

Overhead: what it costs and how to measure it

The usual objection is overhead, and it is the right question, asked with a little more precision. eBPF cost scales with event rate, not with the number of services. Each probe hit costs a small, fixed amount of kernel work plus whatever data the program copies out. A node serving a few thousand requests per second is a very different load from a proxy serving a few hundred thousand, and a probe on a hot kernel function such as a scheduler event fires far more often than one on connect().

So measure it on your own workload rather than trusting anyone's headline number, including ours:

Compare node CPU and the p99 latency of a busy service with the agent on and off, under the same load.
Watch the agent's own CPU and memory, and any event-drop counters it exposes. A full ring buffer drops events rather than slowing the application, which is the right failure mode, but it means your data has gaps.
Use namespace exclusions and drop rules for workloads you do not need, such as batch jobs or system namespaces, so you are not paying to capture data nobody reads.

A rollout checklist for eBPF monitoring

Confirm kernel version and BTF support on every node pool (ls /sys/kernel/btf/vmlinux).
Confirm your platform permits privileged DaemonSets, and get security sign-off on the capabilities the agent requests.
List the TLS libraries your services use and check which ones the agent supports.
Deploy to one node pool first and compare agent-on and agent-off CPU and p99 latency.
Exclude namespaces you do not need before rolling out cluster-wide.
Compare the generated service map with what teams believe the architecture is. The differences are often the most useful finding of the week.
Pick the services where SDK spans would add business context, and plan that work separately.

How axonIQzerocode fits

aiAxonIQ's Zero-Code eBPF instrumentation module, axonIQzerocode, is built on the approach described here. It runs as a DaemonSet installed with Helm, attaches eBPF kprobes and uprobes to socket paths, reconstructs HTTP/1.1, HTTP/2, gRPC, PostgreSQL, MySQL and Redis calls, and adds pod, namespace, container and node labels before exporting OTLP. It needs a Linux kernel 5.4 or later with BTF, and it supports namespace exclusion filters and data drop rules for the overhead reasons above.

Because its output is OTLP, kernel telemetry lands next to the spans from services you instrument with OpenTelemetry SDKs, and the two correlate where SDK spans exist. If you run Kubernetes, the Kubernetes monitoring page shows how the DaemonSet sits alongside an existing Collector and Prometheus scrapes, and the Go monitoring guide covers the choice between compiling in the SDK and observing a binary from the kernel.

eBPF Observability Explained: Zero-Code Telemetry from the Kernel