Linux user_events, explained: from application events to OpenTelemetry

A checkout application has just finished a request. The response status was 200 and the request took 12.5 milliseconds. Most of the time, collecting that detail for every request is unnecessary. But when checkouts start slowing down, an operator would like to see exactly those events for a few minutes, without changing the service's configuration or restarting it.

Linux user_events supports this once the application is instrumented and has registered its events. It is a kernel interface for publishing structured application events into Linux tracing. A local collector is a separate program on the same machine that reads and processes these events. It can start and stop collection without changing the running application.

The whole path in four steps

1. Register: the application tells Linux about an event it can produce.
2. Enable: a collector asks Linux to collect that event.
3. Write: the application checks whether collection is enabled, then writes the event.
4. Read: the collector reads and processes the captured records.

We'll first use a simple event with status and duration to explain the Linux mechanism, then follow the same checkout scenario through OpenTelemetry: a Rust application creates an OpenTelemetry log record, sends it through user_events, and a collector writes it into a JSON file. The small C example in section 2 is optional; it shows the Linux write interface.

Reading guide: Basic programming knowledge and familiarity with processes and memory are assumed, but not prior knowledge of the named Linux interfaces. Sections 1-3 explain the Linux mechanism. Sections 4-6 follow the two programs and their data. Section 7 covers history and compatibility. Section 8 connects the ideas with short code snippets. Sections 9-10 cover operational limits and design choices.

1. Start with one application event

Near the end of request handling, the checkout code reports that the request completed. That place in the code is an instrumentation point. Each time it runs, it can produce an event record describing one occurrence: here, request_completed with status=200 and duration_us=12500. The point stays in the program; the values can differ from request to request.

A logging call does not automatically create a Linux tracing event. To publish through user_events, the application first registers the event with the kernel. This creates a tracepoint: a named hook that tracing tools can attach to. A program that attaches and reads records is a consumer.

Figure 1 shows the split. The reporting point lives in the application; the collector is a separate process. Follow the arrows for a record written through the kernel when collection is enabled.

Enlarge diagram +

Linux does not know what status 200 means. The application, the producer, supplies the values, and the collector decides what to collect.

2. Register, enable, then write

These are steps 1-3 from the opening: the application registers its event, the collector enables collection, and a guarded application writes records while collection is enabled. The kernel checks and dispatches each write; the application and collector give it meaning.

The collection path described here uses perf, Linux's performance-monitoring subsystem. It stores event samples in kernel buffers that a local program can read. Nothing on this path sends the record to a remote backend by itself.

Register the event

First, the application tells the kernel the event's name and field layout through tracefs, a special filesystem usually mounted at /sys/kernel/tracing. It opens user_events_data and sends a registration request with ioctl, the system call for control operations on an open file.

A plain field description could be:

request_completed u32 status; u32 duration_us

u32 means an unsigned 32-bit integer. Write the values in field order using native byte order, without structure padding; 12.5 ms becomes 12,500 microseconds. The OpenTelemetry integration later needs a richer encoding.

uint32_t write_index = registration.write_index;
uint32_t payload[2] = {200, 12500};
struct iovec parts[2] = {
    {.iov_base = &write_index, .iov_len = sizeof(write_index)},
    {.iov_base = payload, .iov_len = sizeof(payload)},
};
ssize_t written = writev(fd, parts, 2);

Registration returns a write index for later writes through that open file. It also tells the kernel where to find an enable bit in the application's memory, which the kernel will update.

Let the collector enable the event

When the collector attaches and enables the event, the kernel sets the enable bit. The application checks it with an atomic load, reading its own memory while the kernel can update it, instead of asking the kernel on every request. Registration can happen before the collector starts. With retry configured, the collector can start first, but attachment waits for registration.

In Figure 2, follow the dashed setup arrows first: register, attach, update the bit. Only then does the solid data path carry an event.

Enlarge diagram +

Enablement reports attachment, not delivery. The checkout service can use the bit to skip work when nobody is listening, but not as an acknowledgment.

Write the event

When the bit is set, the application encodes the checkout values and hands them to the kernel with writev, a system call that sends the write index and payload together. The native interface treats each call as one registered Linux event; the index identifies which event.

The kernel checks the record, and perf records samples in its ring buffer, a bounded circular buffer that the collector reads. There is one ring for each CPU the collector covers, which matters in section 5.

If nobody is listening, a guarded application skips the write. Nothing is saved for a collector that attaches later.

3. Check first, then prepare

When no consumer has enabled the registered event, the guarded path below checks the enable flag and skips constructing, encoding, and writing the record. The checkout request still runs; work done before the check still happens.

A common mistake is preparing diagnostic details before checking whether anyone is collecting them. Compare these pseudocode examples:

Before: preparation always runs.

details = build_details(request)
if collection_enabled():
    emit(details)

After: preparation runs only when collection is enabled.

if collection_enabled():
    details = build_details(request)
    emit(details)

Figure 3 shows the same pattern. Follow the "no" branch: it skips preparing, encoding, and writing the event. Both branches rejoin at "Request continues"; the enabled path gets there after the write returns.

Enlarge diagram +

When collection is enabled, the application still prepares and encodes the event and makes one system call per write. The collector copies and decodes it. The guard avoids those event-only costs when disabled, but the enablement check and normal request work remain.

The OpenTelemetry Rust logger exposes event_enabled. A producer can call logger.event_enabled(...) before logger.create_log_record(). Section 8 follows the resulting record through selected excerpts.

flowchart TB
    subgraph U["USERSPACE: application"]
        A["Handle request (business logic)"]
        M["Enable bit (set by kernel)"]
        B{"Collection enabled?"}
        D["Prepare details, encode payload"]
        R["Request continues"]
    end
    subgraph K["KERNEL SPACE"]
        W["user_events tracepoint dispatch"]
    end
    A --> B
    M -.->|"read enable bit"| B
    B -->|"no"| R
    B -->|"yes"| D
    D -->|"writev"| W
    W -->|"return from write"| R
    style U fill:#f3f7fc,stroke:#c5d2e1
    style K fill:#fff7e8,stroke:#b29a6a
    linkStyle 0,2,3,5 stroke:#536579,stroke-width:2px
    linkStyle 1 stroke:#7559a6,stroke-width:2px
    linkStyle 4 stroke:#2764b4,stroke-width:3px

4. Connecting user_events to OpenTelemetry

The Linux path gets checkout data into a collector. OpenTelemetry provides standard APIs and data models for telemetry; using those models involves four separate jobs.

EventHeader is an event encoding format. In the self-describing form used here, field names and types travel in the user_events payload alongside the values.

Linux defines how to register and write an event, not which telemetry layout you must use. EventHeader describes fields; the schema chooses their names and meanings; the mapper translates them into OpenTelemetry. Producer and collector must agree on all three.

In section 2's plain-field path, the receiver gets field names, types, and offsets from tracefs format metadata and applies that layout to the payload's values. With EventHeader, it decodes application fields such as message and duration_us using metadata inside the payload; they are not individually listed in this producer's tracefs format. Tracefs still describes the registered Linux event. Neither source of field metadata decides what belongs in an OpenTelemetry log's body, severity, or resource.

A Rust application can report the checkout result as an OpenTelemetry log. The Rust SDK (software development kit) creates that record. An application-side exporter encodes it and writes it through user_events. A collector-side exporter instead writes processed telemetry to a file or sends it to a backend. These are different components in different programs.

Figure 4 shows the two programs. Follow the arrows from the application at the top, through the kernel, to the collector at the bottom.

Enlarge diagram +

The application supplies the log values and configures the SDK provider's resource with service.name. For a successful checkout, the message is "checkout completed", status is 200, and duration is 12500 microseconds. Its application-side exporter checks enablement, encodes the fields using EventHeader, then writes through user_events. In this synchronous integration, it implements the SDK's LogProcessor interface and writes during emit, directly on the application thread. This processor runs inside the application, not inside the collector.

The collector can use OpenTelemetry Arrow's Rust engine, which runs telemetry pipelines by connecting receivers, processors, and exporters. Its existing user_events receiver reads and decodes the event. A collector-side mapping processor, or mapper, places decoded values into standard log fields. The existing collector-side file exporter can write OTLP JSON, a JSON form of the OpenTelemetry Protocol.

Decoding is not interpreting

The receiver's event_header decoder recovers these fields as log attributes. It knows message is a string, but not that it belongs in the log's body. That's the mapper's job; section 6 shows the before-and-after.

Keeping mapping in a collector-side processor makes it reusable with different collector-side exporters. A purpose-built collector-side exporter could combine interpretation and forwarding instead, but an existing exporter will not infer a custom field's meaning.

For a versioned layout, the mapper must recognize the schema version.

Choose the format your pipeline needs

You can adopt an existing layout or implement one for your requirements. Rust contrib's opentelemetry-user-events-logs is an example of the first choice: it registers its events, uses EventHeader, checks enablement before encoding, and writes synchronously with its own predefined field layout. Use it when that contract suits your producer and collector.

For the second choice, write or adapt an application-side exporter to carry the fields your application needs. The small, flat layout in this article illustrates that approach. Reusing EventHeader can reuse the receiver's supported-field decoder, but the collector still needs the matching interpretation of those fields. Changing the contrib logs exporter's layout requires changing or replacing its implementation, not selecting a different format in its configuration.

5. How the collector receives events

The collector's experimental, Linux-only user_events receiver listens for specific named events. Its configuration lists those names; it does not discover every application's events or read records from the application's user_events_data file.

The plain-field event is request_completed. An EventHeader producer groups events under a provider name. With a provider named checkout, Info and Error use the following names. Use the same provider and level names in the receiver configuration.

checkout is the provider name. L4 and L2 select EventHeader's Informational and Error levels, not the SDK's severity numbers 9 and 17. K1 means keyword 1: a keyword is a numeric grouping mask within a provider, set to 1 here. The payload's event_name is request_completed in both cases. It describes the occurrence, while the subscription name tells the receiver which group to collect.

Several payload event names can share the same provider/level/keyword group, so its enable flag is not specific to request_completed.

The receiver uses one_collect, a Rust event-collection library, to read tracefs format metadata and attach through the perf_event_open system call. It then drains memory-mapped perf buffers. This does not require running the perf command or loading an extended Berkeley Packet Filter (eBPF) program. Reading tracefs metadata is not reading an event stream from trace_pipe.

Each pipeline instance attaches to its assigned logical CPU only, following the engine's thread-per-core design. Figure 5 shows an application writing on CPU 0: follow its bytes into the CPU 0 ring, then up to the receiver.

Enlarge diagram +

The CPU 1 ring does not receive a duplicate of CPU 0's record. It is another possible collection point, for writes performed on CPU 1. The important question is whether a session covers the CPU where the application emits.

The same request under three collection conditions

Assume successful registration, no other listeners, and the same guarded producer in each case.

The enable bit is event-wide, not per-CPU. The last row is therefore not a contradiction: somebody attached to the event, but not on the emitting CPU. It also need not produce a perf-ring loss notification; this was missing coverage, not a full ring.

6. Mapping event fields to OpenTelemetry logs

The receiver copies samples from the perf ring into a bounded pending queue and decodes their fields. It builds Apache Arrow columnar log batches, arranging values in typed columns. This OTAP representation, short for OpenTelemetry Protocol with Apache Arrow, begins in the collector, not in the kernel write.

In the EventHeader path, the collector receives the checkout values as attributes. The upstream decoder does not automatically populate the SDK log's typed fields from them.

What the mapper receives (schematic view):

attributes: message = "checkout completed", severity_number = 9,
            severity_text = "INFO", service_name = "checkout-service",
            status = 200, duration_us = 12500
body:       (empty)
severity:   (unset)
resource:   (no attributes)

The message is available, but the log's body is still empty. The mapper places it in the standard body field so downstream tools can treat it as the log message. The same applies to severity and service identity.

Mapping the checkout fields:

After mapping (schematic view):

attributes: (unchanged)
body:       "checkout completed"
severity:   number = 9, text = "INFO"
resource:   service.name = "checkout-service"

This example copies values into the standard fields while keeping the original attributes. Another mapper could remove redundant attributes, but that is optional.

The service name identifies the source service, while status and duration describe one request. Duration remains in microseconds; mapping does not silently change its unit. Severity comes from the explicit field, not from guessing what status 200 should mean.

The scope name identifies the instrumentation. If the producer includes scope or trace-context fields, the collector also needs rules to restore them and validate their representation. Carrying trace/span IDs allows correlation; it does not create a span.

Concept-only stopping point

If you only need the mental model, sections 1-6 are sufficient. The remaining sections cover compatibility, selected implementation details, and operational choices.

7. Kernel support and compatibility

An earlier user_events interface first appeared in Linux 5.18. The original implementation and versioned documentation describe a shared status mapping that producers read to check enablement. The interface used here arrived in Linux 6.4, with the application-owned enable_addr/enable_bit interface documented in the 6.4 guide. This is interface evolution: examples using the earlier producer interface are not drop-in instructions for the current one.

A kernel version or distribution name is not enough. The code must be present, CONFIG_USER_EVENTS=y and CONFIG_PERF_EVENTS=y must be built into the booted kernel, tracefs must be mounted and visible, and the producer and collector need the relevant tracefs and perf permissions.

8. How the record changes across the pipeline

These fragments connect an EventHeader layout to a receiver and mapping processor. Logger setup, event registration, and shutdown are omitted.

Create and emit the log record

Assume logger is an OpenTelemetry Rust SDK logger configured with a user_events-writing SDK processor, and its provider's resource has service.name = "checkout-service". The relevant Logger and LogRecord traits are in scope. Check enablement before constructing the checkout record:

if logger.event_enabled(Severity::Info, "checkout", Some("request_completed")) {
    let mut record = logger.create_log_record();
    record.set_event_name("request_completed");
    record.set_body("checkout completed".into());
    record.set_severity_number(Severity::Info);
    record.set_severity_text("INFO");
    record.add_attribute("status", 200_i64);
    record.add_attribute("duration_us", 12500_i64);
    logger.emit(record);
}

The application-side exporter still checks its own event set, because collection may have changed or another SDK processor may have wanted the record. Below, event_set is an already registered EventHeader set. body, severity_number, severity_text, status, and duration_us come from the SDK record; service_name comes from the provider's resource, supplied to the processor through set_resource. The fragment belongs in a function returning std::io::Result<()>; registration and cleanup are omitted.

if event_set.enabled() {
    let mut event = EventBuilder::new();
    event.reset("request_completed", 0);
    event.add_str("message", body, FieldFormat::Default, 0);
    event.add_value("severity_number", severity_number, FieldFormat::SignedInt, 0);
    event.add_str("severity_text", severity_text, FieldFormat::Default, 0);
    event.add_str("service_name", service_name, FieldFormat::Default, 0);
    event.add_value("status", status, FieldFormat::SignedInt, 0);
    event.add_value("duration_us", duration_us, FieldFormat::SignedInt, 0);

    let errno = event.write(event_set, None, None);
    if errno != 0 {
        return Err(std::io::Error::from_raw_os_error(errno));
    }
}

These are the six fields from section 6's checkout record. EventHeader describes them; it does not decide which OpenTelemetry fields the collector should populate.

Map the decoded fields

The collector-side mapper copies values from that same payload into the standard fields, leaving the original attributes unchanged in this sketch. This is mapping pseudocode, not a built-in processor API:

log.body = attributes["message"]
log.severity_number = attributes["severity_number"]
log.severity_text = attributes["severity_text"]
resource.attributes["service.name"] = attributes["service_name"]

The implementation must validate field types and define what to do when a required field is missing. Status and duration can remain log attributes. Resource information belongs to the emitting service, not to a single request.

This configuration excerpt shows the three collector stages in sequence:

{
  "connections": [
    { "from": "receiver", "to": "mapper" },
    { "from": "mapper", "to": "output" }
  ]
}

Here receiver, mapper, and output are configuration names for the chosen nodes. A layout-specific mapper must be implemented or supplied by a component that supports that layout; these connections alone do not create it.

How to tell it worked

Inspect the collector's output, not just the producer's return value. For the checkout record, look for checkout completed in the log body, the intended severity and service resource, and unchanged status/duration attributes. Check supplied scope and trace context too.

A missing record can mean disabled collection, missing CPU coverage, a decoding failure, or a later loss. A successful writev alone cannot distinguish those outcomes. The buffering and delivery limits in section 9 still apply.

9. Buffering, limits, and data loss

Where buffering happens

A batch groups records for processing or export together. A typical batched OTLP setup uses the SDK's BatchLogProcessor: it keeps records in an application-side memory queue and exports batches on a background worker. A checkout log can therefore wait inside the application before the application-side exporter sends it.

The synchronous user_events design writes enabled records directly instead. There is no application-side SDK batch queue on this path, but event values and temporary encoding buffers still use application memory.

In Figure 6, compare where the checkout log can wait. The upper part of each path is application memory; the user_events path then crosses into the kernel's perf ring and a separate collector process. Follow the boundaries, not just the arrows.

Enlarge diagram +

Buffering and batching are different jobs. The perf ring holds individual samples; it is not an OpenTelemetry batch processor. The Rust receiver drains samples into a bounded pending queue, decodes them, and assembles log batches for its downstream pipeline. In this version, its batching defaults are 512 records or a 50 ms timer. These are configurable flush triggers, not end-to-end latency guarantees. Receiver batching.

This batching happens inside the receiver; it does not require a separate collector batch-processor node.

What survives an application exit?

A sample already committed to an active collector's perf ring can remain available after the producing application exits. The sample is no longer in the application's heap, and the independent collector can continue reading it while its perf session and the host remain alive.

Suppose one checkout record has reached the perf ring; a second is still being encoded inside the application. If the application crashes now, the first can still be collected. The second may never be emitted. With an SDK batch queue, records still waiting in that application's queue are also lost when the process dies.

The benefit is removing that application-side queue-loss window, not making all events crash-proof. A successful writev alone does not prove a sample entered the ring. Missing CPU coverage and buffer overflow can prevent capture; collector failure, perf-session teardown, or host failure can lose records that have not reached durable storage. Receiver queues and partially built batches are volatile too.

What happens under sustained load?

Suppose the collector receives checkout records faster than its downstream pipeline can process them. Buffers absorb a burst, but cannot make sustained overload disappear.

When downstream channels fill, the receiver waits and stops draining perf. That pressure does not reliably slow application writers: the perf ring becomes the overflow point. Pending admission can also drop records, and process memory pressure can discard decoded batches. Counters distinguish reported perf loss, queue overflow, memory shedding, and send failures; they cannot account for events never observed on uncovered CPUs.

Do not assume global ordering across CPUs. Batching and optional round-robin subscription draining also affect downstream order. There is no durable spool, replay protocol, or exactly-once guarantee in this path.

Four different sizes matter here. The single-event limit bounds one payload. The perf ring capacity bounds captured samples waiting in the kernel. The pending queue bounds parsed samples waiting inside this receiver. The batch size and timer decide when decoded logs are sent downstream. They are not one shared user_events buffer.

Different receiver policies can change queue capacities and flush thresholds, but not the recording path's per-event size limit.

Kernel constraint

A single event must fit the recording path's per-event size limit. Encoding headers, field metadata, and values all consume that budget. In the reviewed perf path, an oversized sample can be dropped before entering the ring even when the producer's write reports success.

If you pack multiple telemetry records into one Linux event, its wrapper and all records share that event's byte budget. This is not a limit on the collector's batches assembled after capture. A larger perf ring cannot raise the per-event limit.

[write index for a batch event]
[batch metadata][record 1][record 2]...[record N]

batch metadata + sum(encoded record sizes) <= 8,184 bytes

10. When to use user_events

Return to the checkout that started slowing down. Once the application is instrumented and has registered its events, a collector can enable a short diagnostic window without restarting the application. When collection stops, the guarded path can skip the event-only work again.

Choose the producer layout and the collector's handling together: reuse an existing contract or define fields for your needs. Enablement, CPU coverage, and semantic mapping are separate responsibilities; none promises delivery.

These paths can coexist: routine telemetry could continue over OTLP while selected diagnostics use user_events. A local collector can forward mapped logs through its OTLP exporter, keeping network-endpoint and authentication handling there for that route. Mixed SDK outputs have the enablement interaction described in section 3.

Before switching, compare the same checkout workload with your existing path: the application's work, the collector's work, and the records actually received.

Keep these four ideas:

• user_events is a local Linux tracing interface, not a remote telemetry protocol.
• An operator can enable collection from outside the application, without restarting it. The application must already be instrumented and its events registered. An enablement check can avoid event-only work, but it does not acknowledge delivery.
• Perf collection is per CPU and bounded. The collector must cover the emitting CPU and keep up with its records.
• Use it for selective Linux-local diagnostics. Prefer ordinary logging or direct OTLP when portability and standard semantics matter more.

Version scope: Linux development snapshot (Makefile version 7.3.0-rc4) and OpenTelemetry Arrow snapshot, reviewed September 22, 2026; Rust contrib snapshot, rechecked September 25, 2026. Code snippets show integration patterns; they are not a complete program. To validate this explanation, one EventHeader-based logs path (register, enable, write, decode, map) was verified end to end on Ubuntu 24.04.3 LTS, kernel 7.0.0-1014-azure, using sudo.