NUMA and eBPF: getting incoming traffic onto the right worker

A telemetry receiver can have spare CPU capacity while a few worker threads struggle to keep up. Pinning the workers does not fix this if most of the incoming work reaches the same two threads.

In the first NUMA article, we separated where work runs from where its memory lives. This follow-up adds a third decision: which worker gets the work in the first place. For a network receiver, that decision can begin with the listener chosen for an incoming TCP connection.

Linux already distributes connections across sockets. An eBPF program can customize that choice, using the application's knowledge of worker placement. The interesting part is not just writing the selector. It is making socket ownership, the receive path, memory placement, and connection fan-out agree.

Reading guide: Sections 1-3 build the networking and eBPF model. Sections 4-6 walk through a two-node policy and its limits. Section 7 connects it to otel-arrow; sections 8-9 explain what to inspect and measure. The receive-path mechanics in section 5 are optional on a first pass.

Implementation scope: This article uses otel-arrow's planned NUMA-aware load balancer as a design case study, not as an already available feature. The inspected engine implements topology-aware worker placement and listener metadata, but not the coordinated eBPF socket-selection path described here. The examples and diagrams are illustrative, not benchmark results.

1. A worker can be local and still receive the wrong share of work

Consider the checkout service from the first article. It sends logs to a telemetry engine, which receives them, decodes batches, applies processing, and exports the result.

In a thread-per-core design, each worker runs a complete pipeline replica on one execution thread. A replica is another instance of the configured pipeline, not another copy of every incoming record. The receiver's share of the traffic becomes that worker's work.

Suppose a machine exposes two NUMA nodes, with four receiver workers on each. NUMA means Non-Uniform Memory Access: a CPU can access memory attached to another node, but that access can have a different latency and bandwidth cost from nearby memory. Nodes are locality domains, not necessarily a one-to-one count of physical processor packages.

Each worker would ideally initialize and repeatedly use its own decoding buffers and processing state. Keeping the worker and those pages together can reduce remote-memory traffic. But there are three separate questions:

CPU affinity does not relocate existing buffers. Under the usual local allocation policy, initializing newly backed pages on their owning worker can help. It is not a guarantee that every allocation is fresh or local, and it does not place all kernel networking buffers. Linux memory-policy documentation.

Now imagine that six busy connections reach one worker while another worker receives none. Both workers may have excellent memory locality. The first still queues work while the second has nothing to do.

We need to understand the traffic-placement decision before improving it.

2. One service port, several listening sockets

A socket is an operating-system endpoint for communication. A TCP server first creates a listening socket, binds it to an address and port, and starts listening. An accept() operation returns a connected socket for a client connection. That connected socket is distinct from the listener.

With Linux's SO_REUSEPORT option, several compatible sockets can listen on the same address and port. Each worker can own its own listening socket and accept queue. Clients still connect to one service endpoint; they do not need a different port for every worker.

In the plain path used by the inspected engine, Linux chooses a listener using a connection hash. The source and destination addresses and ports provide connection diversity. Hashing does not promise an equal share when there are only a few connections. Engine socket creation; Linux selection path.

Enlarge diagram +

Once a TCP connection is established and accepted, its bytes are read through its connected socket. The receiver on that worker decodes and processes them. Adding another listener does not redistribute the existing connection.

This is why connection lifetime matters. An unlucky choice for a short connection disappears quickly. A busy connection that lasts for hours can keep one worker overloaded for hours.

There is no requirement to put a shared application-level acceptor in front of these workers. Linux already has a point at which it chooses a socket. We can change the policy at that point.

3. eBPF changes the selection policy, not the application protocol

eBPF, or extended Berkeley Packet Filter, lets an application supply a small program that Linux runs inside the kernel. Three pieces matter here.

The program contains the decision logic: for example, choose a listening socket. A hook is a defined point in kernel processing that invokes the program. Attaching to a socket-selection hook means the program runs when Linux needs that decision, not whenever the application wants another worker scheduled.

A map is a kernel-managed data structure that stores configuration or state between invocations. Userspace can publish socket references and NUMA ranges in maps; the program can read those entries and advance a counter.

Userspace loads the program and attaches it to the hook. Before accepting it, the kernel verifier checks memory accesses and execution constraints. Once attached, Linux invokes the program directly: the selection does not require a call out to a separate user-space service.

In observability, eBPF programs collect observations. Here, a networking hook makes a decision: which socket should receive this new connection or datagram?

The program type is BPF_PROG_TYPE_SK_REUSEPORT, attached to a reuseport group using SO_ATTACH_REUSEPORT_EBPF. For TCP, the selector chooses a listening socket for a new connection, not for each packet or HTTP/2 stream on an established connection.

Applications keep ordinary TCP sockets, accept(), TLS, and protocol decoding; this is not an XDP or AF_XDP ingestion architecture. Linux reuseport selftest; TCP socket lookup.

Userspace describes the workers; the kernel makes the small decision

For this selector, the maps hold:

The socket map is specifically a BPF_MAP_TYPE_REUSEPORT_SOCKARRAY. Userspace supplies socket file descriptors when populating it; the kernel holds the corresponding socket references. The BPF helper bpf_sk_select_reuseport() selects a socket through a map key and checks that it belongs to the applicable reuseport group. Helper contract; kernel checks.

Enlarge diagram +

This separates two responsibilities. The controller knows which workers exist and where they should run. The BPF program uses a compact description of those decisions. It does not discover the application's topology, decode logs, or predict how expensive a connection will become.

The verifier does not prove that this policy is fair, fast, or operationally correct. Those remain application and measurement questions.

4. Walk through a two-node selection policy

Return to our two-node machine. Use these illustrative worker assignments:

The CPU numbers are deliberately not the socket-map keys. Real CPU and NUMA IDs need not be dense. Userspace builds a dense socket layout and records where each node's range starts.

Linux provides bpf_get_numa_node_id(), which returns the NUMA node of the CPU executing the BPF program. At this hook, that is the CPU doing this part of receive processing, not the worker that userspace has yet to select. Helper documentation.

The proposed policy is:

1. Find the socket range for that receive CPU's NUMA node.
2. If the range has listeners, choose the next one by round-robin.
3. Otherwise, round-robin across all eligible sockets.

Enlarge diagram +

Follow one incoming connection

An incoming TCP connection reaches the selector on CPU 9, on node 1. Its candidates are socket-map keys 4 through 7. Suppose the round-robin counter supplies ticket 6:

start = 4
count = 4
ticket = 6

key = start + (ticket % count)
    = 4 + (6 % 4)
    = 6

Key 6 selects the listener owned by the worker on CPU 10. Linux handles TCP connection setup and queues the connection for that listener. The worker calls accept() and receives a connected socket. The next tickets select keys 7, 4, and 5.

Both CPU 9 and CPU 10 are on node 1. NUMA-local does not mean the same CPU, and it does not mean that every byte involved is already in node-1 memory.

If receive processing instead runs on a node with no registered local listeners, the policy uses the global range. Unknown worker placement should not be mislabeled as node 0; those sockets can remain global candidates without a claim of local placement.

Two different fallbacks

The global policy fallback is not the same as the kernel fallback:

Here is policy pseudocode, not a compilable program or an otel-arrow API:

range = local_ranges.lookup(bpf_get_numa_node_id())
if range is absent or range.count == 0:
    range = whole_group

if range is absent or range.count == 0:
    increment(metadata_fallbacks)
    return SK_PASS

ticket = atomic_fetch_add(counter_for(range), 1)
key = range.start + (ticket % range.count)

error = bpf_sk_select_reuseport(context, sockets, &key, 0)
if error != 0:
    increment(selection_fallbacks)

return SK_PASS

For this selection hook, SK_PASS with a successfully selected socket uses that socket. With no selected socket, it permits the ordinary kernel selection path. Returning SK_DROP would mean something very different. Program result handling; fallback to normal selection.

The pseudocode assumes a validated, bounded layout and an appropriate counter implementation. A real loader must check range bounds, socket eligibility, and map capacities. Configuration updates need a coherent publication strategy; separate map writes are not automatically one atomic configuration change.

Fallback avoids an intentional selector drop; it is not a delivery guarantee. Full queues, socket failures, or a worker that stops accepting can still lose or delay traffic.

5. Which NUMA node is receiving the traffic?

The selector observes the CPU currently doing receive processing, not a fixed hardware location. Locality-first selection can overload one node while another remains idle.

Expand Optional advanced reading below for the receive-path mechanics and Figure 4, or skip it on your first pass.

Optional advanced reading: receive queues, CPU steering, and Figure 4

A network interface controller, or NIC, receives packets. Many NICs provide multiple receive queues. Receive Side Scaling (RSS) usually hashes packet headers to choose a queue. Interrupt affinity influences which CPU handles that queue's notifications and receive work.

Linux can steer processing further. Receive Packet Steering (RPS) can send subsequent protocol processing to another CPU. Receive Flow Steering (RFS) additionally uses information about where the application consumes a flow. Interrupt handling, driver polling, and protocol processing are related, but they are not all the same execution point. Linux networking scaling guide.

Enlarge diagram +

That node ID does not tell us:

• where the client is;
• which NUMA node the NIC is attached to;
• where every packet buffer was allocated;
• where future packets of the established connection will be processed.

Selecting a node-1 listener does not move earlier receive work to node 1. Nor does it migrate buffers allocated on node 0. It aligns one application ownership decision with the receive-processing location observed at that moment.

The caveat matters even without the networking details. Suppose almost all receive processing occurs on node 0, and both nodes have four receiver workers. A locality-first selector keeps choosing node-0 listeners. Those four workers can saturate while the four on node 1 remain idle.

The global fallback does not solve that case: node 0 has listeners, so the policy never enters it. "No local listener" and "local workers are too busy" are different conditions.

A capacity-aware spillover policy is possible, but it would need additional load signals, decisions about their freshness, and a way to avoid oscillating between groups. It is not hidden inside the simple round-robin policy.

Locality and utilization have to be evaluated together. Spreading receive work across nodes may help utilization, but can itself introduce traffic across the network device's memory domain. There is no universal instruction to spread all receive interrupts across all CPUs.

6. A connection is not a request, and UDP is different again

HTTP/2 multiplexing sets a hard boundary

gRPC carries remote procedure calls (RPCs) as HTTP/2 streams. A stream is an independently identified exchange within a connection. Many streams can share one HTTP/2 connection, which normally means one TCP connection.

If the checkout service sends most of its logs through one such connection, the listener choice gives that connection to one receiver worker. Opening more RPC streams inside it does not ask reuseport to choose another worker.

Enlarge diagram +

For this architecture, upstream connection fan-out is part of the design. Verify actual transport connections, not just the number of application stubs, channels, RPCs, or OTAP streams. A gRPC channel is an abstraction that can use zero or more HTTP/2 connections; library behavior and connection reuse matter. gRPC performance guidance.

More connections cost sockets, handshakes, buffering, and potentially more independent compression state. OTAP (OpenTelemetry Protocol with Apache Arrow) maintains per-stream schemas and dictionaries, so distributing work and preserving compression efficiency are related trade-offs. otel-arrow load-balancing discussion.

Recycling an OTAP stream can bound its state without replacing the underlying TCP connection. Conversely, periodically replacing one connection can change its worker over time, but does not create concurrent fan-out.

An HTTP/2-aware proxy can distribute separate RPCs across backend connections. It does not split an already-running streaming RPC into independent work without application-level changes. Another option is to decode on an ingress worker and send batches through an internal topic to processing workers. That adds an application queue and a possible NUMA handoff, but moves the balancing boundary from connections to batches.

Equal connections can carry unequal work

Imagine four workers with one connection each. Three connections carry a small trickle of logs; the fourth carries large batches at a much higher rate. The connection histogram is perfect and the CPU balance is not.

The selector cannot know a connection's future record rate, payload shape, or decoding cost. It improves placement opportunities, not knowledge of future load. One sufficiently hot connection can still dominate one core.

UDP can distribute a sender and lose worker affinity

UDP delivers datagrams rather than a TCP byte stream, and has no equivalent accepted-connection model. For datagrams arriving through a reuseport socket group, selection can happen per datagram. A round-robin selector can therefore distribute one busy sender's datagrams across workers instead of keeping the sender's flow on one socket.

That can be useful for independently decodable messages. It can be wrong for a receiver that keeps per-sender parsing, aggregation, deduplication, or other flow state on one worker. Successive messages can reach different workers, and concurrent processing can change application-observed order. UDP itself does not promise ordered, reliable delivery, but removing an existing worker-affinity property still changes application behavior.

The cost also moves to a hotter path. The shared atomic counter now sees datagrams, not just connection-selection activity. High packet-rate UDP needs a separate measurement and semantic review, not a conclusion borrowed from a TCP benchmark.

7. Implementation deep dive: OpenTelemetry Arrow

OpenTelemetry Arrow's Rust engine is a useful case study because it already knows where its pipeline workers are placed. The proposed selector can consume that knowledge instead of creating a second topology planner.

At the inspected source revision, the boundary is:

Placement implementation; topology discovery; listener contract; socket helpers.

The engine attempts worker affinity before building the pipeline. Affinity failure is logged but execution continues, so a planned CPU is not proof that a deployment actually pinned successfully. Worker initialization.

Also, the default placement policy is packing, not automatic spreading across two nodes. Our four-plus-four example requires an actual placement with workers on both nodes. The selector cannot invent a local worker where the controller has placed none.

Startup coordination is different from a shared acceptor

The later design proposal describes a lazy startup quorum: all expected receivers arrive before a group is materialized. The intended sequence is:

1. The controller resolves placement and registers the expected listener group before launching the workers.
2. Each receiver reaches its socket-acquisition step.
3. Once every expected member has arrived, the manager creates the group's sockets. TCP sockets must be bound and listening before insertion into the reuseport socket array; UDP sockets must be bound.
4. The loader publishes the socket layout and attaches the selector.
5. Each receiver gets its assigned socket and begins serving it.

This coordination is setup work. It does not introduce an application thread that accepts every connection and forwards it to another worker. Historical design proposal.

Why wait for the receivers? Creating eight listeners eagerly when only seven receivers ever start can leave a listening socket with no accept loop. The kernel can queue traffic to it even though no worker serves it. Startup quorum avoids publishing a socket for a receiver that never reaches that startup step.

Quorum is not perpetual health monitoring. A receiver can still stall or fail afterward. Even during normal startup, there is a short interval between listen() or UDP bind() and selector attachment; traffic in that interval can use normal kernel selection.

The map must describe live ownership, not just intended ownership

The kernel's reuseport group and the application's logical group must agree. Two pipeline configurations that reuse the same effective bind identity can collide at the kernel level. A controller cannot create isolation just by assigning them different application names.

The proposal treats ambiguous or overlapping groups conservatively rather than attaching a selector that knows about only some of the sockets. Wildcard addresses, dual-stack behavior, and independently created sockets all deserve attention here.

If a socket closes, the kernel removes its reuseport-sockarray reference. Separate range and count metadata do not automatically shrink with it. Selecting the now-empty slot can fail and fall back to normal selection. That is degraded placement, not live repair. Sockarray detach behavior.

An open socket whose worker has stopped accepting is worse: it can remain selectable. The BPF map is not a worker-health check. Shutdown, descriptor ownership, and replacement must be coordinated with the receiver lifecycle. Repairing maps or replacing groups coherently is distinct from migrating already accepted TCP connections and protocol state.

The initial proposal leaves live group replacement as future work. It should not be read as a promise of zero-downtime worker migration.

8. Inspect the machine before changing the policy

A useful first exercise does not require loading BPF or changing an IRQ. Inspect the topology and the actual receiver process on a Linux test host. These commands are read-only; replace the example identifiers.

lscpu -e=CPU,NODE,SOCKET,CORE,ONLINE
numactl --hardware

pid=1234  # Replace with the receiver's process ID.
grep -E 'Cpus_allowed_list|Mems_allowed_list' "/proc/$pid/status"
ps -L -p "$pid" -o pid,tid,psr,comm
taskset -a -p -c "$pid"

The PSR column is an observation of a thread's processor, not its allowed CPU set. taskset with -a queries all the process's tasks. Check the worker threads rather than assuming the main thread's affinity describes all of them. A logical CPU may also share execution resources with an SMT sibling.

Next inspect the interface that actually receives the traffic:

iface=eth0  # Replace with the receiving interface.
cat "/sys/class/net/$iface/device/numa_node"
ethtool -l "$iface"
ethtool -x "$iface"
grep -E 'CPU|eth0' /proc/interrupts
cat "/sys/class/net/$iface/queues/rx-0/rps_cpus"

Change the interrupt search to match the driver or queue names on the machine; they may not contain the interface name. Inspect all relevant queues, not only rx-0. After identifying a queue's IRQ number, read its /proc/irq/IRQ/smp_affinity_list.

Not every interface exposes a PCI device NUMA node or an RSS table. A node value of -1 means unknown, not node 0. Virtual interfaces and containers can hide or change the visible receive path. A reported error or missing file is information to investigate, not a reason to invent topology.

Inspect transport fan-out as well:

ss -tn state established '( sport = :4317 )'

This shows established server-side TCP connections for the example port in the current network namespace. It does not count HTTP/2 streams and does not, by itself, identify the owning pipeline worker. Inside a proxy deployment, the relevant receiver connections are the proxy's backend connections.

Finally, inspect page placement:

numastat -p "$pid"
head -n 20 "/proc/$pid/numa_maps"

These show page placement and related accounting, not a count of later remote-memory reads. To investigate actual memory-access behavior, use processor-supported performance counters or sampling, with their documented limitations. There is no one portable perf event that measures the same thing on every processor.

Do not tune RSS, IRQ affinity, memory policy, and the selector all at once. Record the baseline first. Use a dedicated environment for changes rather than altering shared-host networking to follow an article.

9. Measure efficiency, not just a prettier connection histogram

Once an implementation is available, test the whole ingestion path. A single-node machine can exercise attachment, fallback, and per-worker distribution. It cannot establish a cross-NUMA benefit.

Begin with comparable worker CPUs, actual connection counts, batch sizes, record shapes, downstream capacity, and offered traffic. Warm up the workload, repeat runs, and record the kernel, hardware topology, allocator, receive steering, and resource limits.

A useful comparison separates the layers:

The global-only variant is an experimental control, not a claim that the current engine exposes such a configuration.

Use more than one traffic shape:

Measure delivered records per second, tail latency such as p99, CPU time per delivered record, and each worker's utilization and queue behavior. Include drops, rejections, retries, retained memory, and downstream backpressure. More accepted connections or higher apparent ingest throughput is not an improvement if fewer records reach the destination.

Include networking CPU cost, not just the receiver process's user-space CPU time. Moving work into a kernel hook or onto another CPU does not make that work free. Where supported, inspect remote-memory accesses, memory bandwidth, and cache-coherence traffic alongside page placement.

At fixed offered load, compare latency and resource cost. For throughput, increase load to a defined limit while recording loss and latency rather than comparing two arbitrary saturation points. Keep the load generator and downstream destination from becoming the hidden bottleneck.

The outcome may be that plain reuseport is already good enough. It may be that upstream fan-out helps more than a custom selector. Or local selection may reduce cross-node traffic enough to improve useful throughput. Those are all meaningful results when the experiment separates the causes.

The central lesson is small: eBPF can connect worker-placement knowledge to a kernel socket decision. It does not replace worker placement, memory policy, receive-path tuning, or sufficient parallel input. The right worker is not just the nearest one; it is one that can usefully process the traffic the system actually receives.

Version scope: OpenTelemetry Arrow source 2eba9fcab and the historical selector-design revision 7f7fa0de, inspected on October 10, 2026. Kernel implementation details were checked against Linux source f0100363d, a September 21, 2026 development snapshot. These are source references, not a statement about a running or released kernel. Current project behavior and proposed architecture are distinguished above.

No selector was loaded, networking settings changed, or NUMA performance benchmark run for this article. The policy pseudocode is explanatory, not a compiled or runtime-validated implementation.