Hard Real-Time Robot Control with ROS 2 Control: Using lock_memory, CPU Affinity and Thread Priority on Jazzy and Lyrical

Real-time control of a robotic manipulator is rarely the fault of the control law itself. The tracking error on a scope is more often the product of a controller update that arrived too late — delayed by a page fault, stolen off its core by a scheduler, or blocked behind a lower-priority thread holding a lock. For teams building reliable robots with ROS 2 Control, the difference between a demo controller and a production one is exactly this: keeping the controller loop deterministic under load. On the Jazzy and Lyrical distributions, ros2_control exposes runtime knobs — lock_memory, cpu_affinity, thread_priority, plus scheduler policy control from realtime_tools — that together move a controller loop from good enough to genuinely hard real-time. This guide walks through each knob step by step, explains how to measure the improvement in cycle jitter, and highlights the breaking API changes that landed in 2026 that every maintainer must know before upgrading.

Why a standard ROS 2 Control loop is not hard real-time

A stock ros2_control_node runs its update loop on a thread managed by the kernel’s CFS scheduler, in a process whose pages are demand-paged into RAM. Two things can break the cadence that a controller promises. First, a cold page fault stalls the thread for a number of microseconds — sometimes more than one cycle — while the kernel pulls the page in from disk. Second, the thread is a normal citizen of the scheduler: it is preemptible and competes for the core like everything else. Under heavy CPU load from a vision pipeline or a path planner, the controller update stalls, the actuator command goes stale, and the robot’s loop latency spikes.

The fix is the classic three-part real-time recipe on Linux: lock memory so the pages are pinned in physical RAM, pin the thread to a dedicated CPU core so nothing evicts it, and raise its scheduler priority so the kernel always runs it ahead of ordinary threads. On Jazzy and Lyrical this is no longer something you script by hand with chrt and taskset after startup; it is largely configured through parameters on the ros2_control_node itself.

The table below summarises the four principal techniques covered in this guide, the mechanism behind each, how to enable it, and the benefit to expect.

Technique What it does How to enable Typical benefit
lock_memory Locks the process memory into physical RAM at startup to prevent demand-paging page faults in the control loop Set the lock_memory parameter to true on ros2_control_node Eliminates timing spikes caused by page faults; tighter worst-case cycle time
cpu_affinity Binds the controller-manager process to a specific CPU core, isolating the loop from scheduler migration Set the cpu_affinity parameter to the target core (disabled by default) Stable cache locality and no preemption by other processes on that core
thread_priority Sets the scheduler priority of the controller-manager real-time thread Set the thread_priority parameter to the desired priority value Controller always runs ahead of ordinary threads, reducing latency
Scheduler policy (configure_sched_rr()) Applies the SCHED_RR real-time scheduling policy to the calling thread Call realtime_tools::configure_sched_rr() from the control thread (available from realtime_tools 3.12.0) Round-robin real-time scheduling for deterministic periodic execution

Lock memory to eliminate page faults

The first and most mechanical step is memory pinning. When a process touches a page of memory for the first time, the kernel faults it in from the backing store — a stall in the middle of the control cycle. The ros2_control_node node gained a lock_memory parameter in the Jazzy release specifically to avoid this class of fault. According to the Jazzy release notes, the new parameter “lock[s] memory at startup to physical RAM in order to avoid page faults.” In practice this means the process locks the code and data the loop touches into physical RAM before the control loop starts, so they are resident from the first cycle onward.

Enabling it is a one-line addition to the controller-manager parameter file:

ros2_control_node:
  ros__parameters:
    lock_memory: true

With lock_memory enabled, the measured worst-case cycle time stops being dominated by cold-page stalls. For a fresh boot-up where libraries have never been touched, the effect is most visible; after the process has run long enough for memory pressure to swap pages out, the pinning prevents those pages from leaving RAM again.

Pin the process to a CPU core

Locking memory removes one source of delay, but the thread can still be migrated between cores by the kernel, and it still competes with other processes. The cpu_affinity parameter, which the Jazzy release notes describe as “bind[ing] the process to a specific CPU core,” addresses both. It is disabled by default, so nothing changes unless you ask for it.

ros2_control_node:
  ros__parameters:
    cpu_affinity: 3   # bind the controller-manager to core 3

Choosing the right core is an exercise in knowing your machine. The common pattern is to reserve one physical core — ideally not shared with interrupt handlers, the rclcpp executor threads, or the driver real-time threads — and dedicate it to the controller loop. On a multicore board it is typical to leave core 0 for the OS housekeeping, push the controller manager to a higher-numbered core, and let an orchestration layer pin the other real-time processes to their own isolated cores.

The software knobs in this guide assume the rest of the platform is already tuned for deterministic execution — a performance CPU governor rather than the default powersave, and IRQ affinities moved off the reserved core.

Raise the scheduler priority and policy

Memory pinning and core affinity control where the loop runs and what it holds in memory; the scheduler policy and priority control when it runs relative to every other thread on the box.

The thread_priority parameter

Since Jazzy, the ros2_control_node node accepts a thread_priority parameter to set the scheduler priority of the controller manager’s real-time thread (the Jazzy release notes reference #1820 for this change). A higher value means the thread’s update() runs ahead of lower-priority threads on the same core.

ros2_control_node:
  ros__parameters:
    thread_priority: 80

The hardware-communication layer and the control update should sit above the application threads (vision, planning, communication), and the real-time threads should sit above that again. Reasonable relative ordering — rather than blindly picking the maximum value — tends to keep the system responsive across the board.

SCHED_RR via realtime_tools::configure_sched_rr()

Priority alone only meaningfully governs behaviour under the POSIX real-time scheduling policies SCHED_FIFO and SCHED_RR. The realtime_tools package — one of the core real-time utilities maintained by the ros-controls project — historically offered a helper to apply a real-time policy. In the 3.12.0 release of realtime_tools (dated 2026-09-03), a new configure_sched_rr() helper was added specifically for the SCHED_RR scheduling policy. SCHED_RR (round-robin) differs from strict FIFO in that equal-priority threads take turns in time slices, which is often the safer choice when more than one real-time thread of the same priority must share the cores.

A controller author integrates it in the controller’s on_activate() or setup path, typically by calling the helper on the thread that executes the update loop:

// Within the controller, from the realtime update thread
int prio = ...; // your configured priority
if (realtime_tools::configure_sched_rr(prio)) {
  RCLCPP_INFO(get_logger(), "Applied SCHED_RR with priority %d", prio);
}

The helper returns success or failure, so the controller can log clearly whether it actually secured the real-time policy. Note that applying a real-time scheduler policy requires privilege: the process must run as root, carry the CAP_SYS_NICE capability, or be launched under a service that grants real-time scheduling; otherwise the call fails and the thread silently falls back to the normal scheduling class.

Measure the cycle jitter before trusting the knobs

Tuning without measurement is guesswork. The controller manager provides exactly the introspection hooks you need. Since Jazzy, the controller manager publishes ~/statistics/names and ~/statistics/values topics that expose the execution time and periodicity of the different entities running in the realtime loop (see the Jazzy release notes, referencing #2449). These stats let you observe how long each controller’s update() actually runs and how far the actual period drifts from the commanded one.

A simple jitter measurement involves running the controller for a fixed window — say 100,000 cycles at 1 kHz — and computing from the statistics topics the mean and worst-case cycle time, and the period deviation from the nominal.

Measure in a sequence: record a baseline with the machine under synthetic CPU load, then enable lock_memory, add cpu_affinity, and finally raise the priority and apply the real-time policy. The worst-case number should compress meaningfully at each step; when it does, you have evidence the loop is now hard real-time rather than hopeful optimism.

Handle overruns deliberately

Real-time loops do occasionally overrun their budget, and how the system responds matters. The controller manager exposes two parameters to control its behaviour when an overrun occurs: overruns.manage and overruns.print_warnings (from the Jazzy release notes, referencing #2546). These govern whether the controller manager actively manages overrun conditions and whether it prints warnings, rather than silently continuing. During tuning it is useful to leave overruns.print_warnings on so problems are visible; in production, teams often tune overruns.manage to match the safety requirements of the platform.

ros2_control_node:
  ros__parameters:
    overruns:
      manage: true
      print_warnings: true

An overrun that is handled deliberately — a stopped trajectory, a held command, or a logged fault — is far safer than one that is silently absorbed.

The monotonic clock and read-write-update semantics

Real-time loops must not be hostage to wall-clock adjustments. Since Jazzy, when the monotonic clock is in use, “all the hardware components will receive the monotonic time in their read and write method, instead the controllers will always receive the ROS time in their update method irrespective of the clock being used” (Jazzy release notes, referencing #2046). The design intent is clear: the low-level hardware read/write cycle runs on monotonic time that cannot jump when the system clock is adjusted, while controllers that reason about ROS time (for trajectory interpolation and action timeouts) keep getting the ROS time they expect. The realtime loop’s cadence stays uniform even across an NTP sync or an operator manually changing the clock.

Switch controllers inside the real-time loop

A service-robot that must hand control from one controller to another mid-maneuver needs the switch itself to be timely. The controller manager supports switching — activating and deactivating — controllers in both real-time and non-real-time modes. The relevant knob is the --switch-asap option (as quickly as possible), which is the default, set to keep the earlier behaviour (Jazzy release notes, referencing #2610); additionally, the default strictness of switch_controllers can now be chosen using ROS 2 parameters rather than relying on a single hard-coded behaviour.

Two practical costs of a controller switch matter for real-time work. First, during the switch the controller manager runs additional bookkeeping — initialising new controller interfaces and releasing old ones — which briefly competes with the update loop. Second, the strictness setting decides how the manager behaves if the switch cannot be completed exactly as requested. For a gripper hand-off or a leg-controller swap mid-gait, choosing the appropriate strictness helps avoid an abort in the middle of a motion when a late interface claim would otherwise derail the switch.

2026 breaking changes: the handle/export removal

Upgrading to Lyrical brings a set of intentional breaking changes. The most consequential is the removal of the deprecated handle and export APIs. As the ros2_control team’s Lyrical release summary explains, “the handle and export APIs deprecated since Jazzy have been removed.” Concretely, this means:

  • export_state_interfaces() and export_command_interfaces() are gone from hardware components. The framework now creates the interfaces directly from the <ros2_control> tag in the URDF, and values are accessed through set_state / get_state and set_command / get_command.
  • StateInterface and CommandInterface can no longer be built from a raw double*. Handles now own their value, which removes the aliasing hazard that made a pointer owned elsewhere a real-time safety risk.

The migration path is covered in the ros2_control migration guide, which provides step-by-step instructions. Alongside the removal, Lyrical adds float32, int8/16/32 and uint8/16/32 handle data types beyond the traditional double and bool, which matters for hardware that quantises encoders or commands natively, and new interface configuration types INDIVIDUAL_BEST_EFFORT and REGEX for more flexible interface claiming.

For the real-time developer, the handle change is the one to plan for first. Any custom hardware component or chainable controller that still reads or writes values through exported pointer handles will not compile under Lyrical until it is migrated to the setter/getter accessors. Budget migration time for the controller codebase before expecting the real-time tuning knobs above to be usable on the new release.

A complete real-time configuration on Jazzy

Pulling the pieces together, a controller-manager parameter file that pins, locks, and prioritises on Jazzy looks like this:

ros2_control_node:
  ros__parameters:
    update_rate: 1000          # 1 kHz control rate
    lock_memory: true          # pin RAM to avoid page faults
    cpu_affinity: 3            # dedicated core for the control loop
    thread_priority: 80        # real-time priority for the update thread
    overruns:
      manage: true
      print_warnings: true

When the host is a systemd service, granting the real-time capability is part of making all this work:

# /etc/systemd/system/ros2_control.service.d/override.conf
[Service]
LimitRTPRIO=99
LimitMEMLOCK=infinity

Summary: the path to deterministic control

Hard real-time robot control with ROS 2 Control in Jazzy and Lyrical is a configuration exercise as much as a code exercise. Order the work deliberately: measure a baseline, enable lock_memory to kill page-fault stalls, bind the process with cpu_affinity to give the loop a private core, raise thread_priority and use realtime_tools’s configure_sched_rr() for the actual real-time policy, and then re-measure the cycle jitter from the controller manager’s statistics topics to confirm the improvement. Keep overrun handling explicit through overruns.manage and overruns.print_warnings, lean on the monotonic clock separation for the read-write-update cadence, and choose the right strictness for switch_controllers. Before adopting Lyrical, budget time for the removed handle/export APIs — the exported raw pointers are gone, and handles now own their values. Real-time is earned one deterministic cycle at a time; these parameters are how the ROS 2 Control stack lets you earn it.


Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top