Multiprocessing has become the standard in modern embedded hardware. As systems grow more complex, single-core microcontrollers are making way for multi-core SoCs (System on Chips). For Real-Time Operating Systems (RTOS) like RTEMS, supporting multiprocessing while guaranteeing hard real-time constraints is a fascinating engineering challenge.

In this post, we’ll dive deep into how the RTEMS (Real-Time Executive for Multiprocessor Systems) handles both Symmetric Multiprocessing (SMP) and Asymmetric Multiprocessing (AMP) architectures.

Symmetric Multiprocessing (SMP) in RTEMS

In an SMP architecture, a single instance of the RTOS manages multiple identical processors. All processors share the same memory space, interrupts, and peripherals. RTEMS has robust SMP support built from the ground up to ensure predictability and cache locality.

1. Clustered Scheduling

One of the most complex parts of SMP in a real-time context is scheduling. Global priority-based schedulers (where the highest priority tasks run anywhere on the system) can suffer from poor cache locality and excessive thread migration.

RTEMS solves this with Clustered Scheduling. The system’s processors are partitioned into disjoint subsets called “clusters.” Each cluster is managed by exactly one scheduler instance. This allows system designers to decouple hard real-time tasks from high-throughput background tasks, guaranteeing best-in-class cache utilization and bounding worst-case latencies. You can use schedulers like the Earliest Deadline First (EDF) SMP Scheduler or the Deterministic Priority SMP Scheduler.

2. Real-Time Locking Protocols

In standard uniprocessor systems, disabling preemption or disabling interrupts is an easy way to protect critical sections. In SMP, doing this on one core doesn’t stop another core from accessing shared data!

To solve mutual exclusion across clusters without causing devastating priority inversion, RTEMS implements several state-of-the-art multiprocessor locking protocols:

  • MrsP (Multiprocessor Resource Sharing Protocol)
  • OMIP (O(m) Independence-Preserving Protocol)
  • FMLP (Flexible Multiprocessor Locking Protocol)

Through a “Scheduler Helping Protocol”, these primitives allow threads to temporarily migrate between schedulers to prevent unbounded priority inversion when waiting on shared resources.

3. Low-Level Synchronization

Under the hood, all RTEMS SMP synchronization (like internal thread queue manipulation) is built on C11 atomic operations. It avoids heavy locks and relies on fine-grained Ticket Locks, sequence locks, and sense barriers, maintaining strict FIFO fairness which is vital for real-time predictability.

Asymmetric Multiprocessing (AMP) / Heterogeneous Systems in RTEMS

While SMP is great for identical cores, what happens if you have a system with a powerful Cortex-A processor next to a low-power Cortex-M microcontroller? Or what if you want multiple isolated RTEMS instances running alongside each other? This is where Asymmetric Multiprocessing (AMP) shines.

In AMP, you effectively run multiple independent operating systems (or bare-metal applications), each on their own core. RTEMS supports AMP systems natively through its Multiprocessing Manager (MPCI).

1. Nodes and the MPCI Layer

Under the RTEMS MPCI framework, each processor running its own RTEMS instance is considered a Node. The Multiprocessor Communications Interface (MPCI) layer handles the routing of messages between these nodes.

RTEMS allows the application designer to designate certain tasks, queues, semaphores, or events as Global Objects. The beauty of this system is transparency. An RTEMS task on Node 1 can send a message to a Global Queue residing on Node 2 using the exact same API it would use for a local queue. RTEMS automatically intercepts the call, packages it into a Remote Request (RQ), and sends it over the MPCI layer to the destination node.

2. True Heterogeneous Support

RTEMS makes no assumptions about the hardware running on other nodes. It seamlessly supports “Heterogeneous Multiprocessing”.

The nodes don’t need to share the same endianness, processor architecture, or alignment rules. RTEMS packet protocols automatically handle data translation (such as Big-Endian to Little-Endian conversions) for its internal object management, meaning you can have a PowerPC node communicating transparently with an ARM node.

3. Proxies and Remote Blocking

When a task on Node A blocks waiting for a semaphore on Node B, RTEMS creates a Proxy on Node B. This proxy acts as a surrogate thread control block, holding the place in the remote queue until the resource becomes available, at which point the MPCI layer wakes the original task on Node A.

Conclusion

RTEMS proves its versatility by tackling modern multi-core challenges on two completely different fronts.

  • If you have an array of identical cores and need a unified, cache-aware real-time environment, RTEMS SMP offers clustered scheduling and advanced real-time locking protocols.
  • If you are building a distributed system or a heterogeneous SoC with completely different architectures, the RTEMS Multiprocessing Manager lets you connect isolated AMP nodes transparently.

Whether you’re developing for deeply embedded automotive controllers or high-end multi-core space flight computers, RTEMS gives you the multiprocessor architecture that fits your hardware.

TL;DR

  • SMP (Symmetric Multiprocessing): One RTOS handles multiple identical cores sharing memory. RTEMS uses Clustered Scheduling to decouple workload priorities and preserve cache-locality, bypassing global priority bottlenecks.
  • AMP (Asymmetric Multiprocessing): Each core runs its own isolated OS (Heterogeneous System). RTEMS connects them transparently using the Multiprocessing Manager (MPCI), enabling seamless cross-architecture message passing.