The "In-Flight" Window: Task Migration Safety and Kernel Invariants in RTEMS

When you physically rip a running real-time task off one CPU core and teleport it onto another, while the kernel is live, while other tasks are firing, while hardware interrupts are going off, you are doing something deeply dangerous.

This is exactly what the Distributed Flexible Locking Protocol (DFLP) does. And the only reason it doesn’t immediately crash the entire RTEMS (Real-Time Executive for Multiprocessor Systems) kernel is a carefully orchestrated system of kernel invariants: non-negotiable rules the SuperCore enforces during the terrifying milliseconds when a thread is physically “in-flight” between processors.

This post is about those invariants. Specifically, we’re going to look at two of the most critical protection mechanisms in the RTEMS 7 SuperCore: the STATES_LIFE_IS_CHANGING lifecycle bit and the pin_level dispatch counter. Get either one wrong, and your system panics with INTERNAL_ERROR_BAD_THREAD_DISPATCH_DISABLE_LEVEL, probably the most terrifying error message in the RTEMS codebase.

1. The Problem: What Happens During Migration?

Let’s set the scene. A task (Task_A) is executing on Core 3. It requests a DFLP lock bound to Core 1 (the Synchronization Processor where the target hardware lives). The lock is free, so DFLP kicks off Temporary Task Migration.

Here’s the timeline of what happens inside the SuperCore:

Time T0:  Task_A is executing on Core 3.
Time T1:  DFLP begins migration. Task_A is removed from Core 3's scheduler.
Time T2:  ???  <-- THE IN-FLIGHT WINDOW
Time T3:  Task_A is inserted into Core 1's scheduler.
Time T4:  Task_A wakes up on Core 1 and executes the critical section.

T2 is the danger zone. During this window, Task_A is in a transient state. It’s been pulled off Core 3 but hasn’t landed on Core 1 yet. It exists in a liminal space: no longer owned by any scheduler, not legally dispatchable, but still alive in the kernel’s internal data structures.

What Can Go Wrong at T2?

Pretty much everything. While Task_A is in-flight:

• The Tick Interrupt fires on Core 3. The watchdog timer sees that Task_A’s deadline is approaching and tries to preempt it. But Task_A isn’t on Core 3 anymore. The dispatcher tries to look up a Scheduler_Node that no longer belongs to this CPU’s ready queue. Undefined behavior.
• A POSIX signal arrives. Another thread calls pthread_kill() targeting Task_A. The signal delivery path tries to unblock Task_A and insert it into a run queue. But which run queue? It’s between cores. Data structure corruption.
• rtems_task_delete() is called on Task_A. The deletion path tries to extract Task_A from its current scheduler. But it’s mid-migration, and the Thread_Control block is in an inconsistent state. The kernel walks a corrupted linked list. Fatal panic.

These aren’t hypotheticals. These are the exact failure modes that multi-processor locking protocol implementers have to guard against.

2. The Shield: STATES_LIFE_IS_CHANGING

The RTEMS SuperCore has a dedicated protection mechanism for exactly this situation: the STATES_LIFE_IS_CHANGING bit in the thread’s lifecycle state word.

Every Thread_Control block in RTEMS has a Life.state field, a bitmask that tracks the thread’s lifecycle status. The STATES_LIFE_IS_CHANGING bit is basically the kernel saying: “This thread is in a transient state. Do not touch it.”

How DFLP Uses It

Before starting migration, DFLP sets this bit:

/* Protect the thread from asynchronous kernel interference */
thread->Life.state |= STATES_LIFE_IS_CHANGING;

Once this bit is set, the kernel’s internal subsystems follow a strict contract:

This bit is the single most important invariant for migration safety. Without it, every asynchronous kernel event becomes a crash vector during the in-flight window.

The Handshake

After migration completes, once Task_A has been inserted into Core 1’s scheduler and is safely dispatchable, DFLP clears the bit:

/* Migration complete. Thread is safe to dispatch. */
thread->Life.state &= ~STATES_LIFE_IS_CHANGING;

This clear operation has to happen after the full scheduler insertion is done. If you clear it one instruction too early, before the Scheduler_Node is fully linked into Core 1’s Red-Black Tree, you open a window where the dispatcher can see the thread but the scheduler data is incomplete. The result: a corrupted priority aggregation tree and an eventual panic.

3. The Gate: executing->Scheduler.pin_level

The second critical invariant is the thread’s pin_level. This is a counter in the Scheduler_Node that tracks whether the thread is currently “pinned” to a specific CPU, meaning the scheduler is forbidden from moving it.

What Creates Pin Level?

Pin level gets incremented by operations that need to temporarily lock a thread to its current CPU. The most important one here is Sticky Scheduling, the mechanism used by protocols like FMLP-S (FMLP-Short), where the thread acquires a lock and must not migrate until the lock is released.

When FMLP-S calls _Thread_Priority_update_and_make_sticky(), it effectively does:

executing->Scheduler.pin_level++

When the lock is released and _Thread_Priority_update_and_clean_sticky() runs:

executing->Scheduler.pin_level--

The Invariant: pin_level == 0 Before Migration

Here is the rule that DFLP absolutely cannot break:

A thread MUST NOT be migrated via _Thread_Set_CPU() unless its pin_level is exactly zero.

If pin_level > 0, the thread is currently pinned by an active sticky lock (like an FMLP-S semaphore). Trying to migrate a pinned thread creates an impossible situation:

1. The sticky lock’s _Thread_Dispatch_disable_critical() call was made on the original CPU’s Per_CPU_Control.
2. Migration moves the thread to a different CPU.
3. When the sticky lock is eventually released, _Thread_Dispatch_enable() decrements the dispatch disable counter on the new CPU’s Per_CPU_Control.
4. The original CPU’s counter was never decremented. It is now permanently elevated.
5. At the next dispatch point, the kernel detects the asymmetry: INTERNAL_ERROR_BAD_THREAD_DISPATCH_DISABLE_LEVEL.

This isn’t a subtle data corruption. This is an immediate, unrecoverable kernel panic.

DFLP’s Guard

Before starting any migration, DFLP has to check:

_Assert( executing->Scheduler.pin_level == 0 );

If a task is holding a sticky lock (FMLP-S, MrsP) and also requests a DFLP lock that requires migration, DFLP must either:

1. Reject the request with STATUS_UNAVAILABLE, or
2. Enqueue the task into the DFLP wait queue and defer migration until the sticky lock is released and pin_level drops to zero.

This interaction between sticky and migratory protocols is one of the trickiest design challenges in the entire DFLP port.

4. The Dispatch Disable Symmetry Contract

Both invariants (STATES_LIFE_IS_CHANGING and pin_level) are ultimately enforcing a deeper architectural principle: dispatch disable symmetry.

The RTEMS SuperCore tracks a per-CPU dispatch disable counter:

Per_CPU_Control *cpu = _Per_CPU_Get();
uint32_t disable_level = cpu->thread_dispatch_disable_level;

Every _Thread_Dispatch_disable_critical() call increments this counter on the current CPU. Every _Thread_Dispatch_enable() call decrements it. At the end of any critical kernel operation, the counter must return to the same value it started with.

If a thread migrates between the disable and enable calls, the disable happens on CPU A and the enable happens on CPU B. CPU A’s counter goes up but never comes back down. CPU B’s counter goes down without going up. Both CPUs now have corrupted dispatch state.

The Lesson from FMLP

This exact scenario is what I ran into during the FMLP-S port. The _FMLPS_Claim_ownership() path calls _Thread_queue_Dispatch_disable() on the CPU where the thread is currently executing. If the thread migrates (because of scheduler helping or a bug in the enqueue path), the corresponding _Thread_Dispatch_enable() fires on the wrong CPU.

The fix required making sure that every _Thread_Dispatch_disable_critical() and _Thread_Dispatch_enable() pair operates on the same Per_CPU_Control pointer, and that the pointer is captured before any operation that could cause the thread to move.

For DFLP, this lesson gets amplified. Migration is not a bug to prevent, it’s the entire protocol. The dispatch disable/enable pairs need to be explicitly designed around the migration boundary, with the disable happening on the source CPU, migration executing, and the enable happening on the destination CPU using the source’s saved Per_CPU_Control reference.

5. Wrapping Up

The “In-Flight Window”, that transient period when a real-time task is physically between CPU cores, is the most dangerous state a thread can be in on a modern SMP kernel. Every asynchronous kernel subsystem (dispatcher, signals, deletion, watchdogs) becomes a potential attack vector during this window.

RTEMS protects against this with two critical mechanisms:

1. STATES_LIFE_IS_CHANGING: A lifecycle bit that tells the entire kernel “do not touch this thread, it is in transit.”
2. pin_level == 0: An assertion that the thread is not pinned by any sticky protocol before migration begins. Violating this causes dispatch disable counters to go out of sync across CPUs, triggering an unrecoverable INTERNAL_ERROR_BAD_THREAD_DISPATCH_DISABLE_LEVEL panic.

Understanding these invariants is not academic. It’s the difference between a DFLP implementation that works on a simulator and one that works on real silicon under real-time load. The FMLP port already taught this lesson the hard way.

TL;DR

• The In-Flight Window: During DFLP task migration, a thread is temporarily between CPUs, removed from one scheduler but not yet inserted into another. Every asynchronous kernel event (interrupts, signals, timeouts) is dangerous during this window.
• STATES_LIFE_IS_CHANGING: A lifecycle bit in Thread_Control that marks a thread as “in transit.” The dispatcher, signal delivery, thread deletion, and watchdogs all respect this bit and defer their operations.
• pin_level == 0: Must be true before migration. If pin_level > 0 (the thread holds a sticky lock), migration will cause Per_CPU_Control dispatch disable counter asymmetry, leading to a fatal INTERNAL_ERROR_BAD_THREAD_DISPATCH_DISABLE_LEVEL panic.
• Dispatch Symmetry: Every _Thread_Dispatch_disable_critical() / _Thread_Dispatch_enable() pair must operate on the same CPU’s Per_CPU_Control. Migration makes this tricky because the source CPU’s pointer has to be saved and passed across the migration boundary.