Radboud University Nijmegen

Real-time Linux (Xenomai)

Background

Real-Time Linux Introduction

Linux is a free Unix-like operating system that runs on a variety of platforms, including PCs. Numerous Linux distributions such as Red Hat, Debian and Mandrake bundle the Linux OS with tools, productivity software, games, etc.

• The Linux scheduler, like that of other OSes such as Windows or MacOS, is designed for best average response, so it feels fast and interactive even when running many programs.
  • However, it doesn't guarantee that any particular task will always run by a given deadline. A task may be suspended for an arbitrarily long time, for example while a Linux device driver services a disk interrupt.
  • Scheduling guarantees are offered by real-time operating systems (RTOSes), such as QNX, LynxOS or VxWorks. RTOSes are typically used for control or communications applications, not for general purpose computing.
• Linux has been adapted for real-time support. These adaptations are termed "Real-Time Linux" (RT Linux).
• Numerous versions of RT Linux are available, free or commercial. Three commonly available free RT Linux versions are
  • RTLinux, developed by New Mexico Tech and now maintained by Wind River Systems.
  • RTAI, The Real-Time Application Interface (RTAI), developed by the Milan Polytechnical University and available at https://www.rtai.org/.
  • Xenomai, see www.xenomai.org. While RTAI is focused on lowest technically feasible latencies, Xenomai also considers clean extensibility (RTOS skins), portability, and maintainability as very important goals.
• These RT Linux systems are patches to the basic Linux kernel source code. Instructions for building an RT Linux system from a the Linux source code are provided with these RT Linux systems. Briefly, this involves setting up the basic Linux system, getting the latest Linux kernel source code from www.kernel.org, patching the kernel source code, and compiling the patched kernel.

How does RT Linux work?

The general idea of RT Linux is that a small real-time kernel runs beneath Linux, meaning that the real-time kernel has a higher priority than the Linux kernel. Real-time tasks are executed by the real-time kernel, and normal Linux programs are allowed to run when no real-time tasks have to be executed. Linux can be considered as the idle task of the real-time scheduler. When this idle task runs, it executes its own scheduler and schedules the normal Linux processes.  Since the real-time kernel has a higher priority, a normal Linux process is preempted when a real-time task becomes ready to run and the real-time task is executed immediately.

How is the real-time kernel given higher priority than Linux kernel?

Basically, an operating system is driven by interrupts, which can be considered as the heartbeats of a computer:

• All programs running in an OS are scheduled by a scheduler which is driven by timer interrupts of a clock to reschedule at certain times.
• An executing program can block or voluntary give up the CPU in which case the scheduler is informed by means of a software interrupt (system call).
• Hardware can generate interrupts to interrupt the normal scheduled work of the OS for fast handling of hardware.

RT Linux uses the flow of interrupts to give the real-time kernel a higher priority than the Linux kernel:

• When an interrupt arrives, it is first given to the real-time kernel, and not to the Linux kernel. But interrupts are stored to give them later to Linux when the real-time kernel is done.
• As first in row, the real-time kernel can run its real-time tasks driven by these interrupts.
• Only when the real-time kernel is not running anything, the interrupts which were stored are passed on to the Linux kernel.
• As second in row, Linux can schedule its own processes driven by these interrupt.

Hence, when a normal Linux program runs and a new interrupt arrives:

• it is first handled by an interrupt handler set by the real-time kernel;
• the code in the interrupt handler awakes a real-time task;
• immediately after the interrupt handler, the real-time scheduler is called ;
• the real-time scheduler observes that another real-time task is ready to run, so it puts the Linux kernel to sleep, and awakes the real-time task.

To let the real-time kernel and the Linux kernel coexist on a single machine, a special way of passing of the interrupts between real-time kernel and the Linux kernel is needed. Each flavor of RT Linux does this is in its own way. Xenomai uses an interrupt pipeline from the Adeos project. For more information, see also Life with Adeos.

RT Linux Tasks are not Linux Programs

• In real-time Linux we can make a real-time program by programming real-time threads. None-real-time programs in real-time Linux just use the conventional Linux threads.
• A real-time task can be executed in kernel space using a kernel module, but it can also be executed in user space using a normal C program.
• Running in user space instead of in kernel space gives a little extra overhead, but with the following advantages :
  • A bug in a kernel module can crash the whole system, however a bug in a user space program can only crash the program.
  • In a kernel model one can only use the real-time API and the limited kernel API, but in a real-time user space program the real-time API and the whole Linux API can be used! However when using the Linux API in user space, the program cannot be scheduled by the real-time scheduler (HARD real-time) but must be scheduled by the Linux scheduler (SOFT real-time). So calling a Linux API call from a real-time user space task will degrade its performance from HARD to SOFT real-time. After the call the task will return to the real-time scheduler.

Xenomai

• The Xenomai project was launched in August 2001.
• Xenomai is based on an abstract RTOS core, usable for building any kind of real-time interfaces, over a nucleus which exports a set of generic RTOS services. Any number of RTOS personalities called "skins" can then be built over the nucleus, providing their own specific interface to the applications, by using the services of a single generic core to implement it.
• The following skins on the generic core are implemented :
  • POSIX
  • pSOS+ 
  • VxWorks 
  • VRTX 
  • native: the Xenomai skin
  • uITRON
  • RTAI: only in kernel threads
• Xenomai allows to run real-time threads either strictly in kernel space, or within the address space of a Linux process. A real-time task in user space still has the benefit of memory protection, but is scheduled by Xenomai directly, and no longer by the Linux kernel. The worst case scheduling latency of such kind of task is always near to the hardware limits and predictable, since Xenomai is not bound to synchronizing with the Linux kernel activity in such a context, and can preempt any regular Linux activity with no delay. Hence, he preferred execution environment for Xenomai applications is user space context.
• But there might be a few cases where running some of the real-time code embodied into kernel modules is required, especially with legacy systems or very low-end platforms with under-performing MMU hardware. For this reason, Xenomai's native API provides the same set of real-time services in a seamless manner to applications, regardless of their execution space. Additionally, some applications may need real-time activities in both spaces to cooperate, therefore special care has been taken to allow the latter to work on the exact same set of API objects.
• In our terminology, the terms "thread" and "task" have the same meaning. When talking about a Xenomai task we refer to real-time task in user space, i.e., within the address space of a Linux process, not to be confused with regular Linux task/thread.

Timing in Xenomai

• The system clock in Xenomai is implemented by reading out the Time Stamp Clock (TSC) of the CPU. In case of multiprocessor machines, Xenomai assumes that the TSC of all processors are synchronised. In practice this is a problem with dual core processors, however in newer multiprocessor CPUs the manufacturers have solved this problem. For our exercises, we prevented this problem by building a uniprocessor Xenomai kernel which only uses one processor - even though we might have multiple in hardware.

Scheduling in Xenomai Linux

• In Xenomai Linux, the real-time scheduler is driven by timer interrupts from a special timer chip. This timer chip can be programmed to generate timing interrupts at certain moments.
• A timer chip itself is driven by a crystal oscillator. This oscillator ticks every baseperiod ns.
• Hence, the timer chip can only be programmed to generate an interrupt at the resolution of that baseperiod, e.g., to generate an interrupt after  x baseperiods, where x is an integer.
• For the timer chips in modern PC's the baseperiod is around 1 microsecond, and programming the timer costs around 3 microseconds.
• Each separate CPU has its own  timer chip which can be programmed either in periodic or one-shot mode independent of all the other CPUs.
• Xenomai uses this feature to either schedule a task either to run for once at a certain time (use the one-shot mode of timer chip) or to schedule the task repeatingly every period (use the period mode of the timer chip).
• Periodic scheduling of a task:
  • The period of the task is always a multiple of the timer's baseperiod. Thus taskperiod = X baseperiod  where X is an integer.
  • However for the convenience of the programmer, the Xenomai API lets the programmer specify the time in ns/jiffies (depending on the configuration of the system clock). The API itself recalculates this time into the number of  baseperiods.
• One-shot scheduling of a task:
  • The timer chip is reprogrammed at the end of each task, such that it expires exactly when the next task is scheduled to run. (This can be another task!)
• The advantage of periodic mode above one-shot mode, is that the timer only needs to be programmed once. In one-shot mode, the timer has to reprogrammed everytime when the task is done, costing around 3 microseconds each time.
• It depends on the hardware which timer chip is used, but most PCs nowadays have the following two timer chips:
  • The 8254 Programmable Interval Timer (PIT) chip. (This chip is also used by the normal Linux scheduler.) The resolution is based on frequency of the 8254 chip frequency (1,193,180 Hz), i.e., there is a tick of the 8254 chip every 838 nano seconds, that is around 1 microsecond.
    There is only one 8254 timer for the whole machine. This timer chip is therefore only used for single processor machines.
  • The local APIC timer. The resolution is based on the bus speed, but it is comparable with the 8254 chip.

  There is a local APIC timer for every CPU, and is therefore used in multiprocessor systems. Note that older uniprocessor systems did not have a local APIC timer.

• Some timing considerations :
  • The timer chip baseperiod  is around 1 microsecond, where as the Time Stamp Counter (TSC) of the CPU is incremented every clock cycle of the CPU, which is around 1 nanosecond for a 1 GHz CPU.  
  • Hence, the TSC frequency is  much higher than that of the timer chip (1000 times for a 1 GHz CPU). 
• Remember that the timer chip is meant to generate interrupts. The TSC cannot generate interrupts, but can only be used as a clock.

Note: In the latest Xenomai versions one can have a different periodic mode per skin, and have programs using different skins to run at the same time. To implement this, one does not use the periodic mode of the timer chip anymore. Instead all timings are done using one-shot programming of the timer chip. Periodic mode is emulated by a software driver which implements it by doing one-shot programming of the system timer periodically.

Last Updated: 29 August 2016 (Jozef Hooman)
Created by: Harco Kuppens
h.kuppens@cs.ru.nl