Very Computer

Hi,

I have read lot of good things about Liunx 2.6 Kernel's real-time
scheduling features. Has anybody integrated the kernel into a system
with real-time interrupt requirements in the sub 100-microsecond
range?

Thanks in advance,

Sandeep
--
http://www.EventHelix.com/EventStudio
EventStudio 2.0 - Real-time and Embedded System Design CASE Tool

Quote:>Has anybody integrated the kernel into a system
> with real-time interrupt requirements in the sub 100-microsecond
> range?

What doe you mean by that ?

Linux can't _guarantee_  _any_ latency constrains. It's not supposed to
be a hard realtime system. Kernel 2.6 is supposed to perform much better
in _soft_ realtime systems than it's predecessors.

Do you want to generate a response to your hardware directly in the
interrupt ? Of course this will show much less _average_ latency than
doing it in an upper level of processing.

If you need hard realtime and/or very fast response-time I suppose
adding RTAI to the system is appropriate.

-Michael

I second to what Michael said. Linux is not meant to an RTOS. In ver
2.6, they have added couple of things like fully preemptible kernel,
timers, check points etc again response time cannot be definite as
demanded by hard real time requirements. For definite response time
one has to switch to RTOSs like RTAI, RTLinux etc.

nikhil

<aol>Me, too</aol>

Depending on your needs, it can also be useful to combine Linux with
additional processing hardware.  A second computer (not necessarily
very big, and not necessarily with an operating system: think PIC),
or an FPGA.  The latter is my interest, I build systems that respond
(guaranteed by design) in less than a microsecond.  Linux handles
the housekeeping, including network attachment.

    - Larry

Quote:>I second to what Michael said. Linux is not meant to an RTOS. In ver
>2.6, they have added couple of things like fully preemptible kernel,
>timers, check points etc again response time cannot be definite as
>demanded by hard real time requirements. For definite response time
>one has to switch to RTOSs like RTAI, RTLinux etc.

It depends on how hard your RT requirements are.  For example Linux 2.4 has no
problem reading an SPI input in under 150 microseconds on a 100mhz cris
processor even though I had the software as a user process making system
calls for each bit banging.  A call from one process to another --
A puts msg in B's pipe;  B services request and sends response back to A's
pipe -- takes less than 200us on a 100mhz cris processor.

Quote:> Depending on your needs, it can also be useful to combine Linux with
> additional processing hardware.  

I do this quite often (admittedly mostly with Windows systems, that are
even worse in realtime stuff).

You might want to take a look at the Ubicom processors. For low resource
requirements the 2K (a $7 160 MIPS 8 bit RISC PIC upgrade) is great.
Very little hardware needed (no external RAM or ROM) and you can link it
to the Linux system with TCP/IP/EThernet with just adding the magnetics,
as the Phy is included. For more sophisticated need the 3K (a $15 250
MIPS 32 Bit RISC) can be used. It needs external FLASH (but no RAM) and
due to it's "instruction level hardware multitasking" it can replace an
FPGA in many cases.

-Michael

Quote:> Linux can't _guarantee_  _any_ latency constrains. It's not supposed to
> be a hard realtime system. Kernel 2.6 is supposed to perform much better
> in _soft_ realtime systems than it's predecessors.

I have worked on a real-time system with periodic processing deadlines
in sub-2 ms range. We did not have any problems even though we used
regular Linux (no real-time flavors).

I was wondering if the 2.6 kernel can do even better than that.

Sandeep
--
http://www.EventHelix.com/EventStudio
EventStudio 2.0 - System Architecture Design CASE Tool

It should be able to (without patching even), as the preemptable patch
against 2.4.x should do. And that's improved and included in vanilla 2.6.x
http://www.linuxjournal.com/article.php?sid=6405%20

But if you have i686 (or AMD Athlon) hardware, there is a patch which adds
hard real-time using the APIC.

For 2.4.x with details:
http://linuxdevices.com/news/NS3235024671.html

And now 2.6.x as well:
http://lwn.net/Articles/74301/

(I have not applied that to any box, and/or tested myself though.)

HTH.

--
-Menno.

Quote:> I have worked on a real-time system with periodic processing deadlines
> in sub-2 ms range. We did not have any problems even though we used
> regular Linux (no real-time flavors).

"Hard real-time" is defined by how dramatic the effect of not meeting a
deadline is, not how tight theses deadlines are. E.g. Video streaming is
not hard real-time, though you need to display a frame each some 20 msec
, but dropping frames every some minutes is not problem at all. If your
power plant explodes because your software does not meet the deadline to
shut it down within an hour when the temperature rises slowly above some
limit this _is_ hard real-time.

I did some tests with Kernel 2.4 and the available low latency patches
(that I think are included in Kernel 2.6). I found that the soft
real-time behavior was increased greatly by the patches: much more low
time limit deadlines were met. But the hard real-time behavior was not
improved at all: there were still _some_ >100msec deadlines that were
missed.

The test was done using a high priority process with real-time
scheduling while heavy file copying on an IDE drive was done with a
normal process.

AFAIK the bad boy is the IDE driver. AFAIK, same is completely rewritten
for Kernel 2.6 and thus the real-time behavior is supposed to be
improved a lot.

Nonetheless there is no _guarantee_ that a deadline of > xx seconds is
met, and thus Linux is not a hard real-time system.

-Michael

[ Snip, good explanation of "hard real-time". ]

Quote:> I did some tests with Kernel 2.4 and the available low latency patches
> (that I think are included in Kernel 2.6).

Indeed they are. Here are some (experimental) for 2.6 though:
http://members.optusnet.com.au/ckolivas/kernel/

Quote:> I found that the soft real-time behavior was increased greatly by the
> patches: much more low time limit deadlines were met. But the hard
> real-time behavior was not improved at all: there were still _some_
> >100msec deadlines that were missed.

That probably depends on general load (and for real-time it shouldn't...)
But some monitoring device most likely runs off of ramdisk, and doesn't
have APM / ACPI power management, or any other kernel-task that may keep
it busy (for instance "kswapd" can also be disabled).

Quote:> The test was done using a high priority process with real-time
> scheduling while heavy file copying on an IDE drive was done with a
> normal process.

I have read about some test like that on a lo spec machine. I think some
serial connection started dropping frames while under load. Hence a "ping"
flood over null-modem SLIP connection might be a good test too.

Quote:> AFAIK the bad boy is the IDE driver. AFAIK, same is completely rewritten
> for Kernel 2.6

Well, i haven't realy been following the kernel list. However i don't see
it mensioned in the haloween file:
http://www.codemonkey.org.uk/docs/post-halloween-2.6.txt

Quote:> and thus the real-time behavior is supposed to be improved a lot.

Some article (on linuxdevices.com IIRC) sugested it is. But IIUC it's
the i/o scheduler that's much improved, rather then the device driver:
http://www.linuxjournal.com/article.php?sid=6931

Quote:> Nonetheless there is no _guarantee_ that a deadline of > xx seconds is
> met, and thus Linux is not a hard real-time system.

True.

But it can be patched with "rtirq" on intel.
Or run on top an executive like: RTLinux, RTAI

--
-Menno.

Quote:>> Nonetheless there is no _guarantee_ that a deadline of > xx seconds is
>> met, and thus Linux is not a hard real-time system.

>True.

>But it can be patched with "rtirq" on intel.
>Or run on top an executive like: RTLinux, RTAI

Wouldn't it be enough to simply remove the adaptive time-slicing from
the scheduler ?
---
42Bastian

Quote:> > AFAIK the bad boy is the IDE driver. AFAIK, same is completely rewritten
> > for Kernel 2.6

> Well, i haven't realy been following the kernel list. However i don't see
> it mensioned in the haloween file:
> http://www.codemonkey.org.uk/docs/post-halloween-2.6.txt

They say they started again from 2.4.18 IDE instead of using the already
existing complete rewrite. I can't say if that is good or not (I even
can't see why IDE is such a big issue anyway (I found that it was in
heavy discussions since the beginning) ).

In my tests I found that I had _much_ greater max latency when doing
file copying on IDE vs doing it on a Network drive.

Quote:

> Some article (on linuxdevices.com IIRC) sugested it is. But IIUC it's
> the i/o scheduler that's much improved, rather then the device driver:
> http://www.linuxjournal.com/article.php?sid=6931

I'll take a look at this.

Thanks,
-Michael

These should not happen! (or do you mean usec?)

Did you really turn on DMA on the HDs?

From:
http://www.gardena.net/benno/linux/audio/

"highly tuned harddisk with DMA , 32bit transfer  , multicount , unmask-irq
activated
( hdparm -m 8 -d 1 -u 1 -c 1 /dev/your-ide-hd ) "

Quote:

> The test was done using a high priority process with real-time
> scheduling while heavy file copying on an IDE drive was done with a
> normal process.

One of the tests in the above suite is file copying...
Results should look like this:
  http://www.gardena.net/benno/linux/audio/2.2.10-p133-3x128/3x128.html

Quote:

> AFAIK the bad boy is the IDE driver. AFAIK, same is completely rewritten
> for Kernel 2.6 and thus the real-time behavior is supposed to be
> improved a lot.

IDE driver have not been a problem for some time - worse is CapsLock...
Some USB devices...

Quote:> Nonetheless there is no _guarantee_ that a deadline of > xx seconds is
> met, and thus Linux is not a hard real-time system.

No OS can guarantee that your SYSTEM is hard real-time. At least your
system needs to have an UPS... :-)

/RogerL

--
Roger Larsson
Skellefte?
Sweden

On Mon, 26 Apr 2004 20:14:02 GMT, Roger Larsson

While it is certainly possible to determine the absolute worst case
response time for a simple 1-2 KiB RT kernel running on some simple 8
bit processor without any cache. But how do you determine the worst
case response time for anything as complex as the Linux or WinNT
kernel running on some modern x86 processor ?

First of all, how do you determine the worst path on operating systems
that are that complex (even if designed to be hard real time) as WinNT
or linux ? And how about the effects of the cache ?

In a system with DMA transfers and/or multiple processors in an SMP
system, it is quite possible, that the cache is invalidated after each
byte read from memory, forcing the read from actual physical memory
for each byte. One way of measuring this would be to turn off the
cache during measurements.  

But on the other hand, can we be sure that loading the 32 byte cache
row (on Pentiums) during each byte access is faster than running the
system without any cache during the tests.

So what is the most powerful processor+OS kernel combination, which is
still truly hard realtime, i.e. the absolute maximum response time can
actually be determined ?

Paul

Quote:> One of the tests in the above suite is file copying...
> Results should look like this:
>   http://www.gardena.net/benno/linux/audio/2.2.10-p133-3x128/3x128.html

On the first sight: This is a test on a latency distribution. It does
not seem to show extreme seldom values. I got the 100 mSec latencies
only once in a million or less.

Quote:

> No OS can guarantee that your SYSTEM is hard real-time. At least your
> system needs to have an UPS... :-)

If the user software does not comply, of course the OS can't help. But
if the OS is not hard real-time (like Linux) the user software can't
create a hard real time system with same.

When adding RTAI (or using vxWorks) and appropriate user software you
can do a (deterministic) hard real time system (of course providing much
less features <on the real-time part> then Linux provides. )

-Michael