Linux 2.6 Kernel in Real-time Environment

Linux 2.6 Kernel in Real-time Environment

Post by EventHelix.c » Mon, 19 Apr 2004 10:26:46



Hi,

I have read lot of good things about Liunx 2.6 Kernel's real-time
scheduling features. Has anybody integrated the kernel into a system
with real-time interrupt requirements in the sub 100-microsecond
range?

Thanks in advance,

Sandeep
--
http://www.EventHelix.com/EventStudio
EventStudio 2.0 - Real-time and Embedded System Design CASE Tool

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Michael Schnel » Tue, 20 Apr 2004 15:59:30


Quote:>Has anybody integrated the kernel into a system
> with real-time interrupt requirements in the sub 100-microsecond
> range?

What doe you mean by that ?

Linux can't _guarantee_  _any_ latency constrains. It's not supposed to
be a hard realtime system. Kernel 2.6 is supposed to perform much better
in _soft_ realtime systems than it's predecessors.

Do you want to generate a response to your hardware directly in the
interrupt ? Of course this will show much less _average_ latency than
doing it in an upper level of processing.

If you need hard realtime and/or very fast response-time I suppose
adding RTAI to the system is appropriate.

-Michael

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by nikhil bharg » Tue, 20 Apr 2004 20:55:42



> >Has anybody integrated the kernel into a system
> > with real-time interrupt requirements in the sub 100-microsecond
> > range?

> What doe you mean by that ?

> Linux can't _guarantee_  _any_ latency constrains. It's not supposed to
> be a hard realtime system. Kernel 2.6 is supposed to perform much better
> in _soft_ realtime systems than it's predecessors.

> Do you want to generate a response to your hardware directly in the
> interrupt ? Of course this will show much less _average_ latency than
> doing it in an upper level of processing.

> If you need hard realtime and/or very fast response-time I suppose
> adding RTAI to the system is appropriate.

> -Michael

I second to what Michael said. Linux is not meant to an RTOS. In ver
2.6, they have added couple of things like fully preemptible kernel,
timers, check points etc again response time cannot be definite as
demanded by hard real time requirements. For definite response time
one has to switch to RTOSs like RTAI, RTLinux etc.

nikhil

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Larry Doolittl » Tue, 20 Apr 2004 21:43:10




>> > Has anybody integrated the kernel into a system
>> > with real-time interrupt requirements in the sub 100-microsecond
>> > range?

>> Linux can't _guarantee_  _any_ latency constrains. It's not supposed to
>> be a hard realtime system. Kernel 2.6 is supposed to perform much better
>> in _soft_ realtime systems than it's predecessors.

>> If you need hard realtime and/or very fast response-time I suppose
>> adding RTAI to the system is appropriate.

> I second to what Michael said. Linux is not meant to an RTOS. In ver
> 2.6, they have added couple of things like fully preemptible kernel,
> timers, check points etc again response time cannot be definite as
> demanded by hard real time requirements. For definite response time
> one has to switch to RTOSs like RTAI, RTLinux etc.

<aol>Me, too</aol>

Depending on your needs, it can also be useful to combine Linux with
additional processing hardware.  A second computer (not necessarily
very big, and not necessarily with an operating system: think PIC),
or an FPGA.  The latter is my interest, I build systems that respond
(guaranteed by design) in less than a microsecond.  Linux handles
the housekeeping, including network attachment.

    - Larry

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by TCS » Tue, 20 Apr 2004 22:58:03



Quote:>I second to what Michael said. Linux is not meant to an RTOS. In ver
>2.6, they have added couple of things like fully preemptible kernel,
>timers, check points etc again response time cannot be definite as
>demanded by hard real time requirements. For definite response time
>one has to switch to RTOSs like RTAI, RTLinux etc.

It depends on how hard your RT requirements are.  For example Linux 2.4 has no
problem reading an SPI input in under 150 microseconds on a 100mhz cris
processor even though I had the software as a user process making system
calls for each bit banging.  A call from one process to another --
A puts msg in B's pipe;  B services request and sends response back to A's
pipe -- takes less than 200us on a 100mhz cris processor.
 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Michael Schnel » Tue, 20 Apr 2004 23:23:10


Quote:> Depending on your needs, it can also be useful to combine Linux with
> additional processing hardware.  

I do this quite often (admittedly mostly with Windows systems, that are
even worse in realtime stuff).

You might want to take a look at the Ubicom processors. For low resource
requirements the 2K (a $7 160 MIPS 8 bit RISC PIC upgrade) is great.
Very little hardware needed (no external RAM or ROM) and you can link it
to the Linux system with TCP/IP/EThernet with just adding the magnetics,
as the Phy is included. For more sophisticated need the 3K (a $15 250
MIPS 32 Bit RISC) can be used. It needs external FLASH (but no RAM) and
due to it's "instruction level hardware multitasking" it can replace an
FPGA in many cases.

-Michael

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by EventHelix.c » Thu, 22 Apr 2004 21:32:43


Quote:> Linux can't _guarantee_  _any_ latency constrains. It's not supposed to
> be a hard realtime system. Kernel 2.6 is supposed to perform much better
> in _soft_ realtime systems than it's predecessors.

I have worked on a real-time system with periodic processing deadlines
in sub-2 ms range. We did not have any problems even though we used
regular Linux (no real-time flavors).

I was wondering if the 2.6 kernel can do even better than that.

Sandeep
--
http://www.EventHelix.com/EventStudio
EventStudio 2.0 - System Architecture Design CASE Tool

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Menno Duursm » Fri, 23 Apr 2004 07:49:05



>> Linux can't _guarantee_  _any_ latency constrains. It's not supposed to
>> be a hard realtime system. Kernel 2.6 is supposed to perform much better
>> in _soft_ realtime systems than it's predecessors.

> I have worked on a real-time system with periodic processing deadlines
> in sub-2 ms range. We did not have any problems even though we used
> regular Linux (no real-time flavors).

> I was wondering if the 2.6 kernel can do even better than that.

It should be able to (without patching even), as the preemptable patch
against 2.4.x should do. And that's improved and included in vanilla 2.6.x
http://www.linuxjournal.com/article.php?sid=6405%20

But if you have i686 (or AMD Athlon) hardware, there is a patch which adds
hard real-time using the APIC.

For 2.4.x with details:
http://linuxdevices.com/news/NS3235024671.html

And now 2.6.x as well:
http://lwn.net/Articles/74301/

(I have not applied that to any box, and/or tested myself though.)

HTH.

--
-Menno.

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Michael Schnel » Fri, 23 Apr 2004 17:19:04


Quote:> I have worked on a real-time system with periodic processing deadlines
> in sub-2 ms range. We did not have any problems even though we used
> regular Linux (no real-time flavors).

"Hard real-time" is defined by how dramatic the effect of not meeting a
deadline is, not how tight theses deadlines are. E.g. Video streaming is
not hard real-time, though you need to display a frame each some 20 msec
, but dropping frames every some minutes is not problem at all. If your
power plant explodes because your software does not meet the deadline to
shut it down within an hour when the temperature rises slowly above some
limit this _is_ hard real-time.

I did some tests with Kernel 2.4 and the available low latency patches
(that I think are included in Kernel 2.6). I found that the soft
real-time behavior was increased greatly by the patches: much more low
time limit deadlines were met. But the hard real-time behavior was not
improved at all: there were still _some_ >100msec deadlines that were
missed.

The test was done using a high priority process with real-time
scheduling while heavy file copying on an IDE drive was done with a
normal process.

AFAIK the bad boy is the IDE driver. AFAIK, same is completely rewritten
for Kernel 2.6 and thus the real-time behavior is supposed to be
improved a lot.

Nonetheless there is no _guarantee_ that a deadline of > xx seconds is
met, and thus Linux is not a hard real-time system.

-Michael

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Menno Duursm » Sat, 24 Apr 2004 07:35:55



>> I have worked on a real-time system with periodic processing deadlines
>> in sub-2 ms range. We did not have any problems even though we used
>> regular Linux (no real-time flavors).

[ Snip, good explanation of "hard real-time". ]

Quote:> I did some tests with Kernel 2.4 and the available low latency patches
> (that I think are included in Kernel 2.6).

Indeed they are. Here are some (experimental) for 2.6 though:
http://members.optusnet.com.au/ckolivas/kernel/

Quote:> I found that the soft real-time behavior was increased greatly by the
> patches: much more low time limit deadlines were met. But the hard
> real-time behavior was not improved at all: there were still _some_
> >100msec deadlines that were missed.

That probably depends on general load (and for real-time it shouldn't...)
But some monitoring device most likely runs off of ramdisk, and doesn't
have APM / ACPI power management, or any other kernel-task that may keep
it busy (for instance "kswapd" can also be disabled).

Quote:> The test was done using a high priority process with real-time
> scheduling while heavy file copying on an IDE drive was done with a
> normal process.

I have read about some test like that on a lo spec machine. I think some
serial connection started dropping frames while under load. Hence a "ping"
flood over null-modem SLIP connection might be a good test too.

Quote:> AFAIK the bad boy is the IDE driver. AFAIK, same is completely rewritten
> for Kernel 2.6

Well, i haven't realy been following the kernel list. However i don't see
it mensioned in the haloween file:
http://www.codemonkey.org.uk/docs/post-halloween-2.6.txt

Quote:> and thus the real-time behavior is supposed to be improved a lot.

Some article (on linuxdevices.com IIRC) sugested it is. But IIUC it's
the i/o scheduler that's much improved, rather then the device driver:
http://www.linuxjournal.com/article.php?sid=6931

Quote:> Nonetheless there is no _guarantee_ that a deadline of > xx seconds is
> met, and thus Linux is not a hard real-time system.

True.

But it can be patched with "rtirq" on intel.
Or run on top an executive like: RTLinux, RTAI

--
-Menno.

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by 42Bastian Schi » Sat, 24 Apr 2004 16:28:56


Quote:>> Nonetheless there is no _guarantee_ that a deadline of > xx seconds is
>> met, and thus Linux is not a hard real-time system.

>True.

>But it can be patched with "rtirq" on intel.
>Or run on top an executive like: RTLinux, RTAI

Wouldn't it be enough to simply remove the adaptive time-slicing from
the scheduler ?
---
42Bastian


 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Michael Schnel » Sat, 24 Apr 2004 17:58:16


Quote:> > AFAIK the bad boy is the IDE driver. AFAIK, same is completely rewritten
> > for Kernel 2.6

> Well, i haven't realy been following the kernel list. However i don't see
> it mensioned in the haloween file:
> http://www.codemonkey.org.uk/docs/post-halloween-2.6.txt

They say they started again from 2.4.18 IDE instead of using the already
existing complete rewrite. I can't say if that is good or not (I even
can't see why IDE is such a big issue anyway (I found that it was in
heavy discussions since the beginning) ).

In my tests I found that I had _much_ greater max latency when doing
file copying on IDE vs doing it on a Network drive.

Quote:

> Some article (on linuxdevices.com IIRC) sugested it is. But IIUC it's
> the i/o scheduler that's much improved, rather then the device driver:
> http://www.linuxjournal.com/article.php?sid=6931

I'll take a look at this.

Thanks,
-Michael

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Roger Larsso » Wed, 28 Apr 2004 05:14:02



>> I have worked on a real-time system with periodic processing deadlines
>> in sub-2 ms range. We did not have any problems even though we used
>> regular Linux (no real-time flavors).

> "Hard real-time" is defined by how dramatic the effect of not meeting a
> deadline is, not how tight theses deadlines are. E.g. Video streaming is
> not hard real-time, though you need to display a frame each some 20 msec
> , but dropping frames every some minutes is not problem at all. If your
> power plant explodes because your software does not meet the deadline to
> shut it down within an hour when the temperature rises slowly above some
> limit this _is_ hard real-time.

> I did some tests with Kernel 2.4 and the available low latency patches
> (that I think are included in Kernel 2.6). I found that the soft
> real-time behavior was increased greatly by the patches: much more low
> time limit deadlines were met. But the hard real-time behavior was not
> improved at all: there were still _some_ >100msec deadlines that were
> missed.

These should not happen! (or do you mean usec?)

Did you really turn on DMA on the HDs?

From:
http://www.gardena.net/benno/linux/audio/

"highly tuned harddisk with DMA , 32bit transfer  , multicount , unmask-irq
activated
( hdparm -m 8 -d 1 -u 1 -c 1 /dev/your-ide-hd ) "

Quote:

> The test was done using a high priority process with real-time
> scheduling while heavy file copying on an IDE drive was done with a
> normal process.

One of the tests in the above suite is file copying...
Results should look like this:
  http://www.gardena.net/benno/linux/audio/2.2.10-p133-3x128/3x128.html

Quote:

> AFAIK the bad boy is the IDE driver. AFAIK, same is completely rewritten
> for Kernel 2.6 and thus the real-time behavior is supposed to be
> improved a lot.

IDE driver have not been a problem for some time - worse is CapsLock...
Some USB devices...

Quote:> Nonetheless there is no _guarantee_ that a deadline of > xx seconds is
> met, and thus Linux is not a hard real-time system.

No OS can guarantee that your SYSTEM is hard real-time. At least your
system needs to have an UPS... :-)

/RogerL

--
Roger Larsson
Skellefte?
Sweden

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Paul Keinane » Wed, 28 Apr 2004 06:50:08


On Mon, 26 Apr 2004 20:14:02 GMT, Roger Larsson


>> Nonetheless there is no _guarantee_ that a deadline of > xx seconds is
>> met, and thus Linux is not a hard real-time system.

>No OS can guarantee that your SYSTEM is hard real-time. At least your
>system needs to have an UPS... :-)

While it is certainly possible to determine the absolute worst case
response time for a simple 1-2 KiB RT kernel running on some simple 8
bit processor without any cache. But how do you determine the worst
case response time for anything as complex as the Linux or WinNT
kernel running on some modern x86 processor ?

First of all, how do you determine the worst path on operating systems
that are that complex (even if designed to be hard real time) as WinNT
or linux ? And how about the effects of the cache ?

In a system with DMA transfers and/or multiple processors in an SMP
system, it is quite possible, that the cache is invalidated after each
byte read from memory, forcing the read from actual physical memory
for each byte. One way of measuring this would be to turn off the
cache during measurements.  

But on the other hand, can we be sure that loading the 32 byte cache
row (on Pentiums) during each byte access is faster than running the
system without any cache during the tests.

So what is the most powerful processor+OS kernel combination, which is
still truly hard realtime, i.e. the absolute maximum response time can
actually be determined ?

Paul

 
 
 

Linux 2.6 Kernel in Real-time Environment

Post by Michael Schnel » Wed, 28 Apr 2004 21:05:57


Quote:> One of the tests in the above suite is file copying...
> Results should look like this:
>   http://www.gardena.net/benno/linux/audio/2.2.10-p133-3x128/3x128.html

On the first sight: This is a test on a latency distribution. It does
not seem to show extreme seldom values. I got the 100 mSec latencies
only once in a million or less.

Quote:

> No OS can guarantee that your SYSTEM is hard real-time. At least your
> system needs to have an UPS... :-)

If the user software does not comply, of course the OS can't help. But
if the OS is not hard real-time (like Linux) the user software can't
create a hard real time system with same.

When adding RTAI (or using vxWorks) and appropriate user software you
can do a (deterministic) hard real time system (of course providing much
less features <on the real-time part> then Linux provides. )

-Michael

 
 
 

1. Needed - experienced brain in - MPEG, Windows Media 9, codec in real-time embedded system environment

We have a client - N.Calif. whom desperately needs help.

Name your salary deal. (within reason)

might benefit a member of the group.

a.. We seek your experience in the following areas:

a.. MPEG-2 transport stream
a.. MPEG2/4, H.264, and Windows Media 9 Series Code
a.. Real-time embedded systems (VxWorks, or real time Linux), and network
programming.
a.. C/C++
a.. Video streaming using RTP/RTSP
a.. Built set top box player or AV box dealing with video and audio codec

The following skills are desired:

a.. Hands on networking, system integration, script programming
a.. Has network installation experience
a.. Able to debug network, and can use network debugging tool e.g. sniffer
a.. Has system testing experience
a.. Experience of product development cycles including product release, and
field support, etc.

--

Ref UT002

John D Allen. CEO & President.
Leveridge Systems INC.
v1 (909) 699 3751
f1 (800) 783 4350

2. Sound

3. Real-time Linux kernel available?

4. wmaker error: defaults DB:could not access menu

5. Linux real-time kernel?

6. Internal Zip Drive compatibility

7. Real-Time Linux Kernel Release

8. linux software for project planning

9. Errors compiling Wingz 1.4 Add-ins (Linux 1.3.99, GCC 2.6)

10. Real time dispatch table on Solaris 2.6

11. Using POSIX real-time in kernel mode

12. real-time file monitoring at the kernel level

13. Real-time UNIX Kernel Engineer at Silicon Graphics (SGI), Mountain View, CA