Real Time timing problem

Real Time timing problem

Post by Mark Wils » Tue, 11 Aug 1992 22:54:30



I am looking for help in solving a perfomance problem with UNIX V.4.
I am developing a product that must control a check Reader/Sorter that
processes 1000 documents per minute.  The problem is a very narrow, 37
millisecond, window for processing each check.  Communications is trhrough
a non-standard RS422 paralles interface.  The communications board and
UNIX driver were developed outside my group especially for this project.
The processor is an 80386 running at 20 MHZ.

The Reader/Sorter sends a data packet for a check and requires that a
sort decision (and other control data) for that check be returned within
37-40 milliseconds.  If the sort decision is not received within that
window, the operator must inervene and perform a recovery procedure.
Consequently, we run our application as a real-time process.  At the
moment, we are running at the highes real-time priority with an infinite
time slice.  We have also dropped down to init level 1 and killed cron
and sac for many of our tests.  Most of the time, we are making the sort
window with time to spare, but we have not been able to run for longer that
20 minutes (20,000 documents), without getting a late pocket decision.
20 minutes is the longest we have ever run, most runs crash after about
5 minutes (5,000 documents).

We have used the profiler, the system activity reporter and the clock()
function to try to isolate the problem, but none of these tools is quite
what we need.  The profiler won't show the times for each individual
document or the maximum time in each function.  (The averages that it
does show are very low.)  The system activity report tells us that very little
time is spent executing user code (typically 2% - 3%.  System time is 20%
to 30%.  The rest of the time the processor is idle).  The clock command
would be more helpful if it supported a resolution finer than 10
milliseconds.  It has shown however, that we can get a late pocket even
when we spend less than 10 milliseconds in the application.

The UNIX documentation says that, for an application such as ours, the
developers should know "typical time to preemption," "maximum time to
preemption," and "software switch latency," but the vendor cannot supply
this information.

While the critical code is running, we are seeing some disk activity that
we cannot explain.

We are running tow time-sharing processes concurrent with the real-time
process, but one of them is waiting for serial I/O and the other is the
parent process waiting for the death of a child.

The real time process has been locked into memory so no swapping should
be taking place.  This has been confirmed by the system activities report.

All help will be greatly appreciated.
--
--Mark

 
 
 

Real Time timing problem

Post by kenneth.s.cobl » Thu, 13 Aug 1992 05:05:23



>I am looking for help in solving a perfomance problem with UNIX V.4.
>I am developing a product that must control a check Reader/Sorter that
>processes 1000 documents per minute.  The problem is a very narrow, 37
>millisecond, window for processing each check.  Communications is trhrough
>a non-standard RS422 paralles interface.  The communications board and
>UNIX driver were developed outside my group especially for this project.

Without knowing the specifics of the project, I would ask the group
who designed your board and code to tell you the timing characteristics
of their code.  Does their driver block all lower level interrupts
for the critical code region ?  Does their driver perform the sort
decision or just sets a flag to say a packet has arrived ?

Quote:>The processor is an 80386 running at 20 MHZ.

>The Reader/Sorter sends a data packet for a check and requires that a
>sort decision (and other control data) for that check be returned within
>37-40 milliseconds.  If the sort decision is not received within that
>window, the operator must inervene and perform a recovery procedure.
>Consequently, we run our application as a real-time process.  At the
>moment, we are running at the highes real-time priority with an infinite
>time slice.  We have also dropped down to init level 1 and killed cron
>and sac for many of our tests.  Most of the time, we are making the sort
>window with time to spare, but we have not been able to run for longer that
>20 minutes (20,000 documents), without getting a late pocket decision.
>20 minutes is the longest we have ever run, most runs crash after about
>5 minutes (5,000 documents).

No matter what your settings are, System V R 4 (of stock USL code) is
not a REAL time operating system.  The system cannot guarantee any amount
of time to any particular process. It sounds like your application
must be real time application.  Strongly consider a Real time version of
Unix  (example: QNX).  

Assuming you are using C programming language, what compiler are you
using  ?   You might want to change your compiler to see if your
sort decision routine is any better.  Or, if the sort decision is
so timing critical, you might consider writing that function in assembly.

Quote:

>We have used the profiler, the system activity reporter and the clock()
>function to try to isolate the problem, but none of these tools is quite
>what we need.  The profiler won't show the times for each individual
>document or the maximum time in each function.  (The averages that it
>does show are very low.)  The system activity report tells us that very little

>While the critical code is running, we are seeing some disk activity that
>we cannot explain.

If your code generates data to be filed away on disk, the OS might
kick off the disk write routine and be in the middle of the critical
region.

Quote:

>We are running tow time-sharing processes concurrent with the real-time
>process, but one of them is waiting for serial I/O and the other is the
>parent process waiting for the death of a child.

>The real time process has been locked into memory so no swapping should
>be taking place.  This has been confirmed by the system activities report.

If I recollect, the no swapping flag just locks previously read in pages
into the system memory.  It does not force a read of all pages to
lock them in.  Your disk activity mentioned before could be due to
paging in the next section of code that's stored on disk.

Again, this comment is on the limited knowledge of the application and the
environment its executing on.


AT&T Bell Laboratories      attmail!cbnewsk!ksc or attmail!ihlpm!ken
2000 N. Naperville Rd.
Naperville, IL  60566

 
 
 

Real Time timing problem

Post by Adam » Sun, 16 Aug 1992 08:33:52


Your problem is a hard realtime problem, as processing a check must
be _finished_ within 37 ms.

        FORGET about SysVR[34] for real time applications.
        Get LynxOS, QNX for 386, or think  about other
        hardware and software.

best adams

 
 
 

Real Time timing problem

Post by Craig Spannri » Sun, 16 Aug 1992 08:55:07



>I am looking for help in solving a perfomance problem with UNIX V.4.

Here is some code that should help solve your problem

Quote:>----- cut here ------<

main()
{
        for(;;)
        {
                printf("Unix SVR4 is not a real time operating system!\n\007");
        }

Quote:}
>----- cut here ------<

Carefully watch the output from this program until you have found a solution.

--
====================================================================
 Six of one, 110 (base 2) of       | Craig Spannring

 ----------------------------------+--------------------------------

 
 
 

1. System time warping around real time problem - please help

I am using the exact same kernel (a lot of folks are). There
is no such jumping on my system.
Try this program:

#include <stdio.h>
#include <time.h>
int main() {
   time_t x,y;
   (void)time(&x);
   (void)time(&y);
   for(;;) {
       (void)time(&x);
       if(x < y)
           printf("Prev %ld New %ld\n", y, x);
       y = x;
   }
   return 0;
If this shows time jumping around you have one of either:

(1)     Bad timer channel 0 chip (PIT).
(2)     Some daemon trying to sync time with another system.
(3)     You are traveling too close to the speed of light.

Now, your script shows time in fractional seconds.

You can modify the program to do this:

#include <stdio.h>
#include <sys/time.h>
int main() {
   struct timeval tv;
   double x, y;
   (void)gettimeofday(&tv, NULL);
   x = (double) tv.tv_sec * 1e6;
   x += (double) tv.tv_usec;
   y = x;
   for(;;) {
       (void)gettimeofday(&tv, NULL);
       x = (double) tv.tv_sec * 1e6;
       x += (double) tv.tv_usec;
       if(x < y)
           printf("Prev %f New %f\n", y, x);
       y = x;
   }
   return 0;

There should be no jumping around -- and there isn't on
any system I've tested this on.

Hmmm, software should never crash. Even if the timers jump backwards
as you say, they should eventually time-out. If you have crashes, this
may point to other hardware problems as well.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

2. Jaton 67P Video Card

3. Serial port printing Solaris 1.1.1b

4. convert UT time in local time / local time in UT time

5. removing locale stuff from pre-installed Solaris

6. time time time how can I tell the time

7. Network debugging w/o reboot?

8. "Real-Time" Time Display in Console

9. X Server Time is REAL time of day???

10. Mixed Time-Share & real-Time scheduling

11. How to profile in REAL time (not process-virtual time)

12. real time less than user time?