Achieving high-resolution timeouts in Solaris (w/ Source Code)

Achieving high-resolution timeouts in Solaris (w/ Source Code)

Post by The Sh » Sun, 18 Jul 1999 04:00:00



Greetings,

  I've been trying to make a 'constant bit-rate streamer' on
Solaris.  The problem I've run into is the latency, or overhead
with the real-time (RT) priority class' timing options.  The
goal here is to control a writing loop to 500000 nanoseconds.
That is, I wish to write a handful of bits every 500000 nanoseconds.

I've used two methods to try to get high-resolution pauses
in the writing thread: with nanosleep (from the real time
library librt on Solaris 2.7 and on 2.6, from libposix4);  And
by using pthread_cond_timedwait.

In both cases the observed wait or sleep time far exceeds
the requested time.

When I try to pause a thread for 500000 ns, what I observe is
typically a timeout of 19 times that amount ~ 9900000 ns.

My first question is: what kind of resolution can
you achieve with Solaris?

And my other question is, if you can achieve high
resolution pauses with Solaris (on order of 500000 ns),
how do you do it?

From here, I'll share with the curious what I do know
about RT on Solaris and a test program I put together
to measure high resolution timeouts.

The machines I've used to test this are an Ultra 2 (200mhz)
single processor running 2.7, a sparc5 85mhz also on 2.7,
and an E250 dual (300mhz) processor running 2.6.  The platforms
 yield fairly equivelent results and the sparc 5 was only
neglibly slower then the ultra class machines.  (C compiler
 is SparcWorks 5.0).

I picked 'nanosleep' to use while I investigate the issue,
the use of 'mutex_lock' with 'cond_timedwait' yielded results
with only slightly higher lags.

Here's the outline of a program I put together, "nanotest",
which measures the lag of 'nanosleep':

        1. Put process in class RT, priority 54.
        2. FOR i := 1 to 'iterations'
           BEGIN
        3.     gettimeofday(pre)  // current time clock.
        4.     nanosleep(timeout);
        5.     gettimeofday(post);  // post time clock.

        6.     Diff += (post - pre)  // Diff collects the observed lag.
           END

        7  PRINT Diff / iterations;   // Echo the average observed lag.

Now, for some results.  These tests were run on Sparc 5 (85mhz) running
2.7.

#1. This measured a 500000 ns pause done 100 times:
---
% nanotest 500000
** nanotest: Sched class: RT, CID: 3
** nanotest: maxpri: 59 pri: 54 quantum: 100000000 nanosecs
Will nap for 500000 nanosecs.
Iterations = 100
Target wait = 500000.  Ave wait = 9968000 (ns)
---

The observed average lag is 9968000. 19 times greater than
the target!  Is this the best Solaris can do?

#2. For a reference this tests the lag observed without calling
nanosleep at all -- this measures then the time to call
'gettimeofday' twice consecutively -- which is only 2000 ns.

---
% nanotest -- -1
** nanotest: Sched class: RT, CID: 3
** nanotest: maxpri: 59 pri: 54 quantum: 100000000 nanosecs
Will nap for -1 nanosecs.
Iterations = 100
Target wait = -1.  Ave wait = 2000 (ns)
---

#3.  test the lag for a 1 nanosecond sleep.

---
% nanotest 1
** nanotest: Sched class: RT, CID: 3
** nanotest: maxpri: 59 pri: 54 quantum: 100000000 nanosecs
Will nap for 1 nanosecs.
Iterations = 100
Target wait = 1.  Ave wait = 9978000 (ns)
---

Notice the 'Ave wait' is nearly the same as the test
in #1 (sleep for 500000 ns).

Any pointers or advice would be appreciated.  And also
if you know that Solaris isn't capable of resolutions
on the order of 500 milliseconds or less, I'd like to hear that
too!

Well, the source code for 'nanotest' is included for
anyone to play with.  It compiles on 2.6/7.

On 2.6 compile as:  cc -o nanotest nanotest.c -lposix4
On 2.7 compile as:  cc -o nanotest nanotest.c -lrt

---
James Schumacher (The Shoe)
j...@sprintlabs.com
zapat...@netcom.com
---
/*
 *  nanotest.  Simple routine for measuring nanosleep.
 *             Copyright 1999, by James Schumacher.
 *                             email:  zapat...@netcom.com
 *                                     j...@sprintlabs.com
 *
 *   Author grants use, but publish your modifications
 *   to comp.realtime.
 */
#include <stdio.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/priocntl.h>
#include <sys/rtpriocntl.h>
#include <sys/tspriocntl.h>
#include <sys/time.h>
#include <sys/uio.h>
#include <stdlib.h>

#define PROGNAME "nanotest"

#define DEFITS 100

#define NANOS 1000000000
#define MILLS 1000000

#define USEC_DIFF(res,a,b)              \
        {       \
               long tmp_usecs; \
               res = (a.tv_sec - b.tv_sec) * MILLS; \
               tmp_usecs = a.tv_usec - b.tv_usec; \
               if (tmp_usecs < 0) {  \
                   tmp_usecs -= MILLS; \
                   res += MILLS; \
               } \
               res += tmp_usecs; \
        }

static void setRTPrio(int class,int prio,int quantum_ns);
static void printPrio(char *lwp_name);

static void usage()
{
    fprintf(stderr,"Usage: " PROGNAME " [-i iterations] nanosecs\n");
    exit(1);

}

int main(int argc, char *argv[])
{
    struct timeval currentTime, postTime;
    struct timespec nanopause;
    long long nanoseconds, ndiff;
    unsigned long long accumedWait;
    int i, its;
    extern char *optarg;
    extern int optind;
    int c;

    its = DEFITS;
    while ((c = getopt(argc,argv,"i:o:")) != EOF)
      switch (c) {
      case 'i':
          its = atoi(optarg); break;
      default:
          usage();
          break;
      }

    if (optind >= argc)
        usage();

    nanoseconds = atoll(argv[optind]);
    setRTPrio(P_LWPID,54,0);
    printPrio(PROGNAME);

    printf("Will nap for %lld nanosecs.\n",nanoseconds);
    nanopause.tv_sec = nanoseconds / NANOS;
    nanopause.tv_nsec = nanoseconds % NANOS;

    accumedWait = 0;
    printf("Iterations = %d\n",its);

    for (i = 0; i < its; i++) {

        gettimeofday(&currentTime,0);

        if (nanoseconds >= 0)
           nanosleep(&nanopause,0);

        gettimeofday(&postTime,0);

        USEC_DIFF(ndiff,postTime,currentTime);

        accumedWait += ndiff;

    }

    printf("Target wait = %lld.  Ave wait = %llu (ns)\n",
           nanoseconds,(accumedWait / i) * 1000);

    exit(0);

}

static void setRTPrio(int class, int priority, int quantum_ns)
{
    pcinfo_t            pcinfo;
    pcparms_t           pcparms;
    rtinfo_t           *rtinfop;
    rtparms_t          *rtparmsp;

    memset(&pcinfo, 0, sizeof(pcinfo));
    strcpy(pcinfo.pc_clname, "RT");

    if (priocntl(0, 0, PC_GETCID, (caddr_t) & pcinfo) < 0) {
        perror("flowutils: Couldn't get realtime class");
        exit(3);
    }

    if (priority < 0) {
        fprintf(stderr,"** Invalid priority %d, using 0\n", priority);
        priority = 0;
    }

    rtinfop = (rtinfo_t *) pcinfo.pc_clinfo;
    if (priority > rtinfop->rt_maxpri) {
        fprintf(stderr,"Priority out-of-range: max = %d\n", rtinfop->rt_maxpri);
        priority = rtinfop->rt_maxpri;
    }

    pcparms.pc_cid = pcinfo.pc_cid;
    rtparmsp = (rtparms_t *) pcparms.pc_clparms;
    if (quantum_ns <= 0) {
        rtparmsp->rt_tqsecs = RT_TQDEF;
        rtparmsp->rt_tqnsecs = RT_TQDEF;
    } else {
        rtparmsp->rt_tqsecs = quantum_ns / NANOS;
        rtparmsp->rt_tqnsecs = quantum_ns % NANOS;
    }
    rtparmsp->rt_pri = priority;

    if (priocntl(class, P_MYID, PC_SETPARMS, (caddr_t) & pcparms) != 0) {
        perror("Couldn't set real-time priority: ");
    }

}

static void printPrio(char *lwp_name)
{
    pcinfo_t            pcinfo;
    pcparms_t           pcparms;
    rtinfo_t           *rtinfop;
    rtparms_t          *rtparmsp;
    tsinfo_t           *tsinfop;
    tsparms_t          *tsparmsp;

    memset(&pcparms, 0, sizeof(pcparms));
    pcparms.pc_cid = PC_CLNULL;
    if (priocntl(P_LWPID, P_MYID, PC_GETPARMS, (caddr_t) & pcparms) < 0) {
        fprintf(stderr,"%s: Couldn't get priority: %s\n", lwp_name,strerror(errno));
        return;
    }

    memset(&pcinfo, 0, sizeof(pcinfo));
    pcinfo.pc_cid = pcparms.pc_cid;
    if (priocntl(0, 0, PC_GETCLINFO, (caddr_t) & pcinfo) < 0) {
        fprintf(stderr,"%s: Couldn't get pcinfo: %s\n", lwp_name,strerror(errno));
        return;
    }

    fprintf(stderr,"** %s: Sched class: %s, CID: %d\n", lwp_name,
          pcinfo.pc_clname, pcparms.pc_cid);

    if (strcmp(pcinfo.pc_clname, "RT") == 0) {
        rtinfop = (rtinfo_t *) pcinfo.pc_clinfo;
        rtparmsp = (rtparms_t *) pcparms.pc_clparms;
        fprintf(stderr, "** %s: maxpri: %d pri: %d quantum: %d nanosecs\n", lwp_name,
              rtinfop->rt_maxpri, rtparmsp->rt_pri,
              rtparmsp->rt_tqsecs * NANOS + rtparmsp->rt_tqnsecs);
    }
    else if (strcmp(pcinfo.pc_clname, "TS") == 0) {
        tsinfop = (tsinfo_t *) pcinfo.pc_clinfo;
        tsparmsp = (tsparms_t *) pcparms.pc_clparms;
        fprintf(stderr, "%s: maxupri: %d uprilim: %d upri: %d\n", lwp_name,
              tsinfop->ts_maxupri, tsparmsp->ts_uprilim, tsparmsp->ts_upri);
    } else {
        fprintf(stderr, "%s: %s unknown class\n", lwp_name, pcinfo.pc_clname);
    }

}

 
 
 

Achieving high-resolution timeouts in Solaris (w/ Source Code)

Post by Christopher L Ta » Tue, 20 Jul 1999 04:00:00


:
:   I've been trying to make a 'constant bit-rate streamer' on
: Solaris.  The problem I've run into is the latency, or overhead
: with the real-time (RT) priority class' timing options.  The
: goal here is to control a writing loop to 500000 nanoseconds.
: That is, I wish to write a handful of bits every 500000 nanoseconds.

Check my math -- this is every 500 microseconds, right?  That is,
it's every half-millisecond?  If this is right, then you'll never
be able to see anything like this resolution under Solaris without
significant kernel tuning, and possibly not even then.

: I've used two methods to try to get high-resolution pauses
: in the writing thread: with nanosleep (from the real time
: library librt on Solaris 2.7 and on 2.6, from libposix4);  And
: by using pthread_cond_timedwait.

nanosleep(), deep in its implementation, winds up being a CV wait,
so I'll be *very* surprised if you ever see a significant difference
between these two approaches.

: When I try to pause a thread for 500000 ns, what I observe is
: typically a timeout of 19 times that amount ~ 9900000 ns.

That's 9900 microseconds, or approximately ten milliseconds.  Note
that the fundamental clock frequency under Solaris is 100 Hz, or
ten milliseconds per tick.  This is not a coincidence.

: My first question is: what kind of resolution can
: you achieve with Solaris?

Because *everything* under Solaris other than hardware interrupts is
tied to the fundamental 100 Hz clock, you can only get 10 millisecond
resolution on any kind of non-interrupt-level processing.  This includes
thread wakeup.

: And my other question is, if you can achieve high
: resolution pauses with Solaris (on order of 500000 ns),
: how do you do it?

You can't.  :-(  You *might* be able to tune the kernel to run the main
system clock at a different rate, say 1 ms per tick, but that's as close
as you can come to your desired resolution.  You will always have a
latency of whatever the clock tick is; by default it's 10 milliseconds.

: The machines I've used to test this are an Ultra 2 (200mhz)
: single processor running 2.7, a sparc5 85mhz also on 2.7,
: and an E250 dual (300mhz) processor running 2.6.  The platforms
:  yield fairly equivelent results and the sparc 5 was only
: neglibly slower then the ultra class machines.  (C compiler
:  is SparcWorks 5.0).

This is expected; the 100 Hz clock is quite slow, and doesn't induce any
appreciable load even on old machines like Sparc 5s.

: Any pointers or advice would be appreciated.  And also
: if you know that Solaris isn't capable of resolutions
: on the order of 500 milliseconds or less, I'd like to hear that
: too!

That's the answer, unfortunately.  I'm interested to hear, though, why
you need half-millisecond precision.  That's an extremely tight requirement
for general purpose OS's (i.e. non-RTOSs).  Unix has traditionally had a
10 ms fundamental clock, and most modern implementations including Linux
still use that even though the hardware is vastly faster today than it was
when the 10 ms clock was first chosen.  The best soft RT support I know of
in a general-purpose OS today is in BeOS, where you get something like
250 microsecond precision on real-time thread wakeup (generally; with a
bit of attention to the machine's running state you can keep that down to
around 150 microseconds in the best case).  BeOS does this by not using
a fixed underlying wakeup clock at all; rather, it schedules events
based on the needs of the current threads.  To my knowledge there are
no Unix implementations that don't use a fixed clock, with its associated
latencies on events like process or thread wakeup.

[ObDisclaimer:  I work for Be, Inc.  :-) ]

--



 
 
 

Achieving high-resolution timeouts in Solaris (w/ Source Code)

Post by Alan Stang » Tue, 20 Jul 1999 04:00:00


Hello,

From the Solaris FAQ

7.5) How can I have a clock resolution better than 10ms?

Starting with Solaris 2.6, this can be achieved with the following entry
in /etc/system:

        set hires_tick = 1

This will set the system hz value to 1000.

In principle, you can also set "hz" directly, but that is not supported
nor recommended:

        * Get 0.1 ms clock resolution/timer granularity
        set hz = 10000

As you have the Solaris source code, if you dig around in it for a
while, you'll find a brief discussion of the legal values for this
variable hz.  I don't have my copy of the source online right now and
I don't recall the file name.  In any case, this value can be cranked
up.  I'll leave it as an excise to the reader; if you can't find the
relevant code segment and discussion, then you shouldn't be tweaking
this value.

Note however, that you could potentially find that your system has some
problems after raising this value.  Like most things, I'd say try it and
see what happens (and please report the results back here).  Resetting
the value will make the problems go away.

--

 
 
 

Achieving high-resolution timeouts in Solaris (w/ Source Code)

Post by Casper H.S. Dik - Network Security Engine » Tue, 20 Jul 1999 04:00:00


[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]


Quote:>You can't.  :-(  You *might* be able to tune the kernel to run the main
>system clock at a different rate, say 1 ms per tick, but that's as close
>as you can come to your desired resolution.  You will always have a
>latency of whatever the clock tick is; by default it's 10 milliseconds.

Solaris supports one other setting "set hires_tick = 1" but other settings
work (set hz = 10000) but are not supported and progressively slow
the system down (until it will no longer boot at really high MHz)

In future, Sun will probably deliver a method of better timing.

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

Achieving high-resolution timeouts in Solaris (w/ Source Code)

Post by The Sh » Wed, 21 Jul 1999 04:00:00



Quote:>That's the answer, unfortunately.  I'm interested to hear, though, why
>you need half-millisecond precision.  That's an extremely tight requirement

  Thank you Christopher for a very useful response.  It was much
appreciated.  And thanks even for the Be pitch.

Now, here's why I'm looking for such a hi-res clock:  I'm writing high
bit rate (6-20 mb/s) video pumps that stream on AAL-5 pvc's (that's ATM
stuff).  The writes have to be two MPEG-2 transport frames, which is 376
bytes (3008 bits), per ATM pdu.   For a stream intended at 6 mbs, this means I need
to write my 3k bits approx every 500000 ns.

But I can't put the timer in the inner most loop.  I think with the AAL-5
rate control (where max/ave rates can be set) together with higher level
throttling  such as timing the write of a megabit of data, I might get
to my goal.

The intended machines though are modified set-top boxes that were made
for receiving digital satelite.  These clients do close to zero buffering
and are very sensitive to jitter.

Thanks again,

- Jim Schumacher (The Shoe)


 
 
 

Achieving high-resolution timeouts in Solaris (w/ Source Code)

Post by Kaz Kylhe » Wed, 21 Jul 1999 04:00:00





>>The intended machines though are modified set-top boxes that were made
>>for receiving digital satelite.  These clients do close to zero buffering
>>and are very sensitive to jitter.

>Shouldn't this be handled by traffic shaping?

>If you write enough data into the kernel's stream buffers , it should be possible
>to output a continuously almost jitter free data stream.

Assuming that the network hardware provides interrupts at the right
granularity. I take it that the idea here is that the bits could be
sent much faster.

A purely software traffic shaping scheme is going to be constrained to using
the same 100Hz tick for its internal timers, so it has the same problem.

Ideally, the hardware would provide the ability to clock the bits at
the proper rate!

E.g. when driving a 9600 baud serial port, you don't need to worry about
generating 1/9600th of a second delays to clock the bits properly! :) The UART
chip takes care of that, you just stuff the buffers and registers.

 
 
 

Achieving high-resolution timeouts in Solaris (w/ Source Code)

Post by Guy Mac » Sun, 08 Aug 1999 04:00:00



>Now, here's why I'm looking for such a hi-res clock:  I'm writing high
>bit rate (6-20 mb/s) video pumps that stream on AAL-5 pvc's (that's ATM
>stuff).  The writes have to be two MPEG-2 transport frames, which is 376
>bytes (3008 bits), per ATM pdu.   For a stream intended at 6 mbs, this means I need
>to write my 3k bits approx every 500000 ns.

You could use another aproach and set up a piece of hardware that buffers
the output from the computer and clocks it out according to a hires clock.
20 mb of buffer would allow you to tolerate 500 ms pauses as long as the
computer can put out a high enough average bit rate to keep the buffer
full.

I design Laser Beam Recorders that produce masters for CD and DVD.
We face these same issues a lot.  In my opinion, your resolution
requirements are too tight for a general purpose computer, but would
be fairly easy on a add on board with it's own processor (or maybe a
DSP).

 
 
 

Achieving high-resolution timeouts in Solaris (w/ Source Code)

Post by Emil Roja » Tue, 10 Aug 1999 04:00:00





> >Now, here's why I'm looking for such a hi-res clock:  I'm writing high
> >bit rate (6-20 mb/s) video pumps that stream on AAL-5 pvc's (that's ATM
> >stuff).  The writes have to be two MPEG-2 transport frames, which is 376
> >bytes (3008 bits), per ATM pdu.   For a stream intended at 6 mbs, this
means I need
> >to write my 3k bits approx every 500000 ns.

The normal way of achieving this is to write to a  device which has a FIFO.
The hardware has a clock on pulls the bits out of the FIFO as needed.
The only thing you have to do is keep the FIFO from getting empty.

This true for any parallel to serial interface I have ever seen.

For the high resolution kind of timing you need for you MPEG/ATM stream,
your notion of time should be derived from the rate at which bytes
are sucked from the FIFO.

emil
http://www.lapel.com

 
 
 

1. Code fragments/plug-ins for solaris?

Is there anyway to write code fragments or plug-ins easily for Unix and/or
Solaris?

Essentially, I want to do what something like Photoshop does on my Mac (and
what I can do as code resources on my Mac).

A program looks in a directory for its plug-ins.  It plug-in contains a
fragment of code with a defined entry-point and constant interface.  The
plug-in however, is not fully linked or executable.

I want to be able to jump from the main program into the plug-ins according
to when their execution should be appropriate.

The shared object libraries come close... but I don't want the restriction
of having to link the main program with the plug-ins.

There is problably some deep kernel magic involved here, and or icky assembly
code...

If you know how I can do this, would you please mind sending me an email
describing how to do this (or better yet, source code!)?

Thanks.

-Dan

2. Name server problem ?

3. SiS 6205 & high high resolution modes

4. RH6.1 KickStart Install for large IDE disk ?

5. high-resolution timer

6. HELP URGENT:Lost /etc/shadow on SVR4 Unix any backdoor?

7. What Linux version is needed for Xinerama?

8. S3Virge, High Color & High Resolution

9. Q. How to use high-resolution screen mode?

10. IBM ThinkPad -- 8bit color and/or external high-resolution mode

11. how to achieve high speed rate in file transfer

12. use alarm() to achieve timeout.