signals and wait for event.

signals and wait for event.

Post by Peteris Krumin » Wed, 18 Jun 2003 08:15:05



Hello,

 i have this question:

 My program processes large sets of data (it might process them for
days), if a new data set arrives, a signal is sent to the program, a
signal handler takes place and adds the new data set to the queue
(dinamically allocated) and increases (int) in_queue variable. So, there
can be more thousand data sets added for processing while processing the
current (in_queue is used for checking if maximum data sets in queue are
not exceeded).

 As soon as a data set is processed, the program decreases in_queue
variable and if there are more data sets in the queue processes them, but
if not it should wait until a new data set arrives and start processing
it.

 How should i implement the wait for new data?

 As i understand Linux does not support pselect() syscall (i havent ever
used it on Linux, so does it support or not?).
(Can i emulate pselect() with select() syscall?).
 If it supported, I'd pselect() until desired signal event occurs and
start processing the queue (am I right?).

 Or maybe use a spinlock? while (!in_queue); But spinlocks are very cpu
intensive, arent they?
 (And as in_queue is increased by the signal handler, should it be
defined volitale?)

 Or any other method?

Thanks for your time,
P.Krumins

 
 
 

signals and wait for event.

Post by Fletcher Glen » Wed, 18 Jun 2003 08:37:23



> Hello,

>  i have this question:

>  My program processes large sets of data (it might process them for
> days), if a new data set arrives, a signal is sent to the program, a
> signal handler takes place and adds the new data set to the queue
> (dinamically allocated) and increases (int) in_queue variable. So, there
> can be more thousand data sets added for processing while processing the
> current (in_queue is used for checking if maximum data sets in queue are
> not exceeded).

>  As soon as a data set is processed, the program decreases in_queue
> variable and if there are more data sets in the queue processes them, but
> if not it should wait until a new data set arrives and start processing
> it.

>  How should i implement the wait for new data?

>  As i understand Linux does not support pselect() syscall (i havent ever
> used it on Linux, so does it support or not?).
> (Can i emulate pselect() with select() syscall?).
>  If it supported, I'd pselect() until desired signal event occurs and
> start processing the queue (am I right?).

>  Or maybe use a spinlock? while (!in_queue); But spinlocks are very cpu
> intensive, arent they?
>  (And as in_queue is increased by the signal handler, should it be
> defined volitale?)

>  Or any other method?

> Thanks for your time,
> P.Krumins

In unix, a sleep() will be interrupted by a signal.  All you need do
is sleep() when you have nothing better to do.

--
                Fletcher Glenn


 
 
 

signals and wait for event.

Post by Peteris Krumin » Wed, 18 Jun 2003 09:15:05




[...]

Quote:> In unix, a sleep() will be interrupted by a signal.  All you need do
> is sleep() when you have nothing better to do.

unsigned int sleep(unsigned int seconds);

it takes seconds as parameter, should i
#include <limits.h> and
sleep(UINT_MAX) to wait indefinately?
or is there a trick to sleep until interrupted?

P.Krumins

 
 
 

signals and wait for event.

Post by Kurtis D. Rade » Wed, 18 Jun 2003 11:57:04





>> In unix, a sleep() will be interrupted by a signal.  All you need do is
>> sleep() when you have nothing better to do.

> unsigned int sleep(unsigned int seconds);

> it takes seconds as parameter, should i #include <limits.h> and
> sleep(UINT_MAX) to wait indefinately?  or is there a trick to sleep until
> interrupted?

One option is to use sigsuspend() for this rather than sleep. However,
to avoid race conditions you need to be very careful how you structure
the code and handle the signal. For example, a naive implementation
would do something like this (psuedo-code) in the main loop:

    if( in_queue == 0 ) sigsuspend();

The problem is that between checking "in_queue" and actually going to
sleep waiting for a signal via sigsuspend() a signal could be delivered
that results in a data set being queued. The sigsuspend() will then
sleep until the next occurance of the signal is delivered.

Also, remember that there are very few functions that can be safely
called from within an asynchronous interrupt handler. And malloc()
isn't one of them.

 
 
 

signals and wait for event.

Post by Valentin Nechaye » Wed, 18 Jun 2003 15:47:23


PK>  My program processes large sets of data (it might process them for
PK> days), if a new data set arrives, a signal is sent to the program, a
PK> signal handler takes place and adds the new data set to the queue
PK> (dinamically allocated) and increases (int) in_queue variable. So, there
[...]

PK>  How should i implement the wait for new data?
PK>  As i understand Linux does not support pselect() syscall (i havent ever
PK> used it on Linux, so does it support or not?).
PK> (Can i emulate pselect() with select() syscall?).

Yes. But not directly. Any time you want pselect() and haven't it seeing
implemented, use notification pipe. Signal handler write any data to write
end of notification pipe (one arbitrary byte is enough, write error is allowed
without algorithm failure); main cycle waits for read end of notification
pipe in select() and drops all pending data from it on each cycle.
This prevents race condition when signal may occur and dataset may be got
between checking of queue and entering select().

PK>  If it supported, I'd pselect() until desired signal event occurs and
PK> start processing the queue (am I right?).
PK>  Or maybe use a spinlock? while (!in_queue); But spinlocks are very cpu
PK> intensive, arent they?

This isn't spinlock, it is spinning condition checking;))
No, spinning isn't needed. Your main cycle can be similar to:

block signals;
forever {
        if( no data in queue ) {
                unblock signals;
                sleep in select (reading from notification pipe);
                empty notification pipe;
                block signals;
        }
        else {
                get dataset from queue;
                unblock signals; /* if processing of dataset is simple */
                process dataset;
                block signals; /* if unblocked 2 lines above */
        }

Quote:}

unblock signals;

Note signal blocking framing here. Signals should be blocked only when
needed; for your case it is working with queue (checking for emptiness
and getting dataset from it), but possibly not during dataset processing.
That's why I wrote unblocking immediately when queue isn't already touched
(I suppose that getting dataset from queue detaches it from queue structures).
But, if your signal handler uses functions outside of async-signal-safe
set (stdio functions, memory allocation, etc.), this won't work directly and
you'll need explicit opening of time windows for signal handling, either
around select() and in dataset processing. (Another approach is when
signal handler only notifies main cycle, and main cycle gets datasets from
external source. This has less problems with async-signal-safe function set.)

PK>  (And as in_queue is increased by the signal handler, should it be
PK> defined volitale?)

You mean volatile? If it, answer is yes.

PK>  Or any other method?

Multithreading can help to write this in another way and possibly more
correct. If speed of getting datasets on signal is important, and dataset queue
is large, consider creating separate thread to get datasets and put them
to internal queue. If dataset source can wait for the longest processing
time, remain with complicated signal handler.

-netch-

 
 
 

signals and wait for event.

Post by Peteris Krumin » Thu, 19 Jun 2003 05:25:18





> PK>  My program processes large sets of data (it might process them
> for PK> days), if a new data set arrives, a signal is sent to the
> program, a PK> signal handler takes place and adds the new data set to
> the queue PK> (dinamically allocated) and increases (int) in_queue
> variable. So, there [...]

> PK>  How should i implement the wait for new data?
> PK>  As i understand Linux does not support pselect() syscall (i
> havent ever PK> used it on Linux, so does it support or not?).
> PK> (Can i emulate pselect() with select() syscall?).

> Yes. But not directly. Any time you want pselect() and haven't it
> seeing implemented, use notification pipe. Signal handler write any
> data to write end of notification pipe (one arbitrary byte is enough,
> write error is allowed without algorithm failure); main cycle waits
> for read end of notification pipe in select() and drops all pending
> data from it on each cycle. This prevents race condition when signal
> may occur and dataset may be got between checking of queue and
> entering select().

How is this notification pipe created?

- Show quoted text -

Quote:> PK>  If it supported, I'd pselect() until desired signal event occurs
> and PK> start processing the queue (am I right?).
> PK>  Or maybe use a spinlock? while (!in_queue); But spinlocks are
> very cpu PK> intensive, arent they?

> This isn't spinlock, it is spinning condition checking;))
> No, spinning isn't needed. Your main cycle can be similar to:

> block signals;
> forever {
>      if( no data in queue ) {
>           unblock signals;
>           sleep in select (reading from notification pipe);
>           empty notification pipe;
>           block signals;
>      }
>      else {
>           get dataset from queue;
>           unblock signals; /* if processing of dataset is simple */
>           process dataset;
>           block signals; /* if unblocked 2 lines above */
>      }
> }
> unblock signals;

> Note signal blocking framing here. Signals should be blocked only when
> needed; for your case it is working with queue (checking for emptiness
> and getting dataset from it), but possibly not during dataset
> processing. That's why I wrote unblocking immediately when queue isn't
> already touched (I suppose that getting dataset from queue detaches it
> from queue structures). But, if your signal handler uses functions
> outside of async-signal-safe set (stdio functions, memory allocation,
> etc.), this won't work directly and you'll need explicit opening of
> time windows for signal handling, either around select() and in
> dataset processing. (Another approach is when signal handler only
> notifies main cycle, and main cycle gets datasets from external
> source. This has less problems with async-signal-safe function set.)

Hmm, what happens when signal is being blocked and a new one arrives? Or
more than one arrives, a bunch of them to indicate more data is
available. Are those signals caught after i unblock signals?

Also, the data is received via a named pipe. Is this a good way to
receive something from other programs? In certain cases the named pipe is
opened by multiple writers, and as soon as they write something to it
they send a signal to processing application. If there are multiple
writers to a single pipe, can the data mess up? Like if two writes write
at the same time. Can named pipes be locked? Also, is there a way to
check if a file is locked and if it is wait until its unlocked?

Quote:> PK>  (And as in_queue is increased by the signal handler, should it be
> PK> defined volitale?)

> You mean volatile? If it, answer is yes.

Yes, a typo.

Quote:> PK>  Or any other method?

> Multithreading can help to write this in another way and possibly more
> correct. If speed of getting datasets on signal is important, and
> dataset queue is large, consider creating separate thread to get
> datasets and put them to internal queue. If dataset source can wait
> for the longest processing time, remain with complicated signal
> handler.

I'll stay to complicated signal handler.

P.Krumins

 
 
 

signals and wait for event.

Post by Peteris Krumin » Thu, 19 Jun 2003 05:30:55







>>> In unix, a sleep() will be interrupted by a signal.  All you need do
>>> is sleep() when you have nothing better to do.

>> unsigned int sleep(unsigned int seconds);

>> it takes seconds as parameter, should i #include <limits.h> and
>> sleep(UINT_MAX) to wait indefinately?  or is there a trick to sleep
>> until interrupted?

> One option is to use sigsuspend() for this rather than sleep. However,
> to avoid race conditions you need to be very careful how you structure
> the code and handle the signal. For example, a naive implementation
> would do something like this (psuedo-code) in the main loop:

>     if( in_queue == 0 ) sigsuspend();

> The problem is that between checking "in_queue" and actually going to
> sleep waiting for a signal via sigsuspend() a signal could be
> delivered that results in a data set being queued. The sigsuspend()
> will then sleep until the next occurance of the signal is delivered.

> Also, remember that there are very few functions that can be safely
> called from within an asynchronous interrupt handler. And malloc()
> isn't one of them.

So, what countermeasures should be taken to safely add data to queue
within interrupt handler? Can i block inside a signal handler, add data
to the queue and then unblock? But this could cause my program to go in a
tight loop - if a new signal is caught just after unblocking in signal
handler and so forever - my program would never reach the code which
processes the queue.

P.Krumins

 
 
 

signals and wait for event.

Post by Peteris Krumin » Thu, 19 Jun 2003 05:58:22








[...]

Quote:> How is this notification pipe created?

[continuing myself]

So, from what i have found out, i create usual pipe(2) and make a write to
it in a signal handler, while there is a select(2) waiting for data to
apperar on pipe (is this called 'to wait pipe to unblock'? if not, how is
it called?). Am i right?

P.Krumins

 
 
 

signals and wait for event.

Post by those who know me have no need of my nam » Thu, 19 Jun 2003 06:24:56


in comp.unix.programmer i read:



>> In unix, a sleep() will be interrupted by a signal.  All you need do
>> is sleep() when you have nothing better to do.

>unsigned int sleep(unsigned int seconds);

>it takes seconds as parameter, should i
>#include <limits.h> and
>sleep(UINT_MAX) to wait indefinately?

no, sorry.  in fact lots of sleep's cannot handle too large a value, and
will only sleep for at most a few hours.

Quote:>or is there a trick to sleep until interrupted?

there is not.  use a loop.  check the return value, which will be zero if
the sleep was not interrupted, e.g.,

  while (0 == sleep(1000)) continue;

if you only need to wait indefinitely, until something happens, exactly
what you don't care, then use pause() instead, e.g.,

  pause();

do not be tempted to use pause in a loop testing a flag set by a signal
handler, as that usage contains a race condition.

--
a signature

 
 
 

signals and wait for event.

Post by Kurtis D. Rade » Thu, 19 Jun 2003 11:44:28





>> Also, remember that there are very few functions that can be safely
>> called from within an asynchronous interrupt handler. And malloc() isn't
>> one of them.

> So, what countermeasures should be taken to safely add data to queue
> within interrupt handler? Can i block inside a signal handler, add data
> to the queue and then unblock? But this could cause my program to go in a
> tight loop - if a new signal is caught just after unblocking in signal
> handler and so forever - my program would never reach the code which
> processes the queue.

It's not clear what you mean by "block inside a signal handler". But
in that doesn't really matter since the key thing to keep in mind is
that most functions may not be safely called from a function called
in response to an asynchronously delivered signal. It doesn't matter
if you "block" or not.  Consider this scenario: Your mainline has just
called malloc(). The malloc() function has partially updated its data
structures. A signal arrives that invokes your function whic in turn
calls malloc(). What will happen? If the malloc() function is not
thread-safe then the heap will be corrupted. If it is thread-safe,
and you have a single thread or the thread which handles the signal
is the one currently handling the malloc() call, then your program
will deadlock.

The usual solution involves having the signal handler notify the main
thread that an event needs handling. The main thread then handles the
event synchronously relative to its other work. One of the ways to do
this is a simple atomic counter shared by the signal handler and main
line loop.  Alternatively, use a pipe or socket that the signal handler
stuffs a byte into which will cause the main line loop to be awakened
from a select() or poll().

 
 
 

signals and wait for event.

Post by Peteris Krumin » Fri, 20 Jun 2003 03:12:25







>>> Also, remember that there are very few functions that can be safely
>>> called from within an asynchronous interrupt handler. And malloc()
>>> isn't one of them.

>> So, what countermeasures should be taken to safely add data to queue
>> within interrupt handler? Can i block inside a signal handler, add
>> data to the queue and then unblock? But this could cause my program
>> to go in a tight loop - if a new signal is caught just after
>> unblocking in signal handler and so forever - my program would never
>> reach the code which processes the queue.

> It's not clear what you mean by "block inside a signal handler". But
> in that doesn't really matter since the key thing to keep in mind is
> that most functions may not be safely called from a function called
> in response to an asynchronously delivered signal. It doesn't matter
> if you "block" or not.  Consider this scenario: Your mainline has just
> called malloc(). The malloc() function has partially updated its data
> structures. A signal arrives that invokes your function whic in turn
> calls malloc(). What will happen? If the malloc() function is not
> thread-safe then the heap will be corrupted. If it is thread-safe,
> and you have a single thread or the thread which handles the signal
> is the one currently handling the malloc() call, then your program
> will deadlock.

I meant blocking signals inside a signal handler just after it executes
and unblock just before it ends it's execution, so if any other signals
arrive whileexecuting a signal handler they'd block.
Is this possible, if so, then malloc() is safe, isnt it, because signals
are blocked and the signal handler code can never be interrupted?

Quote:> The usual solution involves having the signal handler notify the main
> thread that an event needs handling. The main thread then handles the
> event synchronously relative to its other work. One of the ways to do
> this is a simple atomic counter shared by the signal handler and main
> line loop.  Alternatively, use a pipe or socket that the signal
> handler stuffs a byte into which will cause the main line loop to be
> awakened from a select() or poll().

The solution with threading sounds very difficult, do you have any
examples i could follow?

I am doing it by pipe method, but what is unclear to me is what happens
right after a signal handler is called, is everything (registers etc)
pushed to the stack and after the signal handler executes, everything is
popped from the stack and the code flow continues as nothing happened?

P.Krumins

 
 
 

signals and wait for event.

Post by Peteris Krumin » Fri, 20 Jun 2003 08:22:08







> The usual solution involves having the signal handler notify the main
> thread that an event needs handling. The main thread then handles the
> event synchronously relative to its other work. One of the ways to do
> this is a simple atomic counter shared by the signal handler and main
> line loop.  Alternatively, use a pipe or socket that the signal
> handler stuffs a byte into which will cause the main line loop to be
> awakened from a select() or poll().

ok, i just coded working version with the pipe method.
after i stuff a byte in a pipe, how do i empty it? issue a read?
also what happens if i do not read from it and keep writing to it from
signal handler? a memory leak?

P.Krumins

 
 
 

signals and wait for event.

Post by Kurtis D. Rade » Fri, 20 Jun 2003 11:41:32



> I meant blocking signals inside a signal handler just after it executes
> and unblock just before it ends it's execution, so if any other signals
> arrive whileexecuting a signal handler they'd block. Is this possible, if
> so, then malloc() is safe, isnt it, because signals are blocked and the
> signal handler code can never be interrupted?

No, malloc() is still not safe. Not even if you block all other signals
at the time the signal is delivered (see the sa_mask member of struct
sigaction) rather than from within your signal handler.  The reason is
that the signal may have arrived while the thread was executing code
that manipulated the malloc() data structures.  That is, the signal may
be delivered while your process is executing a malloc(), realloc(),
free(), or related call. It isn't enough to keep another signal from
being delivered.

Quote:> I am doing it by pipe method, but what is unclear to me is what happens
> right after a signal handler is called, is everything (registers etc)
> pushed to the stack and after the signal handler executes, everything is
> popped from the stack and the code flow continues as nothing happened?

Exactly. Unless, of course, your signal handler calls siglongjmp() or
performs some other action that alters the calling context. Also, of
course, ignoring any side effects (e.g., manipulating a global variable)
that your signal handler may have.
 
 
 

signals and wait for event.

Post by Sean Burk » Sat, 21 Jun 2003 15:41:28








> > The usual solution involves having the signal handler notify the main
> > thread that an event needs handling. The main thread then handles the
> > event synchronously relative to its other work. One of the ways to do
> > this is a simple atomic counter shared by the signal handler and main
> > line loop.  Alternatively, use a pipe or socket that the signal
> > handler stuffs a byte into which will cause the main line loop to be
> > awakened from a select() or poll().

> ok, i just coded working version with the pipe method.
> after i stuff a byte in a pipe, how do i empty it? issue a read?
> also what happens if i do not read from it and keep writing to it from
> signal handler? a memory leak?

Then the pipe will fill. You should make the pipe
non-blocking, so that your signal-handler won't
block writing to the pipe.

-SEan

 
 
 

signals and wait for event.

Post by Peteris Krumin » Sun, 22 Jun 2003 05:30:52










>> > The usual solution involves having the signal handler notify the
main
>> > thread that an event needs handling. The main thread then handles
the
>> > event synchronously relative to its other work. One of the ways to
do
>> > this is a simple atomic counter shared by the signal handler and
main
>> > line loop.  Alternatively, use a pipe or socket that the signal
>> > handler stuffs a byte into which will cause the main line loop to be
>> > awakened from a select() or poll().

>> ok, i just coded working version with the pipe method.
>> after i stuff a byte in a pipe, how do i empty it? issue a read?
>> also what happens if i do not read from it and keep writing to it from
>> signal handler? a memory leak?

> Then the pipe will fill. You should make the pipe
> non-blocking, so that your signal-handler won't
> block writing to the pipe.

Yes, i figured that out.

But now i have run in bigger problem, if you have followed the thread
then you know that the program i write processes data sets and more can
arrive at any time. As the data processing takes reasonable time after
getting the first data set my program only executes that one (processes)
and the notification pipe method wont work - only the code of data
processing is executed and it's not waiting for select() to return, so as
a new data set arrives, a write to pipe is issued (from signal handler),
but as there is no select(), the program just continues processing the
data set and does not add the next one to the queue.
Processing can take hours, data sets are small (< 512 bytes), new ones
can arrive each minute.

P.Krumins