Very Computer

I was wondering if anyone could describe to me the "joys" of implmenting a user
level locking system which would work in a multi-process/multi-threaded environment.

For years I've been using the likes of lockf(), flock(), and finaly now that I
am a thread-head I use SYSV semaphores. Trouble is functions like lockf() and
flock() are basicly useless in a multi-threaded arena and SYSV semaphores are a
scarce resource (I need thousands of locks for my latest creation).

I also don't like process level mutexes as there is no graceful way of
recovering from process/thread death on orphaned mutex lock.

So I've snarfed some user level "compare_and_swap" assembler routines from a
Linux pthreads implementation and it appears to work find. But being the person
I am I would like to know why why works. My gut feeling on using this user level
locking (low level or now) is to run away screaming and go and seek confort in a
nice safe kernel level locking system?.

Cheers...

--
The only problem with seeing too much is that it makes you insane.
                -- Phaedrus

Quote:>So I've snarfed some user level "compare_and_swap" assembler routines from a
>Linux pthreads implementation and it appears to work find. But being the person
>I am I would like to know why why works. My gut feeling on using this user level
>locking (low level or now) is to run away screaming and go and seek confort in a
>nice safe kernel level locking system?.

You could also have some hybrid form of locking, a locking library
which is implemented on top of both flock/lockf and mutexes;  to acquire
a lock, a thread needs to acquire both a process-private mutex and a file lock.

When the process goes away, it's file locks are released and the mutexes
cease to exist.

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

I am using Solaris and Linux (RedHat 7.0) pthreads and solaris mutex's and
neither of them will survive process death gracefully.

If I could have a mutex that I could release in a controlled fashion that would
be fine. Which is basically my logic of talking somebody's implementation of
pthread mutexes and using the atomic user locking to do what I need to do.

However since I don't understand exactly why it works I am worried about using
it in anger. Hence my require for information.

Quote:> How does your compare and swap approach help you against processes
> dying? (Or are you now updating data this way)?

Quote:> You could also have some hybrid form of locking, a locking library
> which is implemented on top of both flock/lockf and mutexes;  to acquire
> a lock, a thread needs to acquire both a process-private mutex and a file lock.

And in any case flock and lock suffer from the major disadvantage that they
require an active filehandle. I need thousands of locks and filehandles are
another scarce resource that I don't need to deplete/run out of.

Basically I am/have developed an ultra fast database module (using mmap()) which
is designed to allow no less than 5000 transactions per second to go through it.
I need to lock parts of the indices and that could mean anything from 1 lock to
a million.

Quote:> When the process goes away, it's file locks are released and the mutexes
> cease to exist

Ideally I would gmutex the memory blocks I require and if a thread/process died
in a lock I could then have my overseer process detect its death free off the
block and resume the entire system elegantly.

With standard mutex I would have to restart pretty much the entire system, which
from mine and what it's intended for is unrealistic.

Regards Simon

--
The idle man does not know what it is to enjoy rest.

    USYNC_PROCESS_ROBUST
           The mutex can synchronize threads in this process  and
           other  processes  robustly.  At  the  time  of process
           death, if the lock is  held  by  the  process,  it  is
           unlocked. The next owner of this mutex will acquire it
           with an error return of   EOWNERDEAD.  Note  that  the
           application  must  always  check  the return code from
           mutex_lock() for a mutex of this type. The  new  owner
           of  this  mutex  should then attempt to make the state
           protected by the mutex consistent,  since  this  state
           could  have been left inconsistent when the last owner
           died. If the new owner is able to make the state  con-
           sistent,  it  should  re-initialize the mutex and then
           unlock the mutex. If the new owner is not able to make
           the  state  consistent, for whatever reason, it should
           not re-initialize the mutex, but  should  just  unlock
           the  mutex.  If  the  latter event occurs, all waiters
           will  be  woken  up  and  all  subsequent   calls   to
           mutex_lock()  will fail in acquiring the mutex with an
           error code of  ENOTRECOVERABLE. mutex can be made con-
           sistent      by      un-initializing     the     mutex
           (mutex_destroy())     and      re-initializing      it
           (mutex_init()). If the process which got the lock with
           EOWNERDEAD died, the next owner will get the lock with
           an  error  return  of  EOWNERDEAD. arg is ignored. The
           object initialized with this attribute must  be  allo-
           cated  in  memory  shared between processes, either in
           System V shared memory (see  shmop(2))  or  in  memory
           mapped  to  a  file  (see  mmap(2)) and memory must be
           zeroed  before  initialization.  All   the   processes
           interested  in  the robust lock must call mutex_init()
           at least once to register robust mutex with the system
           and  potentially  initialize  it. If the object is not
           allocated in such shared memory, it will not be shared
           between  processes.  If   mutex_init()  is called on a
           previously initialized  mutex  mutex_init()  will  not
           re-initialize the mutex.

Quote:>Trust me on this
>lockf() is basically useless in a multi-threaded environment with many threads
>accessing different locks due to its process level deadlock checking.

Quote:>And in any case flock and lock suffer from the major disadvantage that they
>require an active filehandle. I need thousands of locks and filehandles are
>another scarce resource that I don't need to deplete/run out of.

Solaris will actually handle thousands of files opened in a single process.

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Quote:> [[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]

> The Solaris 7 and later manual page for mutex_init lists the
> following lock type:

>     USYNC_PROCESS_ROBUST
>            The mutex can synchronize threads in this process  and
>            other  processes  robustly.  At  the  time  of process
>            death, if the lock is  held  by  the  process,  it  is
>            unlocked. The next owner of this mutex will acquire it
>            with an error return of   EOWNERDEAD.  Note  that  the
>            application  must  always  check  the return code from
>            mutex_lock() for a mutex of this type. The  new  owner

If forgot to mention we are on Solaris 2.6, attempts to install/use Solaris 2.7
resulted in out machine crashing/hanging and generaly acting in a unstable way.

I understand that Solaris 2.8 is a lot more "robust" but I think I'll wait and
see what problems (if any) the everyone else has before going for that one.

Plus this doesn't help with Linux I would prefer a more "general" solution, I
suspect that I would get a big slap if I wanted to increase the cost of my
system by a few 100k..<grin>

Quote:> This is why you would need to implement a second layer; within a process
> your require threads to use mutexes to  lock each other out; outside
> the process, you need to use lockf to lock other processes out.

Because your effectly serialising any access from a single processes threads to
the lockf call. That means that only a single resource can be locked at any time.

In practice I found that all of my threads in a single process would sepnd over
50% of there time waiting for lockf().

Quote:>> And in any case flock and lock suffer from the major disadvantage that they
>> require an active filehandle. I need thousands of locks and filehandles are
>> another scarce resource that I don't need to deplete/run out of

Quote:> You can have many locks per file handle.

Quote:> Solaris will actually handle thousands of files opened in a single process.

Imagine a database which is effectly several million 32 bytes blocks each with
its own lock. Now its unlikey that it will ever require every single node to be
locked but I can think of a few "extreme" instances where its might get close.
Still its an interesting question?, how many file handles can a single process
have open at one time without it causing performance problems.`

Regards Simon

Quote:><Grin> That would work but I've already been down this road a few years back..
>Because your effectly serialising any access from a single processes threads to
>the lockf call. That means that only a single resource can be locked at any time.
>In practice I found that all of my threads in a single process would sepnd over
>50% of there time waiting for lockf().

Quote:>True but the diffirence in preforance between locking an entire file or an
>offset is several thousands. I did preformance testing on this a while back
>and lockf is capable of many thousands of lock requests on a while file this
>soon drops as you increase the number of offsets.  Basicly its quicker to lock
>the entire file than use multible areas.

Quote:>> Solaris will actually handle thousands of files opened in a single process.
>But not hundreds of thousands.

Quote:>Imagine a database which is effectly several million 32 bytes blocks each with
>its own lock. Now its unlikey that it will ever require every single node to be
>locked but I can think of a few "extreme" instances where its might get close.
>Still its an interesting question?, how many file handles can a single process
>have open at one time without it causing performance problems.`

When using large numbers of fds, make sure you allocate the highest
numbered one first as the table grows; growing it n one is cheaper than
growing it in steps.  close(fcntl(0, F_DUPFD, 1000000));

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Quote:> [[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]
> Running Solaris is basically free, so it wouldn't cost another
> $100K.

I don't particularly want to wave the strengths of Linux of Solaris, I've used
Sun equipment and O/S's for a lot of years now.  It took a lot to convince me
that Linux is a better development environment and that performance wise it
beats the pants of Solaris (Won't even name Microsloth O/S's).

But here I sit developing a real-time multi-threaded module on my Linux (RH7.0)
laptop for a system that is mostly Solaris based.

Sad but true.

Quote:>> In practice I found that all of my threads in a single process would sepnd over
>> 50% of there time waiting for lockf().
> Depends very Sigh.  If you implement it inclorrectly, then yes.  If you have
> some sort of filelock <-> mutex mapping, you can get any granulrity you want.

I'll certainly have another look at flock methinks..

In fact I think I'll go and do some performance figures now..

Quote:> Actually, it does.  Not that we really support > 64K, but larger numbers
> work as long as you stay away from poll/select.  You will want to run
> Solaris 8 though, as it is *much* more efficient in using many
> file descriptors.

Quote:> In Solaris 8, well over a million.  (The filehandle allocation procedures
> in S8 are O(2log(n)), i.e., it's twice as costly to allocate fd #1000,000
> than it is to allocate fd #1000 and only 3 times as costly to allocate
> fd #1000,000,000 (but surely something will break before that,)

Oh well things will come out in the wash (performance testing)

Quote:> When using large numbers of fds, make sure you allocate the highest
> numbered one first as the table grows; growing it n one is cheaper than
> growing it in steps.  close(fcntl(0, F_DUPFD, 1000000));

Regards Simon

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Have a good xmas...

<Wave>

Quote:>I don't particularly want to wave the strengths of Linux of Solaris, I've used
>Sun equipment and O/S's for a lot of years now.  It took a lot to convince me
>that Linux is a better development environment and that performance wise it
>beats the pants of Solaris (Won't even name Microsloth O/S's).

>But here I sit developing a real-time multi-threaded module on my Linux (RH7.0)
>laptop for a system that is mostly Solaris based.

--
Andrew Gabriel
Consultant Software Engineer

 Andrew> Reading through this thread, I'm at a loss to see why you
 Andrew> don't just use a PTHREAD_PROCESS_SHARED mutex. That will
 Andrew> work both between threads in one process, and between
 Andrew> [threads in] different processes, thus providing you with
 Andrew> a multi-thread/multi-process locking primitive.

in the original post he pointed out (correctly) that this gives you
no clean way to handle the case where a process dies or is killed
while holding a mutex.

--
Andrew.

comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
                           or <URL: http://www.whitefang.com/unix/>