LINUX POSIX threads - running 1500+ threads

LINUX POSIX threads - running 1500+ threads

Post by Andrew Ar » Fri, 22 Feb 2002 06:54:12



my server is tuned to handle up to 2048 POSIX threads (GLIBC recompiled)

this is a test C code

it starts ~1500 threads in less a second - and SEGMENTATION FAULT!!!!

Am i crazy?

I do NOT see any places here to have a FAULT

PS: "wake" is not printed - all this happend in less a second on my Athlon 1.4

===============
#include <pthread.h>
#include <pwd.h>
#include <stdio.h>
#include <errno.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/un.h>
#include <unistd.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <netdb.h>

#define MAX_THREADS 1700

int thread_counter;

void *entry_point(void *);

int main() {
 pthread_t child_id;
 int result;

 while(1) {
  thread_counter++;
  if(thread_counter>MAX_THREADS) {
   break;
  }
  result = pthread_create(&child_id, NULL, &entry_point, (int *)thread_counter);
  if(result==-1) {
   printf("FATAL: error creating thread\n");exit(1);
  }
  if(thread_counter % 100==0) {
   printf("Started %d threads...\n", thread_counter);
  }
 }
 printf("Weating all threads to finish...\n");
 while(thread_counter>0) {
  sleep(1);                      
 }
 return 0;

Quote:}

void *entry_point(void * thread_id) {
 int counter, par, file_pos, pos;
 char *cur_ip;
 FILE *report;
 sleep(10);
 printf("wake\n");
Quote:}

 
 
 

LINUX POSIX threads - running 1500+ threads

Post by Jeff Grei » Fri, 22 Feb 2002 14:24:39


Using this many threads is usually considered to be a Bad Idea (TM).
Most likely you get the segfault because the default stacksize for each
thread is 1MB or something sufficiently large to run your process out of
virtual memory.  Try reducing your VM limit (see ulimit in the bash
docs) to 50MB -- this should cause you to get the segfault at a much
smaller number of threads if this theory is correct.

Thinking you need this many threads is usually the result of a design
that could be significantly improved.  Take a look at the ACE website
www.cs.wustl.edu/~schmidt/ACE.html including the various tutorials about
networking architectures and concurrency patterns.  There are lots of
good papers and interesting books referenced.

There are also some errors, presumably introduced in order to simplify
the example from something longer.
1.  You loop testing the value of thread counter, which is never
decremented.
2.  You should be joining with the threads, or creating them detached.
3.  If the threads are supposed to decrement thread_counter, accesses to
that variable should be with a mutex locked, or using some atomic
instruction (not thread_counter++).
4.  If you *really*, *really*, need a huge number of threads, you should
probably provide a smaller stacksize for each in the pthread_create
call.  But you almost certainly do not need this many.

Jeff


...

Quote:

> #define MAX_THREADS 1700

> int thread_counter;

> void *entry_point(void *);

> int main() {
>  pthread_t child_id;
>  int result;

>  while(1) {
>   thread_counter++;
>   if(thread_counter>MAX_THREADS) {
>    break;
>   }
>   result = pthread_create(&child_id, NULL, &entry_point, (int
*)thread_counter);
>   if(result==-1) {
>    printf("FATAL: error creating thread\n");exit(1);
>   }
>   if(thread_counter % 100==0) {
>    printf("Started %d threads...\n", thread_counter);
>   }
>  }
>  printf("Weating all threads to finish...\n");
>  while(thread_counter>0) {
>   sleep(1);
>  }
>  return 0;
> }

> void *entry_point(void * thread_id) {
>  int counter, par, file_pos, pos;
>  char *cur_ip;
>  FILE *report;
>  sleep(10);
>  printf("wake\n");
> }


 
 
 

LINUX POSIX threads - running 1500+ threads

Post by ctr2s.. » Fri, 22 Feb 2002 17:42:41



> my server is tuned to handle up to 2048 POSIX threads (GLIBC recompiled)

_Why?_

--

"I woke up this morning and realized what the game needed: pirates,
pimps, and gay furries."  - Rich "Lowtax" Kyanka

 
 
 

LINUX POSIX threads - running 1500+ threads

Post by Andrew Ar » Fri, 22 Feb 2002 18:59:28


LOTS of thanx for You answer! I will read the html about another ways
of serving many requests, but:


> Using this many threads is usually considered to be a Bad Idea (TM).
> Most likely you get the segfault because the default stacksize for each
> thread is 1MB or something sufficiently large to run your process out of
> virtual memory.  Try reducing your VM limit (see ulimit in the bash
> docs) to 50MB -- this should cause you to get the segfault at a much
> smaller number of threads if this theory is correct.

> Thinking you need this many threads is usually the result of a design
> that could be significantly improved.  Take a look at the ACE website
> www.cs.wustl.edu/~schmidt/ACE.html including the various tutorials about
> networking architectures and concurrency patterns.  There are lots of
> good papers and interesting books referenced.

> There are also some errors, presumably introduced in order to simplify
> the example from something longer.
> 1.  You loop testing the value of thread counter, which is never
> decremented.

yes, NOW this is true. this example is a CUTTED version of by program
- so, this is why seems to be a dead lock - but anyway it crashes
BEFORE sub-threads wake up

Quote:> 2.  You should be joining with the threads, or creating them detached.

i have tried
 result = pthread_attr_init(&tattr);
 result = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);
 result = pthread_create(&child_id, &tattr, &entry_point, (int
*)thread_counter);

 - but the same SEGM FAULT

Quote:> 3.  If the threads are supposed to decrement thread_counter, accesses to
> that variable should be with a mutex locked, or using some atomic
> instruction (not thread_counter++).

 why can not i use thread_counter--? is thread_counter =
thread_counter - 1
better in case of multithreading? i see no sence in blocking HERE (of
cause i use block in ather places with hashes or some BIG
structutres). Please write a little more details

Quote:> 4.  If you *really*, *really*, need a huge number of threads, you should
> probably provide a smaller stacksize for each in the pthread_create
> call.  But you almost certainly do not need this many.

I have 768Mb RAM, this test soft uses 10Mb OR LESS of RAM on my LINUX
2.4 BOX
No swap is EVER used.

I am sure i do NOT need 8Mb of stack, but i see no reason of SEGM
FAULT here

- Show quoted text -

> Jeff



> ...

> > #define MAX_THREADS 1700

> > int thread_counter;

> > void *entry_point(void *);

> > int main() {
> >  pthread_t child_id;
> >  int result;

> >  while(1) {
> >   thread_counter++;
> >   if(thread_counter>MAX_THREADS) {
> >    break;
> >   }
> >   result = pthread_create(&child_id, NULL, &entry_point, (int
>  *)thread_counter);
> >   if(result==-1) {
> >    printf("FATAL: error creating thread\n");exit(1);
> >   }
> >   if(thread_counter % 100==0) {
> >    printf("Started %d threads...\n", thread_counter);
> >   }
> >  }
> >  printf("Weating all threads to finish...\n");
> >  while(thread_counter>0) {
> >   sleep(1);
> >  }
>  return 0;
> > }

> > void *entry_point(void * thread_id) {
> >  int counter, par, file_pos, pos;
> >  char *cur_ip;
> >  FILE *report;
> >  sleep(10);
> >  printf("wake\n");
> > }

 
 
 

LINUX POSIX threads - running 1500+ threads

Post by Jeff Grei » Sat, 23 Feb 2002 14:48:14



Quote:> > 2.  You should be joining with the threads, or creating them
detached.

> i have tried
>  result = pthread_attr_init(&tattr);
>  result = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);
>  result = pthread_create(&child_id, &tattr, &entry_point, (int
> *)thread_counter);

>  - but the same SEGM FAULT

I was just pointing out a separate error.  Not joining to the threads
will just leak resources, not cause the seg fault.  But you should join
to them or create them detached and have them decrement the thread
counter in order to have an orderly shutdown of your process (so the
main thread can know when to exit).

Quote:

> > 3.  If the threads are supposed to decrement thread_counter,
accesses to
> > that variable should be with a mutex locked, or using some atomic
> > instruction (not thread_counter++).

>  why can not i use thread_counter--? is thread_counter =
> thread_counter - 1
> better in case of multithreading? i see no sence in blocking HERE (of
> cause i use block in ather places with hashes or some BIG
> structutres). Please write a little more details

Two threads could be decrementing thread_counter at the same time, or
one could be decrementing while the other is incrementing.  You have to
synchronize access to this variable.  Furthermore, if you ever run on a
multiprocessor machine, you must synchronize in order to ensure that all
the processors have the same value in their cache when they read or
write this variable.

Quote:

> > 4.  If you *really*, *really*, need a huge number of threads, you
should
> > probably provide a smaller stacksize for each in the pthread_create
> > call.  But you almost certainly do not need this many.

> I have 768Mb RAM, this test soft uses 10Mb OR LESS of RAM on my LINUX
> 2.4 BOX
> No swap is EVER used.

You can get the seg fault by running out of *virtual* memory even though
not much ram is used.  If each of 1500 threads reserves 1MB of stack, I
believe that 1.5GB of virtual memory is used.  If this is over your
per-process limit, the process will crash, even though no thread is
actually using that much memory.

The way to avoid having 2000 threads blocking for i/o you can use an
event-driven architecture.  The i/o completions produce events which are
handled by a single demultiplexer, which simply dispatches them to a
pool of threads for handling.  When a thread finishes dealing with such
an event, it puts itself back into the pool of waiters.  If each thread
in your proposed app would be blocking 99% of the time, you could use
just 20 threads (or maybe a few more) instead of 2000 to handle the i/o.

See the Reactor and Proactor patterns on the ACE website, or in the book
Pattern-Oriented Software Architecture vol 2 also described on that
site.

Jeff

 
 
 

LINUX POSIX threads - running 1500+ threads

Post by Alexander Terekho » Sat, 23 Feb 2002 18:27:46


[...]

Quote:> See the Reactor and Proactor patterns on the ACE website, or in the book
> Pattern-Oriented Software Architecture vol 2 also described on that
> site.

Yeah, Schmidt et al books/papers are definitely worth reading,
and ACE code is worth of brief examination too... ;-) but I
would really recommend to click on these links as well (*after*
completing work with papers/books and ACE code inspection):

http://www.kegel.com/c10k.html
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndl...
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnms...

regards,
alexander.

 
 
 

LINUX POSIX threads - running 1500+ threads

Post by Ed Skinne » Mon, 25 Feb 2002 00:36:14


     The pthread_create() function does not return -1 on an error. It returns
non-zero. Change your code and re-test because something else may be happening that
explains the seg-fault.


> my server is tuned to handle up to 2048 POSIX threads (GLIBC recompiled)

> this is a test C code

> it starts ~1500 threads in less a second - and SEGMENTATION FAULT!!!!

> Am i crazy?

> I do NOT see any places here to have a FAULT

> PS: "wake" is not printed - all this happend in less a second on my Athlon 1.4

> ===============
> #include <pthread.h>
> #include <pwd.h>
> #include <stdio.h>
> #include <errno.h>
> #include <sys/socket.h>
> #include <sys/types.h>
> #include <sys/un.h>
> #include <unistd.h>
> #include <netinet/in.h>
> #include <arpa/inet.h>
> #include <netdb.h>

> #define MAX_THREADS 1700

> int thread_counter;

> void *entry_point(void *);

> int main() {
>  pthread_t child_id;
>  int result;

>  while(1) {
>   thread_counter++;
>   if(thread_counter>MAX_THREADS) {
>    break;
>   }
>   result = pthread_create(&child_id, NULL, &entry_point, (int *)thread_counter);
>   if(result==-1) {
>    printf("FATAL: error creating thread\n");exit(1);
>   }
>   if(thread_counter % 100==0) {
>    printf("Started %d threads...\n", thread_counter);
>   }
>  }
>  printf("Weating all threads to finish...\n");
>  while(thread_counter>0) {
>   sleep(1);
>  }
>  return 0;
> }

> void *entry_point(void * thread_id) {
>  int counter, par, file_pos, pos;
>  char *cur_ip;
>  FILE *report;
>  sleep(10);
>  printf("wake\n");
> }

--