Very Computer

I am really wondering why it was decided to make Linux's Kernel-space
threads seperate processes.  I see the benchmark at the Linux Thread FAQ
site, ...

"Also the scheduler has been
      optomized so that the switching time for threads vs. tasks varies
little--about 1.7us (threads) and 1.8us (fork) on a
      75MHz Pentium."

This statement itself seems to give me the idea that this decision was a
bad one...I mean, if a thread in Lunix is actually a process, it is no
wonder that the 2 are very close in performance.  This doesn't tell me
that Linux processes are fast, but perhaps Linux threads are slow.

In that same website is stated:

"Kernel-space threads often are implemented in the kernel using several
tables (each task gets a table of threads).
      In this case, the kernel schedules each thread within the
timeslice of each process.  "

granted, my knowlage of the workings of an OS is limited (to say the
least), but that does not sound like the description of a seperate
process to me,...even with shared memory.  Why didn't they do it this
way?

Then, in a book on threads (PThreads Primer, available on the net,
though I forget were...) page 52....

"What about shared memory?

Right now you may be asking yourself this question: "What can threads do
that can't be done by processes ahring memory?"

The first answer is "nothing." Anything that you can do with threads,
you can do with processes ahring memory.  Indeed, a number of vendors
implement a significant portion of thier threads library in roughly this
fashion.  If you are thinking about using shared memory in this fashion,
you should make sure you have (a) plenty of time to kill programming,
(b) plenty more time to kill processing, and (c) plenty of money to burn
buying RAM.

You see: (a) debugging gross-process programs is tough, and the tools
that exist for this are not as good as the ones for MT.  (b) Things take
longer.  In Solaris, creating a process is about 30 times slower than
creating a thread, syncronization variables are about 10 times slower,
and contex switching about 5 times slower.  (c) Processes eat up a lot
of kernel memory.  Building a few thousand threads is no big deal.
Building a few thousand processes is.

You can do everything with shared memory.  It just won't be as easy or
run as fast."

So, now I am wondering, with all the things above stated,...why was it
decided that a thread should be a new process?  It sounds as though that
is one of those ugly "Well, it works" hacks.  Though from the look of
what I have read, it probably doesn't work well...

Maybe some hard core benchmarking between Linux's kernel-space threads,
and those of other OSs would better illistrate how Linux threads are
"better".  That is something I would definately like to see.

Quote:

> You are misinterpreting... Linux runs one process per thread, period.

Quote:

> that is all pretty much opinionated nonsense.  threads have their advantages
> in the *everything* is shared.  processes with shared memory also have
> advantages...  Neither is overwhelmingly better than the other, different
> applications fit each model.

Quote:

> The reason creating a process is slower than creating a thread is
> because of the cloning of resources. In Linux, although a thread
> occupies its own process, it is created by a lightweight system call.
> Because it shares the address space, and other resources, with its
> parent, a Linux thread can be created very fast. Linux threads
> are *lightweight* processes.

> Secondly, synchronization variables

One more thing, what about zombie processes?
I couldn't find a clear answer to this.
With each thread creating a new process, when the parent process exits do we
get zombies, or do all threads exit as threads are supposed to do?

All I could find on this was question 4.6 in the linux-threads faqs.
http://linux-rep.fnal.gov/sundocs/faqs/Threads-FAQ/html/

I know that in threads you can get zombie threads that sit around untill the
main program exits,...but they shouldn't collect on the system like processes
can.

Quote:>In that same website is stated:

>"Kernel-space threads often are implemented in the kernel using several
>tables (each task gets a table of threads).
>      In this case, the kernel schedules each thread within the
>timeslice of each process.  "

>granted, my knowlage of the workings of an OS is limited (to say the
>least), but that does not sound like the description of a seperate
>process to me,...even with shared memory.  Why didn't they do it this
>way?

Quote:>Then, in a book on threads (PThreads Primer, available on the net,
>though I forget were...) page 52....

>"What about shared memory?

>Right now you may be asking yourself this question: "What can threads do
>that can't be done by processes ahring memory?"

>The first answer is "nothing." Anything that you can do with threads,
>you can do with processes ahring memory.  Indeed, a number of vendors
>implement a significant portion of thier threads library in roughly this
>fashion.  If you are thinking about using shared memory in this fashion,
>you should make sure you have (a) plenty of time to kill programming,
>(b) plenty more time to kill processing, and (c) plenty of money to burn
>buying RAM.

>You see: (a) debugging gross-process programs is tough, and the tools
>that exist for this are not as good as the ones for MT.  (b) Things take
>longer.  In Solaris, creating a process is about 30 times slower than
>creating a thread, syncronization variables are about 10 times slower,

Secondly, synchronization variables are implemented quite efficiently
in the LinuxThreads library. Locking a mutex does not result in a
system call unless the requesting process must be forced to wait.

What they are talking about above is kernel-level synchronization
mechanisms such as system V semaphores in which every operation
must go through the kernel. This is not true of the LinuxThreads
library (the library API which makes Linux kernel threads look
like POSIX threads).

Quote:>and contex switching about 5 times slower.  (c) Processes eat up a lot
>of kernel memory.  Building a few thousand threads is no big deal.
>Building a few thousand processes is.

>You can do everything with shared memory.  It just won't be as easy or
>run as fast."

Quote:>So, now I am wondering, with all the things above stated,...why was it
>decided that a thread should be a new process?  It sounds as though that
>is one of those ugly "Well, it works" hacks.  Though from the look of
>what I have read, it probably doesn't work well...

Quote:>Maybe some hard core benchmarking between Linux's kernel-space threads,
>and those of other OSs would better illistrate how Linux threads are
>"better".  That is something I would definately like to see.

: I am really wondering why it was decided to make Linux's Kernel-space
: threads seperate processes.  I see the benchmark at the Linux Thread FAQ
: site, ...

: "Also the scheduler has been
:       optomized so that the switching time for threads vs. tasks varies
: little--about 1.7us (threads) and 1.8us (fork) on a
:       75MHz Pentium."

: This statement itself seems to give me the idea that this decision was a
: bad one...I mean, if a thread in Lunix is actually a process, it is no
: wonder that the 2 are very close in performance.  This doesn't tell me
: that Linux processes are fast, but perhaps Linux threads are slow.

: In that same website is stated:

: "Kernel-space threads often are implemented in the kernel using several
: tables (each task gets a table of threads).
:       In this case, the kernel schedules each thread within the
: timeslice of each process.  "

: granted, my knowlage of the workings of an OS is limited (to say the
: least), but that does not sound like the description of a seperate
: process to me,...even with shared memory.  Why didn't they do it this
: way?

You are misinterpreting... Linux runs one process per thread, period.
All scheduling is done in the kernel, and, from a personal perspective,
this is the best way to do it.  I've run under unicos on the Cray's, and
have cursed the multithreading library for years, because user threads
are not mapped one-to-one to kernel processes...

But in any case, *every* thread you create is a separate process.  If you
try the simple examples from the pthread source, where you do a single
pthread_create() call, and do a "system("ps ax"); from one of the threads,
you'll see two processes, plus a third that handles the thread overhead
for thread_join() stuff and the like...  But they are *all* real kernel
threads...

: Then, in a book on threads (PThreads Primer, available on the net,
: though I forget were...) page 52....

: "What about shared memory?

: Right now you may be asking yourself this question: "What can threads do
: that can't be done by processes ahring memory?"

: The first answer is "nothing." Anything that you can do with threads,
: you can do with processes ahring memory.  Indeed, a number of vendors
: implement a significant portion of thier threads library in roughly this
: fashion.  If you are thinking about using shared memory in this fashion,
: you should make sure you have (a) plenty of time to kill programming,
: (b) plenty more time to kill processing, and (c) plenty of money to burn
: buying RAM.

: You see: (a) debugging gross-process programs is tough, and the tools
: that exist for this are not as good as the ones for MT.  (b) Things take
: longer.  In Solaris, creating a process is about 30 times slower than
: creating a thread, syncronization variables are about 10 times slower,
: and contex switching about 5 times slower.  (c) Processes eat up a lot
: of kernel memory.  Building a few thousand threads is no big deal.
: Building a few thousand processes is.

that is all pretty much opinionated nonsense.  threads have their advantages
in the *everything* is shared.  processes with shared memory also have
advantages...  Neither is overwhelmingly better than the other, different
applications fit each model.  On linux, you can do *either* of course,
as the shmget, shmat and so forth also work, so you can create a shared
memory region, then fork and let both separate processes use that with out
any debugging of other data shared by accident, since there is none...

: You can do everything with shared memory.  It just won't be as easy or
: run as fast."

That makes no sense...  I've done both for many years.  There is no
performance penalty that I have seen, other than algorithmic issues
that affect your programming...

: So, now I am wondering, with all the things above stated,...why was it
: decided that a thread should be a new process?  It sounds as though that
: is one of those ugly "Well, it works" hacks.  Though from the look of
: what I have read, it probably doesn't work well...

: Maybe some hard core benchmarking between Linux's kernel-space threads,
: and those of other OSs would better illistrate how Linux threads are
: "better".  That is something I would definately like to see.

They are as good, or as bad, as threads on any other platform.  They work
perfectly, rapidly, can be created quickly, etc...  The stuff you read is
simply misleading, or you are misinterpreting it badly...

--
Robert Hyatt                    Computer and Information Sciences

(205) 934-2213                  115A Campbell Hall, UAB Station
(205) 934-5473 FAX              Birmingham, AL 35294-1170

:>
:>
:> You are misinterpreting... Linux runs one process per thread, period.
:>

: Actually I think you misinterpreted...I was saying that in the Linux FAQs on
: threads, they say that kernel-space threads are often tables , in other OSs.
: I was asking why Linux's kernel-space threads were not.

:>
:>
:> that is all pretty much opinionated nonsense.  threads have their advantages
:> in the *everything* is shared.  processes with shared memory also have
:> advantages...  Neither is overwhelmingly better than the other, different
:> applications fit each model.

: Of course each has its own benifits.  And some apps are better threaded, some
: are better as seperate processes.  But if you want threads, and get processes
: instead, well then your not getting what you want and using threads instead
: seems rather pointless.

With linux you can get *either*.  A thread under linux fits into the
classic "lightweight process" model...  the entire virtual address space
is shared, ditto for all the proc stuff like file descriptors, and
everything else including the heap.  Threads use the "clone()" system
call that is *very* fast.  Processes use the "fork()" system call which
is also *very* efficient, but not as efficient as clone().

So the difference between "threads" and "processes" is quite vague in
linux, but the way they are used/created are different enough that you
can "tell the difference" if you want.  But if you want "threads" you
get "threads"...  just that each thread is a separate kernel process.

--
Robert Hyatt                    Computer and Information Sciences

(205) 934-2213                  115A Campbell Hall, UAB Station
(205) 934-5473 FAX              Birmingham, AL 35294-1170

: One more thing, what about zombie processes?
: I couldn't find a clear answer to this.
: With each thread creating a new process, when the parent process exits do we
: get zombies, or do all threads exit as threads are supposed to do?

Depends.  If you don't create "detached" threads, then the parent process
must "thread_join()" with each thread.  This  acts sort of like a normal
fork() and wait() facility.

If you create the threads with the "detached" attribute set, when the
thread exits, it goes away because the thread library's "extra" thread
will catch the SIGCHLD and do the wait() so it won't hang as a zombie...

: All I could find on this was question 4.6 in the linux-threads faqs.
: http://linux-rep.fnal.gov/sundocs/faqs/Threads-FAQ/html/

: I know that in threads you can get zombie threads that sit around untill the
: main program exits,...but they shouldn't collect on the system like processes
: can.

If you don't care about "how" a child exits, create detached threads.  If
you do, then don't use detached, and join() to get rid of each zombie as
it finishes its "task"...

--
Robert Hyatt                    Computer and Information Sciences

(205) 934-2213                  115A Campbell Hall, UAB Station
(205) 934-5473 FAX              Birmingham, AL 35294-1170

Come on, this is just UNIX 101.

In the LinuxThreads library, threads are actually started by a thread manager
process which is itself a thread. When they die, they are collected by that
thread manager, so this issue is transparent to the programmer who just
sees an API that is largely POSIX.

Quote:>All I could find on this was question 4.6 in the linux-threads faqs.
>http://linux-rep.fnal.gov/sundocs/faqs/Threads-FAQ/html/

>I know that in threads you can get zombie threads that sit around untill the
>main program exits,...but they shouldn't collect on the system like processes
>can.

The zombies created by failing to do a pthread_join() are not actually
Linux kernel zombies, but merely unreclaimed entries in the
Linuxthread library's data structures. They are library-level zombies.

Quote:

>> that is all pretty much opinionated nonsense.  threads have their advantages
>> in the *everything* is shared.  processes with shared memory also have
>> advantages...  Neither is overwhelmingly better than the other, different
>> applications fit each model.

>Of course each has its own benifits.  And some apps are better threaded, some
>are better as seperate processes.  But if you want threads, and get processes
>instead, well then your not getting what you want and using threads instead
>seems rather pointless.

Linux threading is done the way God meant threading to be done.

God doesn't use a de*. :)

I have a C++ library for logging debugging output from a threaded
program. The output from each thread goes into a separate file.
The output is serialized by going through a mutex, and each line
is timestamped and stamped with a counter. From the logs, I can
figure out failure scenarios that are quite complex. I've
discovered subtle race conditions that would probably not have
been apparent had I used an interactive de*.

Interactive de*s disturb the program too much. They are
primarily useful in debugging the sequential logic of an individual thread.
You can do this with GDB if you arrange for your main thread to
execute the code of interest.

Quote:>All scheduling is done in the kernel, and, from a personal perspective,
>this is the best way to do it.  I've run under unicos on the Cray's, and
>have cursed the multithreading library for years, because user threads
>are not mapped one-to-one to kernel processes...

a) ps becomes impossible to use if you have a lot of threads (Digital Unix
has a -m option to ps to show threads).

b) although being able to schedule threads at the same level as processes
is often good, sometimes you want to schedule threads as if they're a
single process. Reasons: prevent a single process from taking over the
entire machine; allow threads to set scheduling parameters WITHOUT having
to be root (thus, scheduling parameters only affect how the threads are
scheduled relative to other threads within the process - POSIX allows
for (but does not require) a mechanism for specifying thread scheduling
context).

c) the kernel can treat it as one process where necessary - it can
clean up threads, even if there is a bug in the thread libraries; it
can make sure that the whole process only receives one SIGINT signal,
rather than have 50 threads all try to execute the signal handler.

None of this means that threads shouldn't be scheduled by the kernel,
just that extending clone() to allow them to share the PID is still an
important step.

Digital Unix implemented threads at the kernel level in 3.x. In 4.0, they
"improved" them by making the thread library create kernel threads as
necessary, with a special call-back to the thread manager whenever a
thread blocks in the kernel. All thread-level blocking (mutex, condition
variables, join, etc) is handled in the thread library, and the kernel
level thread is re-assigned to another thread as necessary. In general,
it won't try to create more real threads than there are CPUs.

Unfortunately, that caused several problems:

a) a process that created a bunch of threads with real-time priority,
then changed to a non-root UID, would get errors when new kernel-level
threads were created (and there was no way to determine when one would
be created), as it no longer had privileges to create them real-time;
and anyway, the thread priority was internal to the thread library
anyway; with several processes at different priorities, and threads
that needed to change their kernel-level priority to interact properly,
that caused problems.

b) for a process with very few threads, or where the threads were
usually blocked in the kernel instead of blocked in the thread library,
the overhead actually INCREASED. A process would get a minimum of two
threads even if it never tried to create any user-level threads, simply
by being linked with the thread library. If threads were doing a lot of
work that blocked in the kernel (say, a polling loop every millisecond, or
lots of disk output), the manager thread got woken up every single time.

One thing that is really necessary is kernel level support of semaphores
and condition variables, rather then emulating them with signals. This
would also allow the sem_init() family of semaphore operations to
work between processes (using shared memory), rather than only being
implemented in the thread library.

1998-05-26-01:02:36 fractture:

Quote:>I was saying that in the Linux FAQs on threads, they say that kernel-space
>threads are often tables, in other OSs. I was asking why Linux's kernel-space
>threads were not.

The thing that's unusual about Linux is that its process primative isn't so
inefficient and slow as to require completely separate machinery to do fast
threads. Linux's context switches are _fast_. It's process creation is as fast
as practical, and the threads support cuts out the parts of Unix process setup
and teardown that aren't needed, so they're fast, period.

Other Unixes often have somewhat slower process setup and teardown (fixable)
and context switch (not, easily), and so may benefit from a completely
separate implementation of threads in the kernel. Other OSes have been
designed by people who never really outgrew a 1960s image of computing; I
mean, if launching a process requires walking down the hall and feeding a
stack of punch-cards into a machine the size of a small car, then it doesn't
really need to be handled in a bare minimum of instructions, no? Hence the
excruciatingly slow and inefficient process creation/teardown/context switch
of most non-Unix systems today.

-Bennett