Kernel-space Threads

Kernel-space Threads

Post by fracttur » Tue, 26 May 1998 04:00:00



I am really wondering why it was decided to make Linux's Kernel-space
threads seperate processes.  I see the benchmark at the Linux Thread FAQ
site, ...

"Also the scheduler has been
      optomized so that the switching time for threads vs. tasks varies
little--about 1.7us (threads) and 1.8us (fork) on a
      75MHz Pentium."

This statement itself seems to give me the idea that this decision was a
bad one...I mean, if a thread in Lunix is actually a process, it is no
wonder that the 2 are very close in performance.  This doesn't tell me
that Linux processes are fast, but perhaps Linux threads are slow.

In that same website is stated:

"Kernel-space threads often are implemented in the kernel using several
tables (each task gets a table of threads).
      In this case, the kernel schedules each thread within the
timeslice of each process.  "

granted, my knowlage of the workings of an OS is limited (to say the
least), but that does not sound like the description of a seperate
process to me,...even with shared memory.  Why didn't they do it this
way?

Then, in a book on threads (PThreads Primer, available on the net,
though I forget were...) page 52....

"What about shared memory?

Right now you may be asking yourself this question: "What can threads do
that can't be done by processes ahring memory?"

The first answer is "nothing." Anything that you can do with threads,
you can do with processes ahring memory.  Indeed, a number of vendors
implement a significant portion of thier threads library in roughly this
fashion.  If you are thinking about using shared memory in this fashion,
you should make sure you have (a) plenty of time to kill programming,
(b) plenty more time to kill processing, and (c) plenty of money to burn
buying RAM.

You see: (a) debugging gross-process programs is tough, and the tools
that exist for this are not as good as the ones for MT.  (b) Things take
longer.  In Solaris, creating a process is about 30 times slower than
creating a thread, syncronization variables are about 10 times slower,
and contex switching about 5 times slower.  (c) Processes eat up a lot
of kernel memory.  Building a few thousand threads is no big deal.
Building a few thousand processes is.

You can do everything with shared memory.  It just won't be as easy or
run as fast."

So, now I am wondering, with all the things above stated,...why was it
decided that a thread should be a new process?  It sounds as though that
is one of those ugly "Well, it works" hacks.  Though from the look of
what I have read, it probably doesn't work well...

Maybe some hard core benchmarking between Linux's kernel-space threads,
and those of other OSs would better illistrate how Linux threads are
"better".  That is something I would definately like to see.

 
 
 

Kernel-space Threads

Post by fracttur » Tue, 26 May 1998 04:00:00


Quote:

> You are misinterpreting... Linux runs one process per thread, period.

Actually I think you misinterpreted...I was saying that in the Linux FAQs on
threads, they say that kernel-space threads are often tables , in other OSs.
I was asking why Linux's kernel-space threads were not.

Quote:

> that is all pretty much opinionated nonsense.  threads have their advantages
> in the *everything* is shared.  processes with shared memory also have
> advantages...  Neither is overwhelmingly better than the other, different
> applications fit each model.

Of course each has its own benifits.  And some apps are better threaded, some
are better as seperate processes.  But if you want threads, and get processes
instead, well then your not getting what you want and using threads instead
seems rather pointless.

 
 
 

Kernel-space Threads

Post by fracttur » Tue, 26 May 1998 04:00:00



> Read it again. They are saying how threads are _often_ implemented
> in typical operating systems, not how they are implemented
> in Linux.  Unless threads are often implemented in Linux but
> sometimes not! :)

I know, thats why I asked why they didn't do it this way, it sounds more like
"threads" then splitting off seperate processes, which have different PIDs and
everything..even if they do share memory and other various recources...Though
I also read that in the next major kernel release clone will have the option
to fork a process that shares the PID of the parent, which may be closer to
what I am expecting from threads...

Quote:

> The reason creating a process is slower than creating a thread is
> because of the cloning of resources. In Linux, although a thread
> occupies its own process, it is created by a lightweight system call.
> Because it shares the address space, and other resources, with its
> parent, a Linux thread can be created very fast. Linux threads
> are *lightweight* processes.

> Secondly, synchronization variables

[snip]Ok, that all makes sence there.
 
 
 

Kernel-space Threads

Post by fracttur » Tue, 26 May 1998 04:00:00


One more thing, what about zombie processes?
I couldn't find a clear answer to this.
With each thread creating a new process, when the parent process exits do we
get zombies, or do all threads exit as threads are supposed to do?

All I could find on this was question 4.6 in the linux-threads faqs.
http://linux-rep.fnal.gov/sundocs/faqs/Threads-FAQ/html/

I know that in threads you can get zombie threads that sit around untill the
main program exits,...but they shouldn't collect on the system like processes
can.

 
 
 

Kernel-space Threads

Post by Kaz Kylhe » Wed, 27 May 1998 04:00:00



>I am really wondering why it was decided to make Linux's Kernel-space
>threads seperate processes.  I see the benchmark at the Linux Thread FAQ
>site, ...

>"Also the scheduler has been
>      optomized so that the switching time for threads vs. tasks varies
>little--about 1.7us (threads) and 1.8us (fork) on a
>      75MHz Pentium."

>This statement itself seems to give me the idea that this decision was a
>bad one...I mean, if a thread in Lunix is actually a process, it is no
>wonder that the 2 are very close in performance.  This doesn't tell me
>that Linux processes are fast, but perhaps Linux threads are slow.

No, but a 1.7 microsecond context switch time is quite fast. :)

Quote:>In that same website is stated:

>"Kernel-space threads often are implemented in the kernel using several
>tables (each task gets a table of threads).
>      In this case, the kernel schedules each thread within the
>timeslice of each process.  "

>granted, my knowlage of the workings of an OS is limited (to say the
>least), but that does not sound like the description of a seperate
>process to me,...even with shared memory.  Why didn't they do it this
>way?

Read it again. They are saying how threads are _often_ implemented
in typical operating systems, not how they are implemented
in Linux.  Unless threads are often implemented in Linux but
sometimes not! :)

Quote:>Then, in a book on threads (PThreads Primer, available on the net,
>though I forget were...) page 52....

>"What about shared memory?

>Right now you may be asking yourself this question: "What can threads do
>that can't be done by processes ahring memory?"

>The first answer is "nothing." Anything that you can do with threads,
>you can do with processes ahring memory.  Indeed, a number of vendors
>implement a significant portion of thier threads library in roughly this
>fashion.  If you are thinking about using shared memory in this fashion,
>you should make sure you have (a) plenty of time to kill programming,
>(b) plenty more time to kill processing, and (c) plenty of money to burn
>buying RAM.

>You see: (a) debugging gross-process programs is tough, and the tools
>that exist for this are not as good as the ones for MT.  (b) Things take
>longer.  In Solaris, creating a process is about 30 times slower than
>creating a thread, syncronization variables are about 10 times slower,

The reason creating a process is slower than creating a thread is
because of the cloning of resources. In Linux, although a thread
occupies its own process, it is created by a lightweight system call.
Because it shares the address space, and other resources, with its
parent, a Linux thread can be created very fast. Linux threads
are *lightweight* processes.

Secondly, synchronization variables are implemented quite efficiently
in the LinuxThreads library. Locking a mutex does not result in a
system call unless the requesting process must be forced to wait.

What they are talking about above is kernel-level synchronization
mechanisms such as system V semaphores in which every operation
must go through the kernel. This is not true of the LinuxThreads
library (the library API which makes Linux kernel threads look
like POSIX threads).

Quote:>and contex switching about 5 times slower.  (c) Processes eat up a lot
>of kernel memory.  Building a few thousand threads is no big deal.
>Building a few thousand processes is.

>You can do everything with shared memory.  It just won't be as easy or
>run as fast."

This is false. There are resources that cannot be shared via shared
memory, such as file descriptors. In a POSIX threaded program, one
thread can do an open() to create a file descriptor and the other
one can read it. Threads also share all of their memory; one
thread could access another's stack for instance, and they
all see the same static data. Thirdly, a shared
memory segment need not have the same virtual address in every
process, so you can't pass a raw pointer from one process to
the other. In a threaded application, each thread really sees the
same address space.

Quote:>So, now I am wondering, with all the things above stated,...why was it
>decided that a thread should be a new process?  It sounds as though that
>is one of those ugly "Well, it works" hacks.  Though from the look of
>what I have read, it probably doesn't work well...

What have you read which suggests that? I suspect that you have
read something about the disadvantage of using processes,
overlooking the fact that in Linux processes have special
features and optimizations that make efficient threading
feasible without the additional infrastructure of breaking
down a process into threads.

Quote:>Maybe some hard core benchmarking between Linux's kernel-space threads,
>and those of other OSs would better illistrate how Linux threads are
>"better".  That is something I would definately like to see.

Linux *processes* (the heavy kind) are launched and switched faster
than threads in some other operating systems. Do a net search.
 
 
 

Kernel-space Threads

Post by Robert Hyat » Wed, 27 May 1998 04:00:00


: I am really wondering why it was decided to make Linux's Kernel-space
: threads seperate processes.  I see the benchmark at the Linux Thread FAQ
: site, ...

: "Also the scheduler has been
:       optomized so that the switching time for threads vs. tasks varies
: little--about 1.7us (threads) and 1.8us (fork) on a
:       75MHz Pentium."

: This statement itself seems to give me the idea that this decision was a
: bad one...I mean, if a thread in Lunix is actually a process, it is no
: wonder that the 2 are very close in performance.  This doesn't tell me
: that Linux processes are fast, but perhaps Linux threads are slow.

: In that same website is stated:

: "Kernel-space threads often are implemented in the kernel using several
: tables (each task gets a table of threads).
:       In this case, the kernel schedules each thread within the
: timeslice of each process.  "

: granted, my knowlage of the workings of an OS is limited (to say the
: least), but that does not sound like the description of a seperate
: process to me,...even with shared memory.  Why didn't they do it this
: way?

You are misinterpreting... Linux runs one process per thread, period.
All scheduling is done in the kernel, and, from a personal perspective,
this is the best way to do it.  I've run under unicos on the Cray's, and
have cursed the multithreading library for years, because user threads
are not mapped one-to-one to kernel processes...

But in any case, *every* thread you create is a separate process.  If you
try the simple examples from the pthread source, where you do a single
pthread_create() call, and do a "system("ps ax"); from one of the threads,
you'll see two processes, plus a third that handles the thread overhead
for thread_join() stuff and the like...  But they are *all* real kernel
threads...

: Then, in a book on threads (PThreads Primer, available on the net,
: though I forget were...) page 52....

: "What about shared memory?

: Right now you may be asking yourself this question: "What can threads do
: that can't be done by processes ahring memory?"

: The first answer is "nothing." Anything that you can do with threads,
: you can do with processes ahring memory.  Indeed, a number of vendors
: implement a significant portion of thier threads library in roughly this
: fashion.  If you are thinking about using shared memory in this fashion,
: you should make sure you have (a) plenty of time to kill programming,
: (b) plenty more time to kill processing, and (c) plenty of money to burn
: buying RAM.

: You see: (a) debugging gross-process programs is tough, and the tools
: that exist for this are not as good as the ones for MT.  (b) Things take
: longer.  In Solaris, creating a process is about 30 times slower than
: creating a thread, syncronization variables are about 10 times slower,
: and contex switching about 5 times slower.  (c) Processes eat up a lot
: of kernel memory.  Building a few thousand threads is no big deal.
: Building a few thousand processes is.

that is all pretty much opinionated nonsense.  threads have their advantages
in the *everything* is shared.  processes with shared memory also have
advantages...  Neither is overwhelmingly better than the other, different
applications fit each model.  On linux, you can do *either* of course,
as the shmget, shmat and so forth also work, so you can create a shared
memory region, then fork and let both separate processes use that with out
any debugging of other data shared by accident, since there is none...

: You can do everything with shared memory.  It just won't be as easy or
: run as fast."

That makes no sense...  I've done both for many years.  There is no
performance penalty that I have seen, other than algorithmic issues
that affect your programming...

: So, now I am wondering, with all the things above stated,...why was it
: decided that a thread should be a new process?  It sounds as though that
: is one of those ugly "Well, it works" hacks.  Though from the look of
: what I have read, it probably doesn't work well...

: Maybe some hard core benchmarking between Linux's kernel-space threads,
: and those of other OSs would better illistrate how Linux threads are
: "better".  That is something I would definately like to see.

They are as good, or as bad, as threads on any other platform.  They work
perfectly, rapidly, can be created quickly, etc...  The stuff you read is
simply misleading, or you are misinterpreting it badly...

--
Robert Hyatt                    Computer and Information Sciences

(205) 934-2213                  115A Campbell Hall, UAB Station
(205) 934-5473 FAX              Birmingham, AL 35294-1170

 
 
 

Kernel-space Threads

Post by Robert Hyat » Wed, 27 May 1998 04:00:00


:>
:>
:> You are misinterpreting... Linux runs one process per thread, period.
:>

: Actually I think you misinterpreted...I was saying that in the Linux FAQs on
: threads, they say that kernel-space threads are often tables , in other OSs.
: I was asking why Linux's kernel-space threads were not.

:>
:>
:> that is all pretty much opinionated nonsense.  threads have their advantages
:> in the *everything* is shared.  processes with shared memory also have
:> advantages...  Neither is overwhelmingly better than the other, different
:> applications fit each model.

: Of course each has its own benifits.  And some apps are better threaded, some
: are better as seperate processes.  But if you want threads, and get processes
: instead, well then your not getting what you want and using threads instead
: seems rather pointless.

With linux you can get *either*.  A thread under linux fits into the
classic "lightweight process" model...  the entire virtual address space
is shared, ditto for all the proc stuff like file descriptors, and
everything else including the heap.  Threads use the "clone()" system
call that is *very* fast.  Processes use the "fork()" system call which
is also *very* efficient, but not as efficient as clone().

So the difference between "threads" and "processes" is quite vague in
linux, but the way they are used/created are different enough that you
can "tell the difference" if you want.  But if you want "threads" you
get "threads"...  just that each thread is a separate kernel process.

--
Robert Hyatt                    Computer and Information Sciences

(205) 934-2213                  115A Campbell Hall, UAB Station
(205) 934-5473 FAX              Birmingham, AL 35294-1170

 
 
 

Kernel-space Threads

Post by Robert Hyat » Wed, 27 May 1998 04:00:00


: One more thing, what about zombie processes?
: I couldn't find a clear answer to this.
: With each thread creating a new process, when the parent process exits do we
: get zombies, or do all threads exit as threads are supposed to do?

Depends.  If you don't create "detached" threads, then the parent process
must "thread_join()" with each thread.  This  acts sort of like a normal
fork() and wait() facility.

If you create the threads with the "detached" attribute set, when the
thread exits, it goes away because the thread library's "extra" thread
will catch the SIGCHLD and do the wait() so it won't hang as a zombie...

: All I could find on this was question 4.6 in the linux-threads faqs.
: http://linux-rep.fnal.gov/sundocs/faqs/Threads-FAQ/html/

: I know that in threads you can get zombie threads that sit around untill the
: main program exits,...but they shouldn't collect on the system like processes
: can.

If you don't care about "how" a child exits, create detached threads.  If
you do, then don't use detached, and join() to get rid of each zombie as
it finishes its "task"...

--
Robert Hyatt                    Computer and Information Sciences

(205) 934-2213                  115A Campbell Hall, UAB Station
(205) 934-5473 FAX              Birmingham, AL 35294-1170

 
 
 

Kernel-space Threads

Post by Kaz Kylhe » Wed, 27 May 1998 04:00:00



>I know, thats why I asked why they didn't do it this way, it sounds more like
>"threads" then splitting off seperate processes, which have different PIDs and
>everything..even if they do share memory and other various recources...Though
>I also read that in the next major kernel release clone will have the option
>to fork a process that shares the PID of the parent, which may be closer to
>what I am expecting from threads...

There is little utility in that however. What I expect from threads is that
they run concurrently and schedule on multiple processors. What happens
with the PID is of no practical consequence, though it is important for
POSIX compliance.
 
 
 

Kernel-space Threads

Post by Kaz Kylhe » Wed, 27 May 1998 04:00:00



>One more thing, what about zombie processes?
>I couldn't find a clear answer to this.
>With each thread creating a new process, when the parent process exits do we
>get zombies, or do all threads exit as threads are supposed to do?

No; when a parent dies, its children are inherited by the init daemon,
which collects them when they terminate.

Come on, this is just UNIX 101.

In the LinuxThreads library, threads are actually started by a thread manager
process which is itself a thread. When they die, they are collected by that
thread manager, so this issue is transparent to the programmer who just
sees an API that is largely POSIX.

Quote:>All I could find on this was question 4.6 in the linux-threads faqs.
>http://linux-rep.fnal.gov/sundocs/faqs/Threads-FAQ/html/

>I know that in threads you can get zombie threads that sit around untill the
>main program exits,...but they shouldn't collect on the system like processes
>can.

Zombies should never sit around until the main program exists, whether they
are fully flegded processes or light-weight ones. The main program should
collect them with the wait system call. The parent can be asynchronously
notified of a change in the status of the child (including termination)
with the SIGCHLD signal.
 
 
 

Kernel-space Threads

Post by Kaz Kylhe » Wed, 27 May 1998 04:00:00




>: One more thing, what about zombie processes?
>: I couldn't find a clear answer to this.
>: With each thread creating a new process, when the parent process exits do we
>: get zombies, or do all threads exit as threads are supposed to do?

>Depends.  If you don't create "detached" threads, then the parent process
>must "thread_join()" with each thread.  This  acts sort of like a normal
>fork() and wait() facility.

>If you create the threads with the "detached" attribute set, when the
>thread exits, it goes away because the thread library's "extra" thread
>will catch the SIGCHLD and do the wait() so it won't hang as a zombie...

The thread manager will collect *all* threads, including non-detached ones.
Only it can do that because it is the parent of these threads. A
thread calling pthread_join() on another thread is not the parent
of that thread---pthread_join therefore does not correspond to a wait
system call. The detachement semantics are just simulated by the library;
it's not done at the kernel level. Remember, any thread can
join to any other. It's just a synchronization primitive.

The zombies created by failing to do a pthread_join() are not actually
Linux kernel zombies, but merely unreclaimed entries in the
Linuxthread library's data structures. They are library-level zombies.

 
 
 

Kernel-space Threads

Post by Kaz Kylhe » Wed, 27 May 1998 04:00:00



>> You are misinterpreting... Linux runs one process per thread, period.

>Actually I think you misinterpreted...I was saying that in the Linux FAQs on
>threads, they say that kernel-space threads are often tables , in other OSs.
>I was asking why Linux's kernel-space threads were not.

In Linux they are a table: the one big process table. :)

Quote:

>> that is all pretty much opinionated nonsense.  threads have their advantages
>> in the *everything* is shared.  processes with shared memory also have
>> advantages...  Neither is overwhelmingly better than the other, different
>> applications fit each model.

>Of course each has its own benifits.  And some apps are better threaded, some
>are better as seperate processes.  But if you want threads, and get processes
>instead, well then your not getting what you want and using threads instead
>seems rather pointless.

No, you are getting what you want, it just has a name that you don't like,
which is a stupid objection.
 
 
 

Kernel-space Threads

Post by Kaz Kylhe » Wed, 27 May 1998 04:00:00



>But when can we debug them with gdb?

Who cares?

Linux threading is done the way God meant threading to be done.

God doesn't use a de*. :)

I have a C++ library for logging debugging output from a threaded
program. The output from each thread goes into a separate file.
The output is serialized by going through a mutex, and each line
is timestamped and stamped with a counter. From the logs, I can
figure out failure scenarios that are quite complex. I've
discovered subtle race conditions that would probably not have
been apparent had I used an interactive de*.

Interactive de*s disturb the program too much. They are
primarily useful in debugging the sequential logic of an individual thread.
You can do this with GDB if you arrange for your main thread to
execute the code of interest.

 
 
 

Kernel-space Threads

Post by Steve Pel » Wed, 27 May 1998 04:00:00




Quote:>All scheduling is done in the kernel, and, from a personal perspective,
>this is the best way to do it.  I've run under unicos on the Cray's, and
>have cursed the multithreading library for years, because user threads
>are not mapped one-to-one to kernel processes...

There are a few disadvantages to having threads be seen by the kernel as
processes;

a) ps becomes impossible to use if you have a lot of threads (Digital Unix
has a -m option to ps to show threads).

b) although being able to schedule threads at the same level as processes
is often good, sometimes you want to schedule threads as if they're a
single process. Reasons: prevent a single process from taking over the
entire machine; allow threads to set scheduling parameters WITHOUT having
to be root (thus, scheduling parameters only affect how the threads are
scheduled relative to other threads within the process - POSIX allows
for (but does not require) a mechanism for specifying thread scheduling
context).

c) the kernel can treat it as one process where necessary - it can
clean up threads, even if there is a bug in the thread libraries; it
can make sure that the whole process only receives one SIGINT signal,
rather than have 50 threads all try to execute the signal handler.

None of this means that threads shouldn't be scheduled by the kernel,
just that extending clone() to allow them to share the PID is still an
important step.

Digital Unix implemented threads at the kernel level in 3.x. In 4.0, they
"improved" them by making the thread library create kernel threads as
necessary, with a special call-back to the thread manager whenever a
thread blocks in the kernel. All thread-level blocking (mutex, condition
variables, join, etc) is handled in the thread library, and the kernel
level thread is re-assigned to another thread as necessary. In general,
it won't try to create more real threads than there are CPUs.

Unfortunately, that caused several problems:

a) a process that created a bunch of threads with real-time priority,
then changed to a non-root UID, would get errors when new kernel-level
threads were created (and there was no way to determine when one would
be created), as it no longer had privileges to create them real-time;
and anyway, the thread priority was internal to the thread library
anyway; with several processes at different priorities, and threads
that needed to change their kernel-level priority to interact properly,
that caused problems.

b) for a process with very few threads, or where the threads were
usually blocked in the kernel instead of blocked in the thread library,
the overhead actually INCREASED. A process would get a minimum of two
threads even if it never tried to create any user-level threads, simply
by being linked with the thread library. If threads were doing a lot of
work that blocked in the kernel (say, a polling loop every millisecond, or
lots of disk output), the manager thread got woken up every single time.

One thing that is really necessary is kernel level support of semaphores
and condition variables, rather then emulating them with signals. This
would also allow the sem_init() family of semaphore operations to
work between processes (using shared memory), rather than only being
implemented in the thread library.

 
 
 

Kernel-space Threads

Post by Bennett To » Wed, 27 May 1998 04:00:00


1998-05-26-01:02:36 fractture:

Quote:>I was saying that in the Linux FAQs on threads, they say that kernel-space
>threads are often tables, in other OSs. I was asking why Linux's kernel-space
>threads were not.

At some point --- and a linux kernel hacker lives in that neighborhood ---
_everything_ is just tables:-). Unix processes are just tables in the kernel.

The thing that's unusual about Linux is that its process primative isn't so
inefficient and slow as to require completely separate machinery to do fast
threads. Linux's context switches are _fast_. It's process creation is as fast
as practical, and the threads support cuts out the parts of Unix process setup
and teardown that aren't needed, so they're fast, period.

Other Unixes often have somewhat slower process setup and teardown (fixable)
and context switch (not, easily), and so may benefit from a completely
separate implementation of threads in the kernel. Other OSes have been
designed by people who never really outgrew a 1960s image of computing; I
mean, if launching a process requires walking down the hall and feeding a
stack of punch-cards into a machine the size of a small car, then it doesn't
really need to be handled in a bare minimum of instructions, no? Hence the
excruciatingly slow and inefficient process creation/teardown/context switch
of most non-Unix systems today.

-Bennett

 
 
 

1. Linux kernel scheduling of kernel-space thread - question

Hello!

Can anyone explain how linux schedules kernel-space threads created with
clone()?

Do kernel-space threads behave the same as normal linux processes, in terms
of scheduling I mean? Are they scheduled in the same way as normal processes
are and is their scheduling done by the same scheduler?

Does every created kernel-space thread have it's own task_struct created and
added in kernel's process list for scheduling? What is the role of
thread_struct inside task_struct of every process and has it anything to do
with thread scheduling?

Thanx,

Best regards,                        

                                        Mario Zagar                        

2. logchecker fails in 2.6

3. Linux kernel scheduling of kernel-space threads - question

4. PS/2 mouse -- three buttons?

5. are pthreads kernel-space threads?

6. RPC: port mapper failure

7. kernel thread VS user space thread in linux

8. Q: How use xdvi (yes, the right one)

9. Synchronizing user space threads with kernel space in linux

10. threads packages: kernel threads vs. user threads

11. user-space <-> kernel-space

12. Need help in understanding the mapping of user-space send, sendto, sendmsg to kernel-space sendmsg

13. Threads in the kernel space