Very Computer

Hello all,

I've got a basic question:
Would it be possible to kill only the process which consumes the most
memory in the last delta t?
Or does somebody have a better idea?

Background:
rsync seems to have problems when it is break off (by an error or by
CTRL-C). After the break off, rsync eats up all the memory of the
machine until it is killed by the kernel.
Unfortunately, the kernel didn't kill rsync at first, but too late, so
that a lot of other, very important processes were killed before.
E.g.:

Mar 24 11:46:05 susi kernel: Out of Memory: Killed process 732 (kdeinit).
Mar 24 11:46:06 susi kernel: Out of Memory: Killed process 734 (kdeinit).
Mar 24 11:46:06 susi kernel: Out of Memory: Killed process 734 (kdeinit).
Mar 24 11:46:21 susi kernel: Out of Memory: Killed process 704 (konsole).
Mar 24 11:46:29 susi kernel: Out of Memory: Killed process 729 (kdeinit).
Mar 24 11:46:35 susi kernel: Out of Memory: Killed process 738 (kdeinit).
Mar 24 11:46:43 susi kernel: Out of Memory: Killed process 576 (kppp).
Mar 24 11:46:49 susi kernel: Out of Memory: Killed process 268 (squid).
Mar 24 11:46:50 susi squid[266]: Squid Parent: child process 268 exited
due to signal 9
Mar 24 11:46:55 susi squid[266]: Squid Parent: child process 1935 started
Mar 24 11:47:01 susi kernel: Out of Memory: Killed process 700 (ktail).
Mar 24 11:47:08 susi kernel: Out of Memory: Killed process 200 (innd).
Mar 24 11:47:18 susi kernel: Out of Memory: Killed process 214 (httpd).
Mar 24 11:47:24 susi kernel: Out of Memory: Killed process 215 (httpd).
Mar 24 11:47:30 susi kernel: Out of Memory: Killed process 216 (httpd).
Mar 24 11:47:36 susi kernel: Out of Memory: Killed process 217 (httpd).
Mar 24 11:47:42 susi kernel: Out of Memory: Killed process 218 (httpd).
Mar 24 11:47:42 susi kernel: Out of Memory: Killed process 218 (httpd).
Mar 24 11:47:47 susi kernel: Out of Memory: Killed process 266 (squid).
Mar 24 11:47:53 susi kernel: Out of Memory: Killed process 1935 (squid).
Mar 24 11:47:53 susi kernel: Out of Memory: Killed process 1935 (squid).
Mar 24 11:48:03 susi kernel: Out of Memory: Killed process 114 (named).
Mar 24 11:48:03 susi kernel: Out of Memory: Killed process 114 (named).
Mar 24 11:48:12 susi kernel: Out of Memory: Killed process 1936 (httpd).
Mar 24 11:48:12 susi kernel: Out of Memory: Killed process 1936 (httpd).
Mar 24 11:48:17 susi kernel: Out of Memory: Killed process 1937 (httpd).
Mar 24 11:48:17 susi kernel: Out of Memory: Killed process 1937 (httpd).
Mar 24 11:48:22 susi kernel: Out of Memory: Killed process 1939 (httpd).
Mar 24 11:48:22 susi kernel: Out of Memory: Killed process 1939 (httpd).
Mar 24 11:48:27 susi kernel: Out of Memory: Killed process 1938 (httpd).
Mar 24 11:48:27 susi kernel: Out of Memory: Killed process 1938 (httpd).
Mar 24 11:48:33 susi kernel: Out of Memory: Killed process 1940 (httpd).
Mar 24 11:48:33 susi kernel: Out of Memory: Killed process 1940 (httpd).
Mar 24 11:48:40 susi kernel: Out of Memory: Killed process 1941 (httpd).
Mar 24 11:48:40 susi kernel: Out of Memory: Killed process 1941 (httpd).
Mar 24 11:48:44 susi kernel: Out of Memory: Killed process 1942 (httpd).
Mar 24 11:48:44 susi kernel: Out of Memory: Killed process 1942 (httpd).
Mar 24 11:48:50 susi kernel: Out of Memory: Killed process 1943 (httpd).
Mar 24 11:48:55 susi kernel: Out of Memory: Killed process 581 (bash).
Mar 24 11:49:00 susi kernel: Out of Memory: Killed process 1944 (httpd).
Mar 24 11:49:06 susi kernel: Out of Memory: Killed process 923 (rsync).
Mar 24 11:49:06 susi kernel: Out of Memory: Killed process 923 (rsync).
Mar 24 11:49:06 susi kernel: VM: killing process rsync

There could have been killed any other process before the evil-doer is
removed from the memory.

Fortunately, sshd wasn't killed, so I could restart all the other
processes again.

rsync is an actual example for the problem, I wrote. This could be any
other process, eating up the memory. Then, the kernel kills wildly some
processes until the right process is killed - and the machine is
probably unavailable meanwhile.

Regards,
Andreas Hartmann

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Version 2.5.2 and 2.5.4 (I don't know 2.5.3)

Regards,
Andreas Hartmann

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

I can't see any difference.

Quote:

> 2.4 hadn't that kind of problem
> I'm wondering if they'll fix it before I go back to 2.4 :-)

That's right - but there is a security-issue why you shouldn't use
earlier versions.

Regards,
Andreas Hartmann

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

The problem is that 'rsync' might as well have been 'scientific
calculation that ran for 3 days'.

One 'solution' could be to let the OOM killer ignore CPU usage
of less than say 1 hour, but it'll always be heuristics that
can go wrong in some scenario.

regards,

Rik
--
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/             http://distro.conectiva.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

In that particular case, rsync eats up the memory within seconds. Not
only 5 MB's but hundreds of MB's. Wouldn't it be possible to detect such
behaviour and kill such processes, if they consume all the memory very
fast (I mean RAM and SWAP-space), that is, if no more virtual memory is
left, or let's say e.g., if 99% percent of virtual memory are used?

Processes, that consume memory during a long term run cannot be detected
with this heuristic.

Maybe, it would be possible to sort all known processes by there memory
usage and combine it with the speed of their memoryrequests.
If memory gets low, and there is a process, which suddenly requests a
lot of memory, this process get's killed, even if there is another
process, which has three times more memory allocated then the "fast
growing" process. If all processes are growing nearly equal and memory
gets low, the process with the most memory usage get's killed - because
with this process, the kernel achieves the target (to get free memspace)
best.

Advantage of combining consumption-speed and memory usage per process
would be, that processes could be filtered, which are obviously broken.
If the behaviour of the process is correct, than the machine hasn't
enough memory. But this is a problem, which cannot be handled by the kernel.

Regards,
Andreas Hartmann

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

I had a patch for 2.4.something which would allow you to configure which
processes were killed first by the OOM killer. You basically gave
processes an oom_nice value, either by pid or process name, and that was
  taken into account by the oom killer. You could also protect a process
completely from the oom killer, which would be good to do for your sshd
process in the example you give.

Look at http://www.uwsg.iu.edu/hypermail/linux/kernel/0011.1/0453.html

chris

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Quote:> Advantage of combining consumption-speed and memory usage per process
> would be, that processes could be filtered, which are obviously broken.
> If the behaviour of the process is correct, than the machine hasn't
> enough memory. But this is a problem, which cannot be handled by the kernel.

With 2.4.19pre3-ac3+ you don't need a heuristic. Do

        echo "2" >/proc/sys/vm/overcommit_memory

The system will then fail allocations before they can cause an OOM status.
It might be interesting to add "except root" modes to this.

Alan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Quote:> I've got a basic question:
> Would it be possible to kill only the process which consumes the most
> memory in the last delta t?
> Or does somebody have a better idea?

At the point you hit OOM every possible heuristic is simply handwaving that
will work for a subset of the user base. Fix the real problem and it goes
away. My box doesn't OOM, the worst case (which I've never seen happen) is
a task being killed by a stack growth failing to get memory.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

If I would ad the option "except root", I would have the same problem,
because rsync must run as root to do a full backup :-(.

If I understand this feature right, always this process gets a problem,
which want's to have memory even if there is no more free. It could be
the wrong process too.
But if a process gets wild like rsync in this situation, it's very
likely that rsync is the first which doesn't get no more memory. But
what's afterwards if it didn't get any more memory?
If the process, which wants to have so much memory, isn't stopped (or
doesn't stop itself), the memory situation isn't getting better. Other
processes, which are working right, will probably fail while the broken
process eats up the new free mem again.

I think that a broken process like rsync should be killed in order to
prevent other processes to be damaged indirectly.

Regards,
Andreas Hartmann

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Would it hard to do some memory allocation statistics, so if some process
at one point (as rsync did) goes crazy eating all memory, that would be
detected?

I'm quite sure other OSes have similar funcitonality, such as AIX

roy

--
Roy Sigurd Karlsbakk, Datavaktmester

Computers are like air conditioners.
They stop working when you open Windows.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

This would and must be the first solution. I agree with you.

On the other hand - nobody is perfect and there can be such situations.
Why shouldn't the kernel be the ultimate checkpoint to prevent greater
damage? That's what I'm thinking.

It's not easy and it takes probably ressources (processor and RAM) to do
some
checks. The idea would be, to do such checks only when the memory-usage is
over a defined value, e.g. 60% or later. Best would be, if it would be
free configurable (to have the checks at all and at which point beginning).

I suggested a heuristic. Maybe, there are better ones. What I want to
say is, that I think that there should be a mechanism to detect and kill
a process as good as possible, which wants to have all the memory and
even more - before the memory is used to 100%.

Regards,
Andreas Hartmann

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

No.  What I doubt however is whether it would be worth it,
since most machines never run OOM.

regards,

Rik
--
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/             http://distro.conectiva.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Well, I think could be worth in terms of security, because a local user could
use a bad memory-eating program to produce an Denial of Service of other
processes.

Unfortunately detecting a program, written to cause harm is harder than
detecting a crazy program.

greetings

Christian
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

If you don't trust your users you should do some editing
of /etc/security/limits.conf instead of relying on the
safety net in the kernel.

regards,

Rik
--
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/             http://distro.conectiva.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Quote:>Well, I think could be worth in terms of security, because a local user could
>use a bad memory-eating program to produce an Denial of Service of other
>processes.

man setrlimit

Quote:>Unfortunately detecting a program, written to cause harm is harder than
>detecting a crazy program.

ACK.
        Bernd
--

g.a.m.s gmbh                                  Fax : +43 1 205255-900
Prinz-Eugen-Stra?e 8                    A-1040 Vienna/Austria/Europe
                     LUGA : http://www.luga.at

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/