WARNING: /tmp: File system full, swap space limit exceeded

WARNING: /tmp: File system full, swap space limit exceeded

Post by Bill Mosele » Tue, 01 Aug 2000 04:00:00



I would like to follow up on a thread from back in April.
( http://x54.deja.com/[ST_rn=ps]/viewthread.xp?AN=605476002 )

SunOS 5.6 Generic_105181-17 sun4u sparc SUNW,Ultra-Enterprise

The issue in the previous thread was some process was eating up memory
and bringing down the system.  The machine was upgraded from 1G to 2G
not too long ago (since that thread).

I have two follow-up questions:  One about the size of the swap, and the
other about the tmpfs.

1) It was suggested that swap size should be 2.5 - 3 times physical RAM.  
This machine is running with .5 swap (1G swap to 2G physical).  

- I know this is completely dependent on what the machine is doing, but,
in the general case, might using a swap at .5 physcal RAM cause any
problems?  

- Can anyone point to an online document that suggests running a swap at
2.5 to 3 times RAM that I can review?

2) Now, when the machine dies we see this message (lots of them):

  WARNING: /tmp: File system full, swap space limit exceeded

I'm not exactly clear what that message means.  This machine is running
tmpfs, yet (once I can get access to the machine) it doesn't look like
/tmp is that full:

  53) ~ %du -s /tmp
  3360    /tmp

So I'm not sure if really /tmp is full, or if, since /tmp is running as
tmpfs, that a full swap is making it look like /tmp is full.

What seems to bring down the system is an old CGI script that gets hung.  
When I finally got on the machine the other day I found 25 'S' processes
running that script, each using about 58M or memory, and each with a
start time of a few days back.  That's a chunk of memory, though I
didn't check to see how much of that memory was shared.

Now, in the previous thread it was suggested that tmpfs could be
disabled:
  #swap  -      /tmp    tmpfs  -      yes    -

Or to perhaps limit /tmp size:
  swap    -       /tmp    tmpfs   -       yes     -     size=500m

Is that second method to limit /tmp to 500MB to keep /tmp from using up
all memory?

Are both those methods just to prevent some program from using up too
much /tmp and thus using up all memory?

But that wouldn't help in my case where 25 instances of some program
each use up 58M, right?

I know this is a vague questions, but any recommendations on how to
detect and prevent some program(s) from eating up all memory?  Is there
a way to set ulimit(1) on what the web server runs as a CGI?  I don't
really want to limit the server other than the number of child
processes, rather limit the programs it runs.  Or is cron my friend in
looking for run away processes?

BTW -- I don't have much info about the machine when it's running out of
memory.  Soon after the machine recovered I did capture a top that shows
swap low:

last pid:  4112;  load averages:  1.17,  1.09,  1.11                                                        
202 processes: 192 sleeping, 7 zombie, 1 stopped, 2 on cpu
CPU states: 46.6% idle, 50.9% user,  2.4% kernel,  0.1% iowait,  0.0%
swap
Memory: 2048M real, 33M free, 987M swap in use, 14M swap free

Any other general problem solving tips?

PS: For process that have been running for days, how to I see the exact
start time?  ps just shows the date.

Thanks for any help you can provide.

--
Bill Moseley

 
 
 

WARNING: /tmp: File system full, swap space limit exceeded

Post by Logan Sh » Tue, 01 Aug 2000 04:00:00




Quote:>1) It was suggested that swap size should be 2.5 - 3 times physical RAM.  
>This machine is running with .5 swap (1G swap to 2G physical).  

>- I know this is completely dependent on what the machine is doing, but,
>in the general case, might using a swap at .5 physcal RAM cause any
>problems?  

It might cause you to run out of available virtual memory.  That would
be the only problem I know of.  It does seem like a little bit of a
waste, though, since generally you want to run your system near its
capacity, which means you will be flirting with the line between having
enough RAM and running out.  If you have no swap space or not much swap
space configured, then flirting with this line is dangerous.  So
basically, swap space is there to (a) allow you to process jobs slowly
that you otherwise wouldn't have been able to process at all, and (b)
flirt with the line so you get maximum return on your investment.

Quote:>2) Now, when the machine dies we see this message (lots of them):

>  WARNING: /tmp: File system full, swap space limit exceeded

>I'm not exactly clear what that message means.

It means the filesystem is full.  The tmpfs filesystem draws on the
virtual memory of the system to fulfill its requirements to store
data.  If the system is running low on virtual memory, there will be no
place to put files that need to go into a tmpfs filesystem.  Thus, the
filesystem is full.

Note that it's also possible to set an artificial limit on the size of
a tmpfs filesystem with a mount option.  Do a "man mount_tmpfs" for
information on that.

Quote:> This machine is running
>tmpfs, yet (once I can get access to the machine) it doesn't look like
>/tmp is that full:

>  53) ~ %du -s /tmp
>  3360    /tmp

It's not whether /tmp is full, it's whether virtual memory is used up.
It's better to do "swap -s" and/or "df -k /tmp" to find out how much
free space is available.

Quote:>So I'm not sure if really /tmp is full, or if, since /tmp is running as
>tmpfs, that a full swap is making it look like /tmp is full.

It is impossible to have a full swap without /tmp being full.

Quote:>What seems to bring down the system is an old CGI script that gets hung.  
>When I finally got on the machine the other day I found 25 'S' processes
>running that script, each using about 58M or memory, and each with a
>start time of a few days back.

I suggest fixing that CGI script.  If you can't fix it, or if you can't
fix it immediately and you need some sort of stop-gap measure, I
suggest writing another script that will run "ps" and determine if
there are any CGI scripts running that are more than 30 minutes old,
and if so, kill them.

Quote:>Is that second method to limit /tmp to 500MB to keep /tmp from using up
>all memory?

Yes.

Quote:>PS: For process that have been running for days, how to I see the exact
>start time?  ps just shows the date.

You can use "ps -e -o pid,etime,comm" or something similar.  It doesn't
tell you what time it started though, just how long it has been running.

  - Logan

 
 
 

WARNING: /tmp: File system full, swap space limit exceeded

Post by Casper H.S. Dik - Network Security Engine » Wed, 02 Aug 2000 04:00:00


[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]


>1) It was suggested that swap size should be 2.5 - 3 times physical RAM.  
>This machine is running with .5 swap (1G swap to 2G physical).  

Depends only on your workload.

Quote:>- I know this is completely dependent on what the machine is doing, but,
>in the general case, might using a swap at .5 physcal RAM cause any
>problems?  

No.  "You need as much swap as you need".

Quote:>- Can anyone point to an online document that suggests running a swap at
>2.5 to 3 times RAM that I can review?

There are none; there is no good rule of thumb.  Solaris runs without
swap (but that's not as efficient as all anonymous pages are locked
in memory, including reserved by as yet unused ones)

Quote:>2) Now, when the machine dies we see this message (lots of them):
>  WARNING: /tmp: File system full, swap space limit exceeded
>I'm not exactly clear what that message means.  This machine is running
>tmpfs, yet (once I can get access to the machine) it doesn't look like
>/tmp is that full:

It means that:

        - something was writing to /tmp
        - swap was full

basically, one of two thing (or both) happened:

        - programs used all available memory
        - someone wrote too much to /tmp

Quote:>  53) ~ %du -s /tmp
>  3360    /tmp

Looks like out of swap then.  If swap is full nothing more will fit in
/tmp

Quote:>  swap    -       /tmp    tmpfs   -       yes     - size=500m
>Is that second method to limit /tmp to 500MB to keep /tmp from using up
>all memory?

Yes, it will prevent /tmp use from overflowing swap.

Disabling /tmp as tmpfs will allow overflowing the root filesystem which
might be worse.

Quote:>Are both those methods just to prevent some program from using up too
>much /tmp and thus using up all memory?

Your problem doesn't appear to be related to /tmp at all, rather it's related
to swap use by hangign programs.

Quote:>But that wouldn't help in my case where 25 instances of some program
>each use up 58M, right?

Correct.

Quote:>I know this is a vague questions, but any recommendations on how to
>detect and prevent some program(s) from eating up all memory?  Is there
>a way to set ulimit(1) on what the web server runs as a CGI?  I don't
>really want to limit the server other than the number of child
>processes, rather limit the programs it runs.  Or is cron my friend in
>looking for run away processes?

No, there are no per process virtual memory limits in Solaris

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

1. WARNING: /tmp: File system full, swap space limit exceeded

[previously posted to comp.unix.misc by mistake -- sorry]

SunOS 5.6 Generic_105181-17 sun4u sparc SUNW,Ultra-Enterprise

I start seeing messages that, for example, httpd processes can't fork
and my messages file contains a number of these messages:

   WARNING: /tmp: File system full, swap space limit exceeded

The machine has 1GB RAM and 1GB swap.  I think that's too small of a
swap space, for one thing.  More RAM would be good, but I'm wondering
about the swap size.

What would be "the" recommended size of the swap partition and can
anyone point to any references on how to determine swap size?

Thanks,

--

2. Oberon/F where?

3. plenty of swap space, but keep receiving error "swap space limit exceeded"

4. KWINTV

5. swap space problem, /tmp becomes 100 % full

6. wget & SSL

7. /tmp full, swap full?

8. java servlet on Apache 1.3.6

9. Mac TCP causes inetd to dump in /tmp space -> SYSTEM IS FULL message

10. 5.6 x86: file system full message, but file system not full!

11. /tmp Swap space running out of space

12. /tmp Filesystem fulll, swap exceeded

13. "Hard limit exceeded on file systems"