inode overflow; c queue max run limit reached

inode overflow; c queue max run limit reached

Post by Bob » Thu, 14 Nov 1996 04:00:00



I've had 2 occurences of the following:

First I receive:

Audit: Collection file inode table overflow
Audit Subsystem termination due to irrecoverable error.

Then about 1 half hour later, something bad happens to cron.  I have a
short job running each minute (does netstat -m to a file); this stops
running; I come in at about 0900 and find the following messages
repeating 6 times each minute:

c queue max run limit reached Wed Nov 13 09:17:00
rescheduling a cron job Wed Nov 13 09:17:00

The date and time are updated to system time.

Using cron tab to stop my 1/minute netstat does not stopp thes messages.

Using ps, I find many job whcih are "/etc/cron" starting about the time
that the last succesful cron job ran, space 1 minute apart for an hour or
two;  similarrly I find a large number of jobs that look like:

root 2794 2793 0 01:57:00 ? 0:00 dlvr_audit 847880520 19 26 2793 cron 6
root 0 run an at/cron session all securi

where the preceding 2 lines are really on long line (and yes, it really
does break off in the middle of the word security).

Any help much appreciated.

This is SCO OSE 3, UNIX 3.2v4.2 via Compaq smartstart.  I've gout two
Pentiums in the box, running SMP, have applied uod393e.

- Bob N.

 
 
 

inode overflow; c queue max run limit reached

Post by Jeff Lieberma » Fri, 15 Nov 1996 04:00:00


: Audit: Collection file inode table overflow
: Audit Subsystem termination due to irrecoverable error.

I would guess that one of your filesystems are out of inodes.
Running news is a great way to have this happen.  Under 3.2v4.2,
you are limited to about 65000 inodes per filesystem.  With
a large number of small files, you can easily run out of inodes
with lots of diskspace left.  Run:
        df -v -i
for the damage report.  For example

Mount Dir  Filesystem           blocks   used   free %used iused ifree %iused
/usr/spool /dev/news            130984  55248  75736  42%  10296  6088   62%

my subminature news filesystem has eaten 62% of the inodes while using
only 42% of the available diskspace.  If left alone, I would run out
of inodes before running out of diskspace.

Another possibility is that your kernel inode table is too small.
Run:
        sar -v
and look at the column labelled "inod-sz".  If there is anything
in the "ov" (overflow) column, dive into the kernel configuration
scripts (/etc/conf/cf.d/configure) and enlarge the inode table.

comix comix 3.2 2 i386    11/14/96
00:00:00 proc-sz ov inod-sz ov file-sz ov lock-sz  
00:15:01  37/100  0 142/550  0 126/550  0   2/100
00:30:01  35/100  0 136/550  0 121/550  0   2/100
00:45:01  38/100  0 138/550  0 125/550  0   2/100

[x] Email to author  [ ] To mailing list  [x] Posted to newsgroup
--
# Jeff Liebermann  Liebermann Design  150 Felker St #D  Santa Cruz  CA  95060

# 408.699.0483 digital_pager    73557,2074  cis [don't]


 
 
 

inode overflow; c queue max run limit reached

Post by Bob » Tue, 19 Nov 1996 04:00:00





:
: : Audit: Collection file inode table overflow
: : Audit Subsystem termination due to irrecoverable error.
:
: I would guess that one of your filesystems are out of inodes.
: Running news is a great way to have this happen.  

My root filesystem has some 50,000 free inodes; all the others have more.

: Another possibility is that your kernel inode table is too small.
: Run:
:       sar -v
: and look at the column labelled "inod-sz".  If there is anything
: in the "ov" (overflow) column, dive into the kernel configuration
: scripts (/etc/conf/cf.d/configure) and enlarge the inode table.

There are only zero entries in the "ov" column.  My inod-sz and file-sz
are nowhere near used up.

Any other clues/ideas/questions?

- Bob N.

 
 
 

inode overflow; c queue max run limit reached

Post by Jeff Lieberma » Tue, 19 Nov 1996 04:00:00


: My root filesystem has some 50,000 free inodes; all the others have more.

: There are only zero entries in the "ov" column.  My inod-sz and file-sz
: are nowhere near used up.

: Any other clues/ideas/questions?

You left out guesses.

1.  Obviously, it's something that's being run from cron.
Could you email to me the entire contents of the:
        /usr/spool/cron/crontabs
directory?  If you have backup copies of edited crontab files
sitting in there, *ALL* of them will be executed including the
backups.  This is a great way to have everything run twice.

2.  Are you running process accounting?

3.  What level of C2 security are you running?  This probably has
nothing to do with the problem but may be screwing up your cron
jobs.

4.  By any chance, do you have multiple cron daemons running
in the process table?

5.  Any clue where the "Audit" program is coming from?

6.  Have you tried running strings on any likely culprits to
see which program is belching the out of inodes error?

7.  Are you by chance running the Compaq Insight Manager (SNMP).
There were a bunch of updates on the Compaq web pile for this.

good luck.
--
# Jeff Liebermann  Liebermann Design  150 Felker St #D  Santa Cruz  CA  95060

# 408.699.0483 digital_pager    73557,2074  cis [don't]

 
 
 

inode overflow; c queue max run limit reached

Post by Bela Lubki » Fri, 22 Nov 1996 04:00:00



> I've had 2 occurences of the following:

> First I receive:

> Audit: Collection file inode table overflow
> Audit Subsystem termination due to irrecoverable error.

These should only occur if you have auditing enabled.  You should have
auditing enabled if you 1) work for a paranoid government agent who, in
the interests of security, want their Pentium Pro systems to run like a
Commodore 64; or 2) are trying to debug a recurring system problem
(break-in or crash), and no other methods seem to work.

So first thing is to make sure you mean to have auditing turned on.

Quote:> Then about 1 half hour later, something bad happens to cron.  I have a
> short job running each minute (does netstat -m to a file); this stops
> running; I come in at about 0900 and find the following messages
> repeating 6 times each minute:

> c queue max run limit reached Wed Nov 13 09:17:00
> rescheduling a cron job Wed Nov 13 09:17:00

> The date and time are updated to system time.

If you don't intend to have auditing turned on, turn it off; end of
problem.  But if you do, you *must* have a policy about what to do when
auditing resources run out.  When the first message appears, you need to
immediately look at the system, figure out what filesystem it's talking
about, and free up space that can be consumed by the auditing data.  You
have committed to giving away great gobs of disk space to auditing.  You
cannot stop feeding the beast.

Auditing probably shut down when some filesystem was nearly full (and
that filesystem was probably /).  After that, other activities continued
to fill that filesystem -- various logs, perhaps mail from your 1/minute
cron job?  Eventually the filesystem filled the rest of the way and
other parts of the system started getting indigestion.

Quote:> Using cron tab to stop my 1/minute netstat does not stopp thes messages.

Not unless you clean up...

Quote:> Using ps, I find many job whcih are "/etc/cron" starting about the time
> that the last succesful cron job ran, space 1 minute apart for an hour or
> two;  similarrly I find a large number of jobs that look like:

> root 2794 2793 0 01:57:00 ? 0:00 dlvr_audit 847880520 19 26 2793 cron 6
> root 0 run an at/cron session all securi

> where the preceding 2 lines are really on long line (and yes, it really
> does break off in the middle of the word security).

Cron sends messages to the auditing subsystem, documenting what
activities cron has initiated.  Apparently this interaction goes awry
when auditing is enabled, but stopped due to running out of space.  Each
job run by cron starts an audit delivery process; those processes hang,
waiting for the auditing subsystem to be ready to receive the messages.

Either turn off auditing or feed it more regularly.

Quote:>Bela<

 
 
 

inode overflow; c queue max run limit reached

Post by James R. Sulliva » Fri, 22 Nov 1996 04:00:00




> > First I receive:

> > Audit: Collection file inode table overflow
> > Audit Subsystem termination due to irrecoverable error.

> These should only occur if you have auditing enabled.  You should have
> auditing enabled if you 1) work for a paranoid government agent who, in
> the interests of security, want their Pentium Pro systems to run like a
> Commodore 64; or 2) are trying to debug a recurring system problem
> (break-in or crash), and no other methods seem to work.

Bob works for a bank, where retired paranoid government employees go to
become paranoid security consultants.  I know of many other paranoid organizations
that have auditing enabled.  Lessons I've learned from these sites:

        Auditing generally collects too much information to really be useful
        and after several gigabytes of data and multiple forests of reports,
        the auditing is turned off.

        Separate, multi-gigabyte drives are the best place to put the Audit
        collection files.

        Paranoid Security Consultants are immune to logic, but would be a good
        person to lock Harry Houdini into a trunk.

----
Jim Sullivan            "Don't plant your bad days.  They grow into bad
SMB Segment Marketing    weeks and then bad months and before you know it

416 216 4611

 
 
 

inode overflow; c queue max run limit reached

Post by Bob » Sat, 23 Nov 1996 04:00:00



:
:       Auditing generally collects too much information to really be useful
:       and after several gigabytes of data and multiple forests of reports,
:       the auditing is turned off.

     This is atrue statement  - the output produced for most reports is
gigantic and sorting thru to get the information is difficult to
impossible.  Is ther any third-party software that can make useable
reports form this stuff?

 
 
 

inode overflow; c queue max run limit reached

Post by Brian Fost » Mon, 25 Nov 1996 04:00:00



 | :    Auditing generally collects too much information to really be useful
 | :    and after several gigabytes of data and multiple forests of reports,
 | :    the auditing is turned off.
 |
 | This is atrue statement  - the output produced for most reports is
 | gigantic and sorting thru to get the information is difficult to
 | impossible.  Is ther any third-party software that can make useable
 | reports form this stuff?

there was at one time.  a company here in the UK (sorry,
I cannot recall their name  ;-(  ) made a filtering tool for
the 3.2v4.2 (and prior?) Auditing report; I'm afraid I cannot
recall the name of the tool either.   ;-(   (fat lot of help
this posting is!)   what I do know is the company demo'ed the
software to SCO here in the UK (and that I broke it with a
then-experimental version of the report generator  <grin> ).
--
"DOS never says `Excellent command or  | Brian Foster, SGS-Thomson, c/o PACT

                   alt.fan.pratchett   | (+44 or 0)117 9707 156     England
                                       | http://www.pact.srf.ac.uk

 
 
 

1. grow inodes: inode-max limit reached error.

I run a small shell service and now after about a week and always after cron
jobs by my users, I'm getting this error (which pretty much makes the
machine not usable): "grow inodes: inode-max limit reached". It looks like a
person's crontab job is opening up WAY too many files or soemthing over the
course of time. How do I prevent this from happening?

Debian 2.2 unstable - 2.2.14+lids+ow2

Thanks

-reid

2. PPP error: proc file system not mounted

3. grow-inodes: inode-max limit reached

4. (UNIX) reading files that another process is writing too

5. "c queue max run limit reached" in cron logs, ???

6. Remote access

7. kernel: grow_inodes: inode-max limit reached

8. Changes to "Welcome to comp.unix.shell" [Frequent posting]

9. kernel error : grow_inodes inodes max limit reached

10. Max Queue Limit Reached

11. cron: 0481-087 The c queue maximum run limit has been reached.

12. cron: b queue max run limit exceeded

13. c queue max run limit