100% CPU utilization - ps reported NO active process!

100% CPU utilization - ps reported NO active process!

Post by Frank Moste » Mon, 16 Apr 2001 05:25:53



Please see environment details below

Perfmeter and sar were reporting 100% CPU utilization, averaging around
75% usr, 25% sys

I ran "ps -ef" to look for the offending process(s), and to my great
surprise, most processes had a zero in the 'C'  column, a couple had a
'1' (the ps process itself),  not a whole lot of CPU activity!
There was also no I/O occuring.
I ran netstat and didn't see a whole lot of network traffic, I unhooked
the network cable to the server just to be sure.

I have since rebooted, and the system is fine.  But this is the 2nd
occurance in the last week.

Any ideas on what else I could look at other than "ps -ef" to see what
is consuming the CPU?
I'm currently installing "top", but this shouldn't show anything if ps
-ef doesn't??

Thank you very much.


Environment:

# uname -a
SunOS <server> 5.7 Generic_106541-04 sun4u sparc SUNW,Ultra-5_10

# /usr/platform/`uname -i`/sbin/prtdiag -v
System Configuration:  Sun Microsystems  sun4u Sun Ultra 5/10 UPA/PCI
(UltraSPARC-IIi 360MHz)
System clock frequency: 90 MHz
Memory size: 256 Megabytes

========================= CPUs =========================

                    Run   Ecache   CPU    CPU
Brd  CPU   Module   MHz     MB    Impl.   Mask
---  ---  -------  -----  ------  ------  ----
 0     0     0      360     0.2   12       9.0

========================= IO Cards =========================

     Bus#  Freq
Brd  Type  MHz   Slot  Name                              Model
---  ----  ----  ----  --------------------------------
----------------------
 0   PCI-1  33     1   ebus
 0   PCI-1  33     1   network-SUNW,hme
 0   PCI-1  33     2   SUNW,m64B                         ATY,GT-C
 0   PCI-1  33     3   ide-pci1095,646

No failures found in System
===========================

========================= HW Revisions =========================

ASIC Revisions:
---------------
Cheerio: ebus Rev 1

System PROM revisions:
----------------------
  OBP 3.19.3 1999/04/07 17:48   POST 3.0.6 1999/04/08 10:56

 
 
 

100% CPU utilization - ps reported NO active process!

Post by Jefferson Ogat » Mon, 16 Apr 2001 05:42:33



> Please see environment details below

> Perfmeter and sar were reporting 100% CPU utilization, averaging around
> 75% usr, 25% sys

> I ran "ps -ef" to look for the offending process(s), and to my great
> surprise, most processes had a zero in the 'C'  column, a couple had a
> '1' (the ps process itself),  not a whole lot of CPU activity!
> There was also no I/O occuring.
> I ran netstat and didn't see a whole lot of network traffic, I unhooked
> the network cable to the server just to be sure.

> I have since rebooted, and the system is fine.  But this is the 2nd
> occurance in the last week.

You may be hacked.

Quote:> Any ideas on what else I could look at other than "ps -ef" to see what
> is consuming the CPU?
> I'm currently installing "top", but this shouldn't show anything if ps
> -ef doesn't??

[snip]

That depends on whether your box was compromised and someone trojaned your ps
and netstat.

Is the box fully patched? If not, is it running any RPC services? Solaris's RPC
vulnerabilities are legendary. cmsd, ttdbserver, sadmind, etc.

You might want to play with The Coroner's Toolkit for a while and see if you
come up with anything interesting. A quick thing to do is check the last
changed time on ps and netstat (ls -lc) and see if they are plausible. Also try
running /usr/proc/bin/ptree and /usr/ucb/ps and see if the results correlate
with what you get from ps. You can nmap your machine from an adjacent box, or
borrow netstat from a machine you know is clean, to check for open ports that
netstat is not showing you.

Take a look at Dave Dittrich's site for good info on forensics. (It's easy to
find.)

--
Jefferson Ogata : Internetworker, Antibozo



 
 
 

100% CPU utilization - ps reported NO active process!

Post by Frank Moste » Tue, 17 Apr 2001 02:02:27


aha!  yes - thanks, looks like someone was kind enough to stop by and pay a visit
on April 7th.
I have a whole bunch of system executables updated, ugh!
I was able to find the offending process.

Using "/usr/ucb/ps -aux" I found a process called in.lpda spinning.
Is this a known trojan?

I needed an excuse to upgrade to Solaris 8 anyway, looks like I'll be reloading
this particular server, and clamping down a lot harder on my internet facing
servers.

I inadvertently left the rpc services open, this won't happen again.  I hate to
ask, but are these solaris rpc vulnerabilities documented anywhere, or is this a
comp.unix.security question?

Thanks again!

Frank



> > Please see environment details below

> > Perfmeter and sar were reporting 100% CPU utilization, averaging around
> > 75% usr, 25% sys

> > I ran "ps -ef" to look for the offending process(s), and to my great
> > surprise, most processes had a zero in the 'C'  column, a couple had a
> > '1' (the ps process itself),  not a whole lot of CPU activity!
> > There was also no I/O occuring.
> > I ran netstat and didn't see a whole lot of network traffic, I unhooked
> > the network cable to the server just to be sure.

> > I have since rebooted, and the system is fine.  But this is the 2nd
> > occurance in the last week.

> You may be hacked.

> > Any ideas on what else I could look at other than "ps -ef" to see what
> > is consuming the CPU?
> > I'm currently installing "top", but this shouldn't show anything if ps
> > -ef doesn't??
> [snip]

> That depends on whether your box was compromised and someone trojaned your ps
> and netstat.

> Is the box fully patched? If not, is it running any RPC services? Solaris's RPC
> vulnerabilities are legendary. cmsd, ttdbserver, sadmind, etc.

> You might want to play with The Coroner's Toolkit for a while and see if you
> come up with anything interesting. A quick thing to do is check the last
> changed time on ps and netstat (ls -lc) and see if they are plausible. Also try
> running /usr/proc/bin/ptree and /usr/ucb/ps and see if the results correlate
> with what you get from ps. You can nmap your machine from an adjacent box, or
> borrow netstat from a machine you know is clean, to check for open ports that
> netstat is not showing you.

> Take a look at Dave Dittrich's site for good info on forensics. (It's easy to
> find.)

> --
> Jefferson Ogata : Internetworker, Antibozo



 
 
 

100% CPU utilization - ps reported NO active process!

Post by Jefferson Ogat » Tue, 17 Apr 2001 03:40:08



> aha!  yes - thanks, looks like someone was kind enough to stop by and pay a visit
> on April 7th.
> I have a whole bunch of system executables updated, ugh!
> I was able to find the offending process.

> Using "/usr/ucb/ps -aux" I found a process called in.lpda spinning.
> Is this a known trojan?

It's pretty common to hide a process with a name like lpd, so it's not much of
a signature on its own. You could search INCIDENTS or Bugtraq
(http://www.securityfocus.com/) for references to that specific process name.
I'm pretty sure I've seen that particular name before, but they all run
together after a while...

From the behavior you described, I'd guess your machine has a network attack
engine on it, possibly part of a Trin00/Stacheldraht or similar network, or a
lone stream attack engine. Bozos use these to overwhelm sites to take them
down, commonly in order to get operator status on an IRC channel by blackholing
the server the current operator is connected to (yes, they hack your machine in
order to play IRC king-of-the-hill, yes, it is that frivolous, bozos with
idleworms). Once the attack is successful, the bozo may be unable shut it off
because it is saturating the network connection on the attacking machine, so
the attack may continue unchecked until the operator notices the slugging
behavior and reboots.

If the in.lpda process is running after boot, it's likely a back door. You can
use lsof to see if it has bound some network ports. Likely a plain /bin/sh back
door, or possibly ssh. Another possibility is that it is a password sniffer.
See if you can find a file called tcp.log, or run strings against the in.lpda
executable to find pathnames. Also run /usr/proc/bin/pmap and
/usr/proc/bin/pfiles against it for additional clues. Look in /etc/inittab, and
all the cron scripts for the trigger that kicks off the back door on boot. Run
tail against all the init scripts as well, especially ones whose last change
time correlates with the intrusion.

You should run forensics on all the Solaris and Linux hosts on your network, at
the very least, to check whether multiple hosts have been compromised. You may
have a lot of work ahead of you, especially if there was a password sniffer on
the box. If you find one, you should also alert any offsite administrators of
hosts your local users like to telnet/rlogin/ftp into, since their credentials
may have been compromised. Don't rely strictly on what you might find in the
sniffer log; bozos clean out the log periodically.

When you reinstall, keep the machine off the network until it is fully patched
and all unnecessary services are shut down, and put tcp wrappers on the
machine. It is quite common for the bozo to return aggressively, and your
machine is vulnerable until it is patched.

Quote:> I needed an excuse to upgrade to Solaris 8 anyway, looks like I'll be reloading
> this particular server, and clamping down a lot harder on my internet facing
> servers.

> I inadvertently left the rpc services open, this won't happen again.  I hate to
> ask, but are these solaris rpc vulnerabilities documented anywhere, or is this a
> comp.unix.security question?

The best place I know to look is in the vulnerability database on
securityfocus.com. By no means restrict your search to RPC vulnerabilities;
these are just the most common on Solaris. In particular they are documented on
the SANS top ten list of threats (http://www.sans.org/topten.htm), and there is
a CERT advisory about them as well.

There are a number of tools for hardening Solaris. I think Casper Dik wrote
one; I think YASSP is another. You can find these in the tools section on
securityfocus.com.

Quote:> Thanks again!

No sweat.

--
Jefferson Ogata : Internetworker, Antibozo


 
 
 

1. New Solaris 8 DHCP server going to 100% cpu utilization and staying there

Hi,

Has anyone out there experienced the aforementioned problem. We have
recently installed the Solaris 8 recommended patch cluster and moved up to
the new 'Enterprise' DHCP server. All is working well but we are having some
problems where the in.dhcpd daemon max's the cpu out and stops responding.
All kernel patches are as per the sun recommendations on sunsolve for the
dhcp server. A truss of the dhcpd process yields absolutely no activity from
the process whatsoever. Anybody have any ideas?

I will get onto sun about it but I thought I would just drop a feeler out
there at the same time!

Cheers,

Scott.

2. Linus SMP / BeOS / NT

3. using second ethernet interface causes 100% CPU utilization

4. Paging, PCS Software for Unix

5. CPU utilization 100% (HELP!!!!!)

6. WORKING: Diskless Linux with 2 MB and No SWAP !!!!!

7. Help Request - CPU Utilization constant 100% !!!!

8. Cut and paste on virtual terms

9. 2.5.18, pdflush 100% cpu utilization

10. Why do top and ps disagree about CPU utilization?

11. HACMP with clsmuxpd process problem with 100% CPU Used.

12. cpu utilization by a specific process with in a given time frame

13. user/sys CPU utilization per process?