System intrusion detection, primarily on linux servers with a handful of others

System intrusion detection, primarily on linux servers with a handful of others

Post by Damon Getsma » Fri, 16 Jan 2009 13:05:08



Okay, we've got some serious issues going on here at work.  We've lost
the dude that knew everything about how our network was set up; myself
and my cow orker are working full time on recovering all of the
information that we can and now we've got the added task of hardening
our systems from outside attack.

Right now I need to know if any of you *NIX networking ppl and/or
sysadmins can possibly recommend an intrusion detection system/suite.
I've heard that 'snort' is the de facto standard.  We're administering
a handful of windows machines, a load of distributed Linux servers
over a WAN with 3 gateways to the outside world + a T1, 2 Linux
asterisk PBX servers, and some laptops that occasionally VPN in.  I'd
at the very least like to be able to monitor all of our linux systems
in a centralized manner.

To this point I've been utilizing nagios, mrtg, and ntop.  As our
potential cracker realizes how we've identified him in bandwidth
spikes, identifying compromised binaries by modification dates, and
top on any potentially compromised machines to watch for processes
spawned.

I'm adding syslogging of login attempts, etc, but the dude that got
into our WAN is sophisticated.  He'll be covering his tracks better in
no time.  And honestly, keeping a tail -f /var/log/messages open on
each server and my eyeballs glued to bandwidth graphs updating at 5
minute intervals will not be an option much longer.

Anybody got any ideas and/or need more information to be able to give
some helpful hints?

TIA

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by terry » Fri, 16 Jan 2009 14:29:58



> Okay, we've got some serious issues going on here at work.  We've lost
> the dude that knew everything about how our network was set up; myself
> and my cow orker are working full time on recovering all of the
> information that we can and now we've got the added task of hardening
> our systems from outside attack.

I would look firstly at your firewall. Remove EVERYTHING that is not
strickly necessary and fire the first arsehole that tries to change it.

Unless, it was done from inside, they must have come in through the
firewall. Most likely they know a password, or through known
vulnerabilities through all that junk that IT like to reward
themselves with (ICQ, skype, etc, etc).

Then you put a monitoring application on the firewall to see what probes
it is receiving from outside.

Which one depends on your skill and support contracts.

Quote:> I'd at the very
> least like to be able to monitor all of our linux systems in a
> centralized manner.

The monitoring you would most likely do is uptime/services related
related.

You have a major security inadequacy if your external cracker has been
able to reach individual systems inside your network.

Quote:

> To this point I've been utilizing nagios, mrtg, and ntop.  As our
> potential cracker realizes how we've identified him in bandwidth spikes,
> identifying compromised binaries by modification dates, and top on any
> potentially compromised machines to watch for processes spawned.

Do not recover the old system from backup tapes unless you adsolutely have
to. The real problem is you will not know when they first compromised your
system.

It may take longer, but you are far better to reinstall the system from
installation media, and then re-install/reconfigure the applications. You
are going to have to learn it anyway if you are going to support it.

Quote:

> I'm adding syslogging of login attempts, etc, but the dude that got into
> our WAN is sophisticated.

how is he getting in?
That is where you should be lookng first and fixing it.

Quote:> He'll be covering his tracks better in no
> time.  And honestly, keeping a tail -f /var/log/messages open on each
> server and my eyeballs glued to bandwidth graphs updating at 5 minute
> intervals will not be an option much longer.

Totally useless.
I'm curious as to what extra bandwidth indicated?
Are they trying to steal your data?

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by Damon Getsma » Fri, 16 Jan 2009 15:05:49



Quote:> I would look firstly at your firewall. Remove EVERYTHING that is not
> strickly necessary and fire the first arsehole that tries to change it.

That's precisely what we're starting with at the moment.  It's been a
little bit difficult to convince everyone of that as we've [very]
recently had a change of management around here, and we're working
with another company providing a service, but we're starting to
implement these very changes at the moment.

Quote:> Unless, it was done from inside, they must have come in through the
> firewall. Most likely they know a password, or through known
> vulnerabilities through all that junk that IT like to reward
> themselves with (ICQ, skype, etc, etc).

I'm pretty sure that the vulnerability was a 'webmin' package that
shouldn't have even been installed on a system that myself and my
colleage were denied access to until this point.  We're now whittling
down the services on the new version of this machine that we're
installing to make sure that any services like this are gone.  It took
a lot of convincing with the other people associated with the machine
that provided the gateway in this instance to get them to trim down
their distributions, but they're complying fully with our security
recommendations and, in fact, having us implement them, from this
point forward.

Quote:> Then you put a monitoring application on the firewall to see what probes
> it is receiving from outside.

> Which one depends on your skill and support contracts.

Looks like we're going to be going with snort right now, and seeing
what that gives us at first.

Quote:> > I'd at the very
> > least like to be able to monitor all of our linux systems in a
> > centralized manner.

> The monitoring you would most likely do is uptime/services related
> related.

> You have a major security inadequacy if your external cracker has been
> able to reach individual systems inside your network.

I was worried that this happened at first, but it's beginning to look
like maybe we got lucky about that.  However our DMZ that the
'gateway' machine should have been in was not correctly implemented
and we're locking that down as I type.

Quote:> how is he getting in?
> That is where you should be lookng first and fixing it.

Yep, that's what we're concentrating on at this point.

Quote:> Totally useless.
> I'm curious as to what extra bandwidth indicated?

Our 'gateway' machine that was definitely compromised showed at the
point of 'recruitment' of this server that our bandwidth spiked up to
around 3-4 times our normal PEAK bandwidth usage on that machine.  It
is normally used for a high bandwidth service that only consumes a
small fraction, even at highest consumption, of what it began to use
at the minute that we detected his actions.  We have logs of bandwidth
going back almost a year so we were able to identify this as
suspicious activity immediately.  Obviously relying on logs and
bandwidth is not enough, as you said, and that's why (now that the
person that was previously holding us back is no longer a problem)
we're now implementing much more comprehensive strategies.

Not sure if any of our data was compromised.  It looks a lot more like
whoever was trying to get in was looking for a machine with an assload
of bandwidth and minimal CPU/RAM requirements.  It was a linux
machine, though, and they were using modified system binaries to
establish back doors, so I'm thinking it was something probably more
malicious than just a spambot.  Multiple connections were established
(presumably generating the bandwidth spikes) to different IRC
servers.  Botnet is my guess.

-Damon

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by Moe Tr » Sat, 17 Jan 2009 04:48:01


On Wed, 14 Jan 2009, in the Usenet newsgroup comp.os.linux.networking in

and in the Usenet newsgroup comp.os.linux.security, in article

NOTE: Posting from groups.google.com (or some web-forums) dramatically
reduces the chance of your post being seen.  Find a real news server.

NOTE: Please don't multipost the same message to multiple groups. If
you MUST post to multiple groups, include them all in the Newsgroups:
header, and set a Followup-To: header as I have done here.

Quote:>As our potential cracker realizes how we've identified him in
>bandwidth spikes, identifying compromised binaries by modification
>dates, and top on any potentially compromised machines to watch for
>processes spawned.

Are you indicating that someone already _HAS_ gotten in?  If so, you
are screwed - as you don't know what that person has done. You mention
identifying compromised binaries - that implies the person has root -
and think you can identify things by modification date.  Boy, are you
ever wrong:

[ibuprofin ~]$ ls FOO
ls: FOO: No such file or directory
[ibuprofin ~]$ touch FOO
[ibuprofin ~]$ ls -l FOO
-rw-r--r--   1 ibuprofi users           0 Jan 15 12:28 FOO
[ibuprofin ~]$ touch -t 102005152001.13 FOO
[ibuprofin ~]$ ls -l --full-time FOO
-rw-r--r--   1 ibuprofi users           0 Sat Oct 20 05:15:13 2001 FOO
[ibuprofin ~]$

Quote:>I'm adding syslogging of login attempts, etc, but the dude that got
>into our WAN is sophisticated. <A0>He'll be covering his tracks better
>in no time.

And the bad guy is getting in exactly how?   You're posting from an
address of a company in the medical business. If there is even the
remote possibility that this is subject to HIPAA, disconnect the network
from the Internet RIGHT NOW.

Quote:>And honestly, keeping a tail -f /var/log/messages open on each server
>and my eyeballs glued to bandwidth graphs updating at 5 minute
>intervals will not be an option much longer.

as it's a waste of time if the bad guy has gotten in once already. How
do you know that the tools you are using haven't been 0wn3d already?

You're posting from what is indicating Ubunto 8.04.  If that is
representative of your Linux, have you been keeping things up to date?
If so, the bad guy may be getting in through a compromised account.
Which one?

In the Usenet newsgroup comp.os.linux.security, Bit Twister has
suggested aide and samhain (and ossec which I have no experience with).
A problem with these tools is that they are best used starting with a
"clean" system, and allowing them to report on changes.  In both Usenet
newsgroups, others have recommended a reinstall. I TOTALLY agree with
that recommendation. If you believe you've been compromised, the _only_
solution is a wipe and reinstall, followed by a full update before
allowing access from the world.

Hmmm, I wonder if your SSH setup is one of the compromised ones. From
the US-CERT Technical * Security Alert TA08-137A from last May:

   Overview

   A  vulnerability  in  the  OpenSSL  package  included  with the Debian
   GNU/Linux   operating  system  and  its  derivatives  may  cause  weak
   cryptographic keys to be generated. Any package that uses the affected
   version of SSL could be vulnerable.

   I. Description

   A  vulnerabiliity  exists  in  the random number generator used by the
   OpenSSL  package included with the Debian GNU/Linux, Ubuntu, and other
   Debian-based   operating   systems.   This  vulnerability  causes  the
   generated numbers to be predictable.

That was patched back in May, but any keys generated before then are
vulnerable.    There are exploits of this in the wild.

        Old guy

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by 1PW » Sat, 17 Jan 2009 05:49:19


On 01/14/2009 08:05 PM, Damon Getsman sent:

              *   *   *   Cross Posted   *   *   *

Snip, snip...

Hello Damon:

Please consider crossposting all future replies to this thread so we
aren't burdened with duplicitous posts at cross purposes.

           <http://en.wikipedia.org/wiki/Crossposting>

I know that once you send a post, another newsgroup comes to mind.  All
of us have done it.

Looking forward to your reply,

Pete
--

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by Damon Getsma » Sun, 18 Jan 2009 05:09:26


Quote:> NOTE: Posting from groups.google.com (or some web-forums) dramatically
> reduces the chance of your post being seen. ?Find a real news server.

Thank you for this information; however, the ISPs that we are using
for the gateway that I'm forced to use at this point prevent me from
having access to a decent raw nntp server AFAIK.  Once I'm not swamped
with other things I'll check this out.

Quote:> NOTE: Please don't multipost the same message to multiple groups. If
> you MUST post to multiple groups, include them all in the Newsgroups:
> header, and set a Followup-To: header as I have done here.

I normally do.  It was a mistake on my part.

Quote:> Are you indicating that someone already _HAS_ gotten in? ?If so, you
> are screwed - as you don't know what that person has done. You mention
> identifying compromised binaries - that implies the person has root -
> and think you can identify things by modification date. ?Boy, are you
> ever wrong:

I am aware of that, and yes, I'm pretty sure that he did get root.
The modification stamps on /bin/bash and /bin/grep didn't get there by
anyone else and I didn't do it personally.  It does tell a little bit
about the sophistication level of the person that I'm dealing with,
though.

I am fully aware of the ability to modify datestamps by 'touch' and
hex filesystem access on a low level.  Obviously it either indicates
that the person thought we were incompetent (or did not have enough
resources available as a target) enough to not bother with the time
for that, or they do not posses the level of sophistication to
implement that.

Quote:> And the bad guy is getting in exactly how? ? You're posting from an
> address of a company in the medical business. If there is even the
> remote possibility that this is subject to HIPAA, disconnect the network
> from the Internet RIGHT NOW.

Almost certainly looks to be from an apache or webmin hole.  I
attempted to close these things a long time ago and was hamstrung by
someone over my head that has been dealt with for this already.  I'm
making sure that I don't make the same mistakes again.

Quote:> as it's a waste of time if the bad guy has gotten in once already. How
> do you know that the tools you are using haven't been 0wn3d already?

I know, that's why I'm looking for something better on the next way
around.

Quote:> You're posting from what is indicating Ubunto 8.04. ?If that is
> representative of your Linux, have you been keeping things up to date?
> If so, the bad guy may be getting in through a compromised account.
> Which one?

We haven't been (AGAINST my advice, which is why the aforementioned
people are being 'dealt with'), and if he's got a brain he's snagged
our password and shadow files, so we're considering everything
compromised.  I've dealt with network security from the other side of
the spectrum when I was an angsty *ager and learned how
exploitations occur.  So I may not have the knowledge to deal with
this on the administration side just yet, but I do know what I'm
looking for, what he's capable of, and have an idea of what we're
playing with.

Quote:> In the Usenet newsgroup comp.os.linux.security, Bit Twister has
> suggested aide and samhain (and ossec which I have no experience with).
> A problem with these tools is that they are best used starting with a
> "clean" system, and allowing them to report on changes. ?In both Usenet
> newsgroups, others have recommended a reinstall. I TOTALLY agree with
> that recommendation. If you believe you've been compromised, the _only_
> solution is a wipe and reinstall, followed by a full update before
> allowing access from the world.

Thank you for the advice.  I'll look into the tools.  I'm doing what I
can to advocate what you've mentioned, but I might be hamstrung from
higher up in being able to do a full cleaning.

Quote:> Hmmm, I wonder if your SSH setup is one of the compromised ones. From
> the US-CERT Technical * Security Alert TA08-137A from last May:

On the old machine it certainly was.  I was aware of this ssl
vulnerability but didn't think of it because I didn't know about
Asterisk's usage of SSL.  I'll look into it more now.  As soon as that
issue was mentioned I made sure to update all of our servers to non-
vulnerable implementation and keyset.

Thank you much, I appreciate the time and consideration that you put
into your message a lot.  I don't have a whole lot of resources to
work with right now and a pointing in the right direction saves me a
lot of time and enables me to do what I can over the weekend when all
of our users are off of the main servers, so quick action is a
necessity.

-D

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by Damon Getsma » Sun, 18 Jan 2009 05:10:26


Gotcha, and I thank you much for bringing it to my attention.  That's
exactly what happened, I'll try to think a little bit more before I
smack send next time.  :)


Quote:> Please consider crossposting all future replies to this thread so we
> aren't burdened with duplicitous posts at cross purposes.

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by Moe Tr » Mon, 19 Jan 2009 03:55:38


On Fri, 16 Jan 2009, in the Usenet newsgroups comp.os.linux.networking and
comp.os.linux.security, in article

NOTE: Posting from groups.google.com (or some web-forums) dramatically
reduces the chance of your post being seen.  Find a real news server.

Quote:>Once I'm not swamped with other things I'll check this out.

Understood - try http://www.veryComputer.com/
when you have time. The 'NOTE:' is added automatically by my news tool
when I'm replying to a googlegroups post.

Quote:>> Are you indicating that someone already _HAS_ gotten in?   If so, you
>> are screwed - as you don't know what that person has done.
>I am aware of that, and yes, I'm pretty sure that he did get root.

The only safe solution is to wipe and reinstall from known clean media.
But you know that too.

Quote:>The modification stamps on /bin/bash and /bin/grep didn't get there by
>anyone else and I didn't do it personally.  It does tell a little bit
>about the sophistication level of the person that I'm dealing with,
>though.

For informational use only - look at your package manager. 'rpm' has a
'-Va' option, and Debian based systems have 'debsums'. Both will do a
md5sum check against a baseline (that may or may not be compromised).
Neither is good enough to use as a IDS, and the 'debsums' man page even
warns about that.  Something as simple as a MD5 hash isn't solid, given
that the /usr/bin/md5sum may be ``fixed'' and the database itself
modified if it's accessible.   See below.

Quote:>Obviously it either indicates that the person thought we were
>incompetent (or did not have enough resources available as a target)
>enough to not bother with the time for that, or they do not posses the
>level of sophistication to implement that.

That may be true, but I don't think you can risk it either.

Quote:>> And the bad guy is getting in exactly how?
>Almost certainly looks to be from an apache or webmin hole.

Yeah, there are plenty of those. Two points: PHP is generally a walking
disaster area - looking at Bugtraq shows more PHP exploits than
anything else, followed fairly closely by SQL.

Second point, if one of your users has been 0wn3d _elsewhere_ and they
had their SSH authentication tokens on "that" system, there is an
exploit out there that uses stolen authentication to get "in" to your
system, uses any number of local exploits to escalate privilege to
root, and then installs the 'Phalanx2' root kit that hides itself
pretty well, and searches for authentication data on your system (which
it mails out to a drop-box).  CERT reported this about five months ago.

Quote:>I've dealt with network security from the other side of the spectrum
>when I was an angsty *ager and learned how exploitations occur.  So
>I may not have the knowledge to deal with this on the administration
>side just yet, but I do know what I'm looking for, what he's capable
>of, and have an idea of what we're playing with.

The only safe solution is the wipe/reinstall/update.  There are a
number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
(http://www.veryComputer.com/), 'rkhunter' (http://www.veryComputer.com/), and
from a cursory look, 'ossec' (http://www.veryComputer.com/) that can be used
to look for many rootkits.  By in large, these are trivial to defeat or
bypass.  As one example, all look for the existence of a file named
'/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
sniffer worm from 2003).  That's all well and good, but what if the
mal-ware author does something terribly complicated, such as changing
the filename to '/tmp/.../A' or '/tmp/.../b' or something...  yup, the
anti-mal-ware tools won't find it.

Quote:>> In the Usenet newsgroup comp.os.linux.security, Bit Twister has
>> suggested aide and samhain (and ossec which I have no experience
>> with).  A problem with these tools is that they are best used
>> starting with a "clean" system, and allowing them to report on
>> changes.

Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
strong point, but I'm much less thrilled by it now.

Quote:>Thank you for the advice.  I'll look into the tools.  I'm doing what I
>can to advocate what you've mentioned, but I might be hamstrung from
>higher up in being able to do a full cleaning.

Many of us suffer through the same interference.  Best advice remains
a wipe, reinstall from trusted media, and updates.  In this case, I'd
also strongly recommend new passwords for ALL accounts.  Then, take a
snapshot of the system using 'aide' (or the fore-runner 'tripwire').
This is what the system looks like "now" - and you can then compare
this snapshot to "later" and look for changes. Big caution: the
aide binary and database go on removable media that is kept in a safe
place when not being used. Also, you have to retake the snapshot each
time you intentionally change something (such as security updates). It
is a pain in the a$$, but less of a pain than what you are going
through now.

Quote:>> Hmmm, I wonder if your SSH setup is one of the compromised ones. From
> the US-CERT Technical * Security Alert TA08-137A from last May:
>On the old machine it certainly was.  I was aware of this ssl
>vulnerability but didn't think of it because I didn't know about
>Asterisk's usage of SSL.  I'll look into it more now.

If you are running a "current" distribution, then a lot of this is
taken care of by the distribution update process. What it doesn't
handle is an authentication token weakness, such as the above.

Quote:>I don't have a whole lot of resources to work with right now and a
>pointing in the right direction saves me a lot of time and enables me
>to do what I can over the weekend when all of our users are off of
>the main servers, so quick action is a necessity.

One further recommendation is the firewall. As of two days ago, there
were 2,771,249,848 IPv4 addresses in use in the world. You may want to
look very strongly at restricting access to all services (except
perhaps mail, DNS, and the web server) to as small a number of address
ranges as practical. For example, you mentioned webmin (and are not
likely to reinstall that again, but) you are probably not to likely
to be needing access to that service from (not pointing fingers at
them, but using them as an example) Central/South America.  That means
you could probably NOT ALLOW (as opposed to blocking) IPv4 addresses
in the 186.x.x.x/7, 189.x.x.x/8, 190.x.x.x/8 and 200.x.x.x/7 ranges.
The same is true for Asia, Europe, and Africa.  Now _blocking_ is
difficult (there are some 93,000 networks in the world, and how do
you know which is which), but _allowing_ access from specific ranges
might prove more practical. Case in point, my home firewall allows
access from a /22 and two /24s "outside" because I can't see any
reason to allow connections from you or anyone else that I haven't
approved in advance, and I really don't expect authorized users to be
connecting from Korea, Kenya, Kuwait or Kazakhstan or a lot of other
places either.  If access isn't specifically allowed, then the packet
hits the default rule, and is blocked.  Much simpler (and safer) than
trying to exclude every address you believe is "bad".

        Old guy

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by 1PW » Mon, 19 Jan 2009 07:04:55


Hello Damon:

Exactly which Linux OSs are you supporting on these servers?
Distro, version...

--

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by dgets.. » Wed, 21 Jan 2009 07:08:10


Ubuntu 8.04, OpenSuSE 10.4, and then I'm working on getting us back onto
a few Solaris systems right now.

Ubuntu 8.04 is definitely our primary; the only different version of
Ubuntu that we have is on a few of our IT personnel personal
workstations, in which case we're using 8.10.


Quote:

> Hello Damon:

> Exactly which Linux OSs are you supporting on these servers?
> Distro, version...

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by dgets.. » Fri, 23 Jan 2009 00:11:46



Quote:> NOTE: Posting from groups.google.com (or some web-forums) dramatically
> reduces the chance of your post being seen.  Find a real news server.

Obviously I've gotten to that step now, something I had been neglecting
previously.  Thanks.

Quote:>>> Are you indicating that someone already _HAS_ gotten in?   If so, you
>>> are screwed - as you don't know what that person has done.

>>I am aware of that, and yes, I'm pretty sure that he did get root.

At this stage I know that he got root on one of our externally facing
machines.  This machine has been wiped and reinstalled but is still not
up to the standards that I would like it to be at.  It also had an
internal interface which was not securely firewalled or DMZed.  Yes, I
understand the horror of that situation, but it was out of my control.
Thanks to that issue I now have the ability to make suggestions that
will be taken much more seriously regarding our handling of network
security in the future.  This does not help the fact that I'm still
learning this as I go, however, at least at this level of
sophistication.

Quote:> The only safe solution is to wipe and reinstall from known clean media.
> But you know that too.

As a starting point I have wiped and reinstalled my own workstation; I
am not totally sure that I had it locked down before I reconnected it to
the WAN, however, so I may just end up doing that again.  I am also
using a startpoint of my own personal laptop which, at this point, is
disconnected from the WAN and I am fairly certain is not compromised at
the moment.  It is running a significantly different kernel from the
other machines, as well.

Quote:>>Almost certainly looks to be from an apache or webmin hole.

> Yeah, there are plenty of those. Two points: PHP is generally a walking
> disaster area - looking at Bugtraq shows more PHP exploits than
> anything else, followed fairly closely by SQL.

Thank you.  As we get downtime where our users are not needing vital
network services I will make sure that we concentrate on these areas and
not providing them where they aren't needed, and maintaining the best I
can find to lock them down where they absolutely are needed.

Quote:> Second point, if one of your users has been 0wn3d _elsewhere_ and they
> had their SSH authentication tokens on "that" system, there is an
> exploit out there that uses stolen authentication to get "in" to your
> system, uses any number of local exploits to escalate privilege to
> root, and then installs the 'Phalanx2' root kit that hides itself
> pretty well, and searches for authentication data on your system (which
> it mails out to a drop-box).  CERT reported this about five months ago.

Yep.  I remember similar .rhost exploits and disasters surrounding them
over a decade ago.

Quote:> The only safe solution is the wipe/reinstall/update.  There are a
> number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
> (http://www.chkrootkit.org), 'rkhunter' (http://www.rootkit.nl), and
> from a cursory look, 'ossec' (http://www.ossec.net/) that can be used
> to look for many rootkits.  By in large, these are trivial to defeat or
> bypass.  As one example, all look for the existence of a file named
> '/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
> sniffer worm from 2003).  That's all well and good, but what if the
> mal-ware author does something terribly complicated, such as changing
> the filename to '/tmp/.../A' or '/tmp/.../b' or something...  yup, the
> anti-mal-ware tools won't find it.

I've actually been able to start going through some OSSEC logs at this
point that are showing things of this sort.  Of course, I do not know
enough about the specifics of OSSEC administration just yet to be able
to eliminate all of the false positives, however there are a few entries
in them that are leading me to believe that we have significant
intrusion on some if not all of our primary internal servers within the
WAN.  IE:

OSSEC HIDS Notification.
2009 Jan 20 15:29:47

Received From: (agentname) 192.168.1.NOU->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event
(rootcheck)."
Portion of the log(s):

Port '41861'(tcp) hidden. Kernel-level rootkit or trojaned version of
netstat.

 --END OF NOTIFICATION

Quote:> Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
> strong point, but I'm much less thrilled by it now.

I'm also working on getting these alternate tools installed on a
completely clean system right now to get some decent monitoring in
place.

Quote:> Many of us suffer through the same interference.  Best advice remains
> a wipe, reinstall from trusted media, and updates.  In this case, I'd
> also strongly recommend new passwords for ALL accounts.  Then, take a
> snapshot of the system using 'aide' (or the fore-runner 'tripwire').
> This is what the system looks like "now" - and you can then compare
> this snapshot to "later" and look for changes. Big caution: the
> aide binary and database go on removable media that is kept in a safe
> place when not being used. Also, you have to retake the snapshot each
> time you intentionally change something (such as security updates). It
> is a pain in the a$$, but less of a pain than what you are going
> through now.

Gotcha.  Your recommendations have given me an excellent starting point
and I thank you much.  Any further feedback is greatly appreciated.

-Damon Getsman

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by 1PW » Fri, 23 Jan 2009 06:12:53




>> NOTE: Posting from groups.google.com (or some web-forums) dramatically
>> reduces the chance of your post being seen.  Find a real news server.

> Obviously I've gotten to that step now, something I had been neglecting
> previously.  Thanks.

>>>> Are you indicating that someone already _HAS_ gotten in?   If so, you
>>>> are screwed - as you don't know what that person has done.
>>> I am aware of that, and yes, I'm pretty sure that he did get root.

> At this stage I know that he got root on one of our externally facing
> machines.  This machine has been wiped and reinstalled but is still not
> up to the standards that I would like it to be at.  It also had an
> internal interface which was not securely firewalled or DMZed.  Yes, I
> understand the horror of that situation, but it was out of my control.
> Thanks to that issue I now have the ability to make suggestions that
> will be taken much more seriously regarding our handling of network
> security in the future.  This does not help the fact that I'm still
> learning this as I go, however, at least at this level of
> sophistication.

>> The only safe solution is to wipe and reinstall from known clean media.
>> But you know that too.

> As a starting point I have wiped and reinstalled my own workstation; I
> am not totally sure that I had it locked down before I reconnected it to
> the WAN, however, so I may just end up doing that again.  I am also
> using a startpoint of my own personal laptop which, at this point, is
> disconnected from the WAN and I am fairly certain is not compromised at
> the moment.  It is running a significantly different kernel from the
> other machines, as well.

>>> Almost certainly looks to be from an apache or webmin hole.
>> Yeah, there are plenty of those. Two points: PHP is generally a walking
>> disaster area - looking at Bugtraq shows more PHP exploits than
>> anything else, followed fairly closely by SQL.

> Thank you.  As we get downtime where our users are not needing vital
> network services I will make sure that we concentrate on these areas and
> not providing them where they aren't needed, and maintaining the best I
> can find to lock them down where they absolutely are needed.

>> Second point, if one of your users has been 0wn3d _elsewhere_ and they
>> had their SSH authentication tokens on "that" system, there is an
>> exploit out there that uses stolen authentication to get "in" to your
>> system, uses any number of local exploits to escalate privilege to
>> root, and then installs the 'Phalanx2' root kit that hides itself
>> pretty well, and searches for authentication data on your system (which
>> it mails out to a drop-box).  CERT reported this about five months ago.

> Yep.  I remember similar .rhost exploits and disasters surrounding them
> over a decade ago.

>> The only safe solution is the wipe/reinstall/update.  There are a
>> number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
>> (http://www.chkrootkit.org), 'rkhunter' (http://www.rootkit.nl), and
>> from a cursory look, 'ossec' (http://www.ossec.net/) that can be used
>> to look for many rootkits.  By in large, these are trivial to defeat or
>> bypass.  As one example, all look for the existence of a file named
>> '/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
>> sniffer worm from 2003).  That's all well and good, but what if the
>> mal-ware author does something terribly complicated, such as changing
>> the filename to '/tmp/.../A' or '/tmp/.../b' or something...  yup, the
>> anti-mal-ware tools won't find it.

> I've actually been able to start going through some OSSEC logs at this
> point that are showing things of this sort.  Of course, I do not know
> enough about the specifics of OSSEC administration just yet to be able
> to eliminate all of the false positives, however there are a few entries
> in them that are leading me to believe that we have significant
> intrusion on some if not all of our primary internal servers within the
> WAN.  IE:

> OSSEC HIDS Notification.
> 2009 Jan 20 15:29:47

> Received From: (agentname) 192.168.1.NOU->rootcheck
> Rule: 510 fired (level 7) -> "Host-based anomaly detection event
> (rootcheck)."
> Portion of the log(s):

> Port '41861'(tcp) hidden. Kernel-level rootkit or trojaned version of
> netstat.

>  --END OF NOTIFICATION

>> Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
>> strong point, but I'm much less thrilled by it now.

> I'm also working on getting these alternate tools installed on a
> completely clean system right now to get some decent monitoring in
> place.

>> Many of us suffer through the same interference.  Best advice remains
>> a wipe, reinstall from trusted media, and updates.  In this case, I'd
>> also strongly recommend new passwords for ALL accounts.  Then, take a
>> snapshot of the system using 'aide' (or the fore-runner 'tripwire').
>> This is what the system looks like "now" - and you can then compare
>> this snapshot to "later" and look for changes. Big caution: the
>> aide binary and database go on removable media that is kept in a safe
>> place when not being used. Also, you have to retake the snapshot each
>> time you intentionally change something (such as security updates). It
>> is a pain in the a$$, but less of a pain than what you are going
>> through now.

> Gotcha.  Your recommendations have given me an excellent starting point
> and I thank you much.  Any further feedback is greatly appreciated.

> -Damon Getsman

Hello Damon:

I know you are still trying to "drain the swamp" there.  At some point,
I hope you, and those you work for, are able to consider using a fairly
recent enterprise solution such as RHEL5/CentOS5.  The addition of
SELinux can help you harden your systems as few other schemes can.

I know that integrating SELinux with Hardy Heron has been done, but it's
a significant undertaking, whereas SELinux has been part of
RHEL/CentOS/Fedora/Debian for awhile.

Below is a URL that points to many guides to system hardening:

<http://www.nsa.gov/ia/guidance/security_configuration_guides/current_...>

In the list, you will find hardening tips for Solaris systems.

I immediately realize that the following does not *directly* apply to
*any* of your systems.  However, I believe that some of these steps may
still have value for you:

  <hXXp://www.nsa.gov/ia/_files/os/redhat/rhel5-pamphlet-i731.pdf>

   <hXXp://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf>

I obfuscated the URLs for your security protection.

I'm hoping that some of this will give you food for thought.  Good luck.

Pete
--

 
 
 

System intrusion detection, primarily on linux servers with a handful of others

Post by Moe Tr » Fri, 23 Jan 2009 10:04:32


On Wed, 21 Jan 2009, in the Usenet newsgroups comp.os.linux.networking
and comp.os.linux.security, in article <gl7dvh$h...@nntp.motzarella.org>,

dgets...@amirehab.net wrote:
>Moe Trin <ibupro...@painkiller.example.tld> wrote:
>> NOTE: Posting from groups.google.com (or some web-forums) dramatically
>> reduces the chance of your post being seen.  Find a real news server.

>Obviously I've gotten to that step now, something I had been neglecting
>previously.  Thanks.

It was a very low priority, but glad to have been able to help.

>>>I am aware of that, and yes, I'm pretty sure that he did get root.

>At this stage I know that he got root on one of our externally facing
>machines.  This machine has been wiped and reinstalled but is still not
>up to the standards that I would like it to be at.

Oh, wonderful.  You may want to look carefully of what applications are
installed.  For example, eliminating file transfer mechanisms (such as
FTP, SCP, wget, web browsers and the like) makes it awfully hard for a
script kiddie to install his r00tkit.  Yes, that also makes life hard on
your end, but some extra thought will get around most of that problem.

External facing systems.  Give some thought to operating from a
read-only device. This could mean something like a CD or DVD if you can
convince the system to boot from there. Some time ago, hard drives used
to have a jumper that could set them to be read-only - often called a
write-enable jumper. Logs (syslogd) can be sent to another networked
system (that only accepts connections from a limited number of hosts),
and volatile data, such as /var/run and /var/lock can go to a small
R/W device (even a floppy) or Ram-disk.  This doesn't prevent the bad
guy from finding a method to download his r00tkit to memory and
installing it as a memory resident thing ONLY, but it makes things a
heck of a lot harder for them.

Another item - if the filesystem has to be mounted R/W, consider
setting "critical" files as immutable (man chattr, man lsattr), as
few people think of extended attributes.  Consider the risk, but you
can also rename /usr/bin/chattr and /usr/bin/lsattr - which will
frustrate the bad guy, and may frustrate you as well, especially when
you try to update things.  THINk.  ;-)

>It also had an internal interface which was not securely firewalled or
>DMZed.  Yes, I understand the horror of that situation, but it was out
>of my control.  Thanks to that issue I now have the ability to make
>suggestions that will be taken much more seriously regarding our
>handling of network security in the future.  This does not help the
>fact that I'm still learning this as I go, however, at least at this
>level of sophistication.

Internal interface - connects to a switch (rather than a hub - only
hears traffic meant for it, and broadcasts), and the switch IF IT IS
MANAGED, does not accept connections from the unsafe side of the tracks.
Others might consider a proxying firewall such that broadcasts don't
get to the unsafe side.  Look at the needed information flow - what
needs to get to/from the external side servers - minimize the allowed
connections.  For example, you might set things up such that nothing
on the "inside" accepts connections from the external side servers, and
all connections have to be initiated from the inside (but that means
server applications on the server, and that requirement may conflict
with preventing the bad guy from being able to transfer files in).

Another thought - look at the times of information transfers on the
internal side.   If it's only needed for administration, you MAY be
able to disconnect that interface (same idea as unplugging the cable)
when not needed.   You have physical access - the bad guy probably
(hopefully) doesn't.

>As a starting point I have wiped and reinstalled my own workstation;
>I am not totally sure that I had it locked down before I reconnected
>it to the WAN, however, so I may just end up doing that again.

You're doing fine.    Only needed stuff installed, and all that.

Did you get a chance to take a snapshot of the system?

>I am also using a startpoint of my own personal laptop which, at this
>point, is disconnected from the WAN and I am fairly certain is not
>compromised at the moment.  It is running a significantly different
>kernel from the other machines, as well.

That should make the kernel level root exploits more difficult.

>As we get downtime where our users are not needing vital network
>services I will make sure that we concentrate on these areas and
>not providing them where they aren't needed, and maintaining the best
>I can find to lock them down where they absolutely are needed.

I know you are already overloaded,  but try to get this done as soon
as possible.  Another reading suggestion (long and expen$ive, but well
worth it) is "Practical Unix And Internet Security" by Spafford,
Garfinkle and Schwartz (O'Reilly, 0-596-00323-4, $55, 954 pgs, 2003).
That can wait a bit, as you've got lots of other things to do.

You had written in an earlier post that /bin/bash and /bin/grep had
altered timestamps.  Several thoughts - do you have backups from
before that date?  Does '/usr/bin/find / -mtime <mumble>' (where mumble
is the number of days back to the altered date) show anything of
interest, or has /usr/bin/find been altered as well?   Does the output
of '/bin/ps aux' and '/usr/bin/pstree' show any surprises (Try also
'/usr/bin/pstree -u'.) Often, when '/bin/ps' gets 0wn3d, they forget
about pstree.  Have you also looked at your package manager? (rpm -Va
or debsums -ca).  They're not perfect, but also get overlooked. I don't
know about debsums, but '/usr/bin/file /bin/rpm' shows it to be static
rather than dynamically linked.

>> there is an exploit out there that uses stolen authentication to get
>> "in" to your system
>Yep.  I remember similar .rhost exploits and disasters surrounding
>them over a decade ago.

I don't know which is worse (boy, were we naive and trusting back then).

>I've actually been able to start going through some OSSEC logs at
>this point that are showing things of this sort.

One of the things about the *nix world is that we don't have the
windoze mal-ware removal tools. Part of this is because we realize
that the malware can morph, and what was seen isn't necessarily what
is actually on your system. I suspect this may account for some of
the enormous numbers of variants of malware in the windoze side.

>Of course, I do not know enough about the specifics of OSSEC
>administration just yet to be able to eliminate all of the false
>positives,

and what hurts is you don't have the free time (Hah!) you need to get
up to speed on this.  I was also a little disappointed in the
documentation of ossec.

>however there are a few entries in them that are leading me to believe
>that we have significant intrusion on some if not all of our primary
>internal servers within the WAN.  IE:

Can you get outside help?   You've got a lot of work here.

>Port '41861'(tcp) hidden. Kernel-level rootkit or trojaned version
>of netstat.

Oh, FSCK!   Yeah, that's not smelling nice.  From outside (or via
localhost) can you connect to that port?   I _think_ that ossec tries
to connect to every port/protocol, was able to connect here, but
netstat doesn't show the connection.

>I'm also working on getting these alternate tools installed on a
>completely clean system right now to get some decent monitoring in
>place.

Yeah, it really appears that you've got a major problem there. Can you
install a system to act as a sniffer as close to the outside server NIC
as practical, and see what traffic is flowing?  Is there an external
firewall you can run - or even just a firewall installed on the 0wn3d
boxes so that you can block all inbound attempts expect to what ever
ports MUST be open to the world?  Can you restrict address ranges on
the outside? Do you have some idea of what legitimate addresses are
on the outside?  As mentioned, you might be able to not allow access
except from "trusted" (well, less un-trusted) addresses. A problem is
that many of the addresses used by bad guys are zombies - often home
windoze boxes ("MS-Windows: 100,000,000 'Bots strong") on US domestic
networks. I recently made a whois query at ARIN about an address
assigned to a major US provider and the response noted

  Comment:    <COMPANY NAME DELETED> currently has 44.3 Million
  Comment:    subscribers with 2.097 Million IP addresses allocated.

and I couldn't help wondering how many of those were zombied.  It may
not be suitable for you, but where I work, we block access to our
non-public systems from all "home" networks (AOL, Comcast, Cox, QWorst,
RR, TimeWarner, etc.) because we can't see a valid reason for those
hosts to connect to our workstations/fileserver/what-ever, AND vice-
versa. That eliminates a lot of 'bot problems. Obviously, YMMV.

Anyway, I think you're at the point where it's necessary to convince
management that it's getting critical that you take the systems down
and reinstall.  I know that's not the advice you want to hear, but I
think it's the only correct answer.  Yes, it would help to be able to
have a plan in place to minimize the chance of re-infection, but don't
take long.

        Old guy

 
 
 

1. New network intrusion detection system for Linux (GPL)

Hi,

This is perhaps a bit OT, or blatent p1mp4ge....

Just to let you know about a new NIDS for Linux, its called firestorm,
you can check it out at http://www.scaramanga.co.uk/firestorm/index.html.
It boasts a very flexible plugin-based architecture and is pretty similar
to snort in terms of functionality (some things it does better, a few
things it doesn't do). The latest version is 0.1.6.

Significant features:
* Multi-threaded capture from libpcap live capture, tcpdump files, and
linux firewall netlink (ala fwmon).
* Plugin-based architecture
* Snort ruleset support
* Efficient, small codebase (~5,000 lines of C)

Anyway, you can read the docs for yourselves if you're interested ;)

I'm basically looking for comments, opinions and of course people to help
testing/writing/documenting it.

Thanks.

--

Zwei erdnusse gingen hinunter die strasse.
Man wurde angegriffen... erdnuss

2. recovering data from an electricuted HD

3. Intrusion Detection Systems: An Introduction

4. Netra T1 & iolan+ terminal server

5. SPECTER Intrusion Detection System

6. Avoiding symbol name clashes

7. RAID controller recommendations

8. new Project : Intrusion Detection System

9. Intrusion Detection System.

10. NYC LOCAL: Thursday 15 November 2001 UNIGROUP: James Licata on Intrusion Detection Systems

11. Intrusion Detection Systems Management Console