Zombie port being used UNKNOWN BY WHAT PROCESS

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by Ignoramus651 » Wed, 19 Nov 2008 03:23:01



We have a server with a variety of processes running and listening to
various ports. They tend to work well.

One in particular, not remarkable in any way, listens on port 45764.

Today it is no longer is able to listen and says listen failed.

My investigation turned up the following facts:

1) If I telnet to this port, connection gets established
2) fuser -t 45764/tcp outputs NOTHING
3) netstat -aep outputs

... stuff ...
tcp        0      0 *:45764                 *:*                     LISTEN      root       12180       -
... stuff ...

Most other lines ...stuff... contact the PID and name of the process
in question, but not this one. This one has a "-" for PID/Name

So, we have a rogue zombie TCP port here, any ideas? Who is listening
on it?

--
   Due to extreme spam originating from Google Groups, and their inattention
      to spammers, I and many others block all articles originating
       from Google Groups. If you want your postings to be seen by
         more readers you will need to find a different means of
                       posting on Usenet.
                   http://improve-usenet.org/

 
 
 

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by Ignoramus651 » Wed, 19 Nov 2008 03:44:15



Quote:> We have a server with a variety of processes running and listening to
> various ports. They tend to work well.

> One in particular, not remarkable in any way, listens on port 45764.

> Today it is no longer is able to listen and says listen failed.

> My investigation turned up the following facts:

> 1) If I telnet to this port, connection gets established
> 2) fuser -t 45764/tcp outputs NOTHING
> 3) netstat -aep outputs

> ... stuff ...
> tcp        0      0 *:45764                 *:*                     LISTEN      root       12180       -
> ... stuff ...

> Most other lines ...stuff... contact the PID and name of the process
> in question, but not this one. This one has a "-" for PID/Name

One more observation. If I telnet to that port and say "*" and hit
ENTER, I get this output in /var/log/messages:

Nov 17 12:29:35 myserver kernel: [131074.840941] RPC: bad TCP reclen 0x6675636b (non-terminal)

i

Quote:> So, we have a rogue zombie TCP port here, any ideas? Who is listening
> on it?

--
   Due to extreme spam originating from Google Groups, and their inattention
      to spammers, I and many others block all articles originating
       from Google Groups. If you want your postings to be seen by
         more readers you will need to find a different means of
                       posting on Usenet.
                   http://www.veryComputer.com/

 
 
 

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by Allen Kistle » Wed, 19 Nov 2008 04:17:36




>> We have a server with a variety of processes running and listening to
>> various ports. They tend to work well.

>> One in particular, not remarkable in any way, listens on port 45764.

>> Today it is no longer is able to listen and says listen failed.

>> My investigation turned up the following facts:

>> 1) If I telnet to this port, connection gets established
>> 2) fuser -t 45764/tcp outputs NOTHING
>> 3) netstat -aep outputs

>> ... stuff ...
>> tcp        0      0 *:45764                 *:*                     LISTEN      root       12180       -
>> ... stuff ...

>> Most other lines ...stuff... contact the PID and name of the process
>> in question, but not this one. This one has a "-" for PID/Name

> One more observation. If I telnet to that port and say "*" and hit
> ENTER, I get this output in /var/log/messages:

> Nov 17 12:29:35 myserver kernel: [131074.840941] RPC: bad TCP reclen 0x6675636b (non-terminal)

> i

>> So, we have a rogue zombie TCP port here, any ideas? Who is listening
>> on it?

Try "rpcinfo -p" then.
 
 
 

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by Cork Soake » Wed, 19 Nov 2008 03:43:26



> We have a server with a variety of processes running and listening to
> various ports. They tend to work well.

> One in particular, not remarkable in any way, listens on port 45764.

> Today it is no longer is able to listen and says listen failed.

> My investigation turned up the following facts:

> 1) If I telnet to this port, connection gets established
> 2) fuser -t 45764/tcp outputs NOTHING
> 3) netstat -aep outputs

> ... stuff ...
> tcp        0      0 *:45764                 *:*                     LISTEN      root       12180       -
> ... stuff ...

> Most other lines ...stuff... contact the PID and name of the process
> in question, but not this one. This one has a "-" for PID/Name

> So, we have a rogue zombie TCP port here, any ideas? Who is listening
> on it?

Well, what's the process?  top will tell you what 12180 is.
 
 
 

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by Ignoramus651 » Wed, 19 Nov 2008 04:53:00





>>> We have a server with a variety of processes running and listening to
>>> various ports. They tend to work well.

>>> One in particular, not remarkable in any way, listens on port 45764.

>>> Today it is no longer is able to listen and says listen failed.

>>> My investigation turned up the following facts:

>>> 1) If I telnet to this port, connection gets established
>>> 2) fuser -t 45764/tcp outputs NOTHING
>>> 3) netstat -aep outputs

>>> ... stuff ...
>>> tcp        0      0 *:45764                 *:*                     LISTEN      root       12180       -
>>> ... stuff ...

>>> Most other lines ...stuff... contact the PID and name of the process
>>> in question, but not this one. This one has a "-" for PID/Name

>> One more observation. If I telnet to that port and say "*" and hit
>> ENTER, I get this output in /var/log/messages:

>> Nov 17 12:29:35 myserver kernel: [131074.840941] RPC: bad TCP reclen 0x6675636b (non-terminal)

>> i

>>> So, we have a rogue zombie TCP port here, any ideas? Who is listening
>>> on it?

> Try "rpcinfo -p" then.

Very interesting!

root:~ ==] rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100021    1   tcp  45764  nlockmgr
    100021    3   tcp  45764  nlockmgr
    100021    4   tcp  45764  nlockmgr
    100024    1   udp  32765  status
    100024    1   tcp  32765  status

So, how can I make portmap allocate ports from only a range that I
specify?

--
   Due to extreme spam originating from Google Groups, and their inattention
      to spammers, I and many others block all articles originating
       from Google Groups. If you want your postings to be seen by
         more readers you will need to find a different means of
                       posting on Usenet.
                   http://www.veryComputer.com/

 
 
 

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by Ignoramus651 » Wed, 19 Nov 2008 04:53:31




>> We have a server with a variety of processes running and listening to
>> various ports. They tend to work well.

>> One in particular, not remarkable in any way, listens on port 45764.

>> Today it is no longer is able to listen and says listen failed.

>> My investigation turned up the following facts:

>> 1) If I telnet to this port, connection gets established
>> 2) fuser -t 45764/tcp outputs NOTHING
>> 3) netstat -aep outputs

>> ... stuff ...
>> tcp        0      0 *:45764                 *:*                     LISTEN      root       12180       -
>> ... stuff ...

>> Most other lines ...stuff... contact the PID and name of the process
>> in question, but not this one. This one has a "-" for PID/Name

>> So, we have a rogue zombie TCP port here, any ideas? Who is listening
>> on it?

> Well, what's the process?  top will tell you what 12180 is.

There is NO such process. That's the thing. By the way, 12180 is a
"inode number" not a process. Whatever it means.
--
   Due to extreme spam originating from Google Groups, and their inattention
      to spammers, I and many others block all articles originating
       from Google Groups. If you want your postings to be seen by
         more readers you will need to find a different means of
                       posting on Usenet.
                   http://improve-usenet.org/
 
 
 

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by Allen Kistle » Wed, 19 Nov 2008 05:31:48




>> Try "rpcinfo -p" then.

> Very interesting!

> root:~ ==] rpcinfo -p
>    program vers proto   port
>     100000    2   tcp    111  portmapper
>     100000    2   udp    111  portmapper
>     100021    1   tcp  45764  nlockmgr
>     100021    3   tcp  45764  nlockmgr
>     100021    4   tcp  45764  nlockmgr
>     100024    1   udp  32765  status
>     100024    1   tcp  32765  status

> So, how can I make portmap allocate ports from only a range that I
> specify?

Actually the question is how to make nfs lockd and nfs statd request
only the port that you specify.  (If the rpc process does not request a
port or if the requested port is unavailable, then portmap picks a
random open port.)  Maybe the first question to answer, though, is why
you're running lockd and statd if you're not sharing anything via nfs.
(I notice mountd and nfsd are not running.)  Also a good follow-up
question would be why you're running portmap (sometimes aka rpcbind) if
you're not mounting anything via nfs, if indeed you're not mounting
anything via nfs.

The answer to how to make lockd and statd use fixed ports depends a bit
on distro.  In RH and Fedora, I fix the lockd port in /etc/modprobe.conf
and the statd port in /etc/sysconfig/nfs as an argument that gets passed
to statd when it starts.

In /etc/modprobe.conf
options lockd nlm_udpport=2050 nlm_tcpport=2050

In /etc/sysconfig/nfs
STATD_PORT=2052

YMMV

 
 
 

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by denni » Wed, 19 Nov 2008 06:13:05





>>> We have a server with a variety of processes running and listening to
>>> various ports. They tend to work well.

>>> One in particular, not remarkable in any way, listens on port 45764.

>>> Today it is no longer is able to listen and says listen failed.

>>> My investigation turned up the following facts:

>>> 1) If I telnet to this port, connection gets established
>>> 2) fuser -t 45764/tcp outputs NOTHING
>>> 3) netstat -aep outputs

>>> ... stuff ...
>>> tcp        0      0 *:45764                 *:*
>>> LISTEN      root       12180       -
>>> ... stuff ...

>>> Most other lines ...stuff... contact the PID and name of the process
>>> in question, but not this one. This one has a "-" for PID/Name

>>> So, we have a rogue zombie TCP port here, any ideas? Who is listening
>>> on it?

>> Well, what's the process?  top will tell you what 12180 is.

> There is NO such process. That's the thing. By the way, 12180 is a
> "inode number" not a process. Whatever it means.

Inodes are what identifies files on a file system.
You could see what file is inode 12180 but I doubt if it will help.
BTW if you listen on that port the application could fail to start anytime.
Ports in that range can be used by things like web browsers and other
applications used by users at "random".
But I am surprised you didn't know that anyway being an expert.
 
 
 

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by Cork Soake » Wed, 19 Nov 2008 06:28:29





>>> We have a server with a variety of processes running and listening to
>>> various ports. They tend to work well.

>>> One in particular, not remarkable in any way, listens on port 45764.

>>> Today it is no longer is able to listen and says listen failed.

>>> My investigation turned up the following facts:

>>> 1) If I telnet to this port, connection gets established
>>> 2) fuser -t 45764/tcp outputs NOTHING
>>> 3) netstat -aep outputs

>>> ... stuff ...
>>> tcp        0      0 *:45764                 *:*                     LISTEN      root       12180       -
>>> ... stuff ...

>>> Most other lines ...stuff... contact the PID and name of the process
>>> in question, but not this one. This one has a "-" for PID/Name

>>> So, we have a rogue zombie TCP port here, any ideas? Who is listening
>>> on it?

>> Well, what's the process?  top will tell you what 12180 is.

> There is NO such process. That's the thing. By the way, 12180 is a
> "inode number" not a process. Whatever it means.

Well then, that's cleared that up! :-)

Also, Wireshark can snoop any traffic that it /may/ be making.

 
 
 

Zombie port being used UNKNOWN BY WHAT PROCESS

Post by Ignoramus651 » Wed, 19 Nov 2008 06:53:22





>>> Try "rpcinfo -p" then.

>> Very interesting!

>> root:~ ==] rpcinfo -p
>>    program vers proto   port
>>     100000    2   tcp    111  portmapper
>>     100000    2   udp    111  portmapper
>>     100021    1   tcp  45764  nlockmgr
>>     100021    3   tcp  45764  nlockmgr
>>     100021    4   tcp  45764  nlockmgr
>>     100024    1   udp  32765  status
>>     100024    1   tcp  32765  status

>> So, how can I make portmap allocate ports from only a range that I
>> specify?

> Actually the question is how to make nfs lockd and nfs statd request
> only the port that you specify.  (If the rpc process does not request a
> port or if the requested port is unavailable, then portmap picks a
> random open port.)  Maybe the first question to answer, though, is why
> you're running lockd and statd if you're not sharing anything via nfs.
> (I notice mountd and nfsd are not running.)  Also a good follow-up
> question would be why you're running portmap (sometimes aka rpcbind) if
> you're not mounting anything via nfs, if indeed you're not mounting
> anything via nfs.

Actually I am using some NFS shares as a client, but this server is
not sharing anything of its own own.

Quote:> The answer to how to make lockd and statd use fixed ports depends a bit
> on distro.  In RH and Fedora, I fix the lockd port in /etc/modprobe.conf
> and the statd port in /etc/sysconfig/nfs as an argument that gets passed
> to statd when it starts.

> In /etc/modprobe.conf
> options lockd nlm_udpport=2050 nlm_tcpport=2050

> In /etc/sysconfig/nfs
> STATD_PORT=2052

> YMMV

We decided to move some ports, and adjust the ephemeral port range by
setting net.ipv4.ip_local_port_range = 58000 65535, in file
/etc/sysctl.conf. We'll move some more ports around and increase the
range from 58000 start to 51000 start.

--
   Due to extreme spam originating from Google Groups, and their inattention
      to spammers, I and many others block all articles originating
       from Google Groups. If you want your postings to be seen by
         more readers you will need to find a different means of
                       posting on Usenet.
                   http://improve-usenet.org/

 
 
 

1. HELP: Unknown process intermittently accessing port 1022 after printer removal

Hi,

We used to have a remote printer on NetWare-NFS to which we print. The
printer was defined as:

lpadmin -p the_printer -s the_netware_server!the_netware_queue

Now, we removed that printer from BOTH unix and netware lately, but we
still have console messages on netware telling us that our UNIX box is
till trying to access a nonexistent quieue on netware.

We tried bouncing lpsched (lpshut, remove LOCKS and
/var/spool/lp/requests and /var/spool/lp/tmp, restart lpsched) but
that did not fix it.

snoop from unixbox to netware_server gives the following every minute
or so (At the same time the netware box gives us the error message):

      produc -> srvsf        PRINTER C port=1022
      produc -> srvsf        PRINTER C port=1022
      produc -> srvsf        PRINTER C port=1022 \2QOPT_PRINCIPALE\n
      produc -> srvsf        PRINTER C port=1022
      produc -> srvsf        PRINTER C port=1022
      produc -> srvsf        PRINTER C port=1022

Yesterday, it was port 1023

After running lsof -i :1023 for a while repeatedly, we found the
following:

COMMAND   PID USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
lp      15388 root    6u  inet 0x6697ec38      0t0  TCP
produc:1023->srvsf:printer (SYN_SENT)

This process is not there long enough to know where it is comming
from. We cannot reboot the server merely to get rid af an annoying
console message on netware.

Printing on solaris as always eb confusing for me, it seems to be a
mix of BSD and System V and proprietary stuff. It was way more easy on
Digital Unix.

If anyone can shed some light on this...

TIA!

2. SCCS Question

3. Zombie processes "owned" by apache processes ...

4. Sound Recording

5. HELP: Unknown process intermittently accessing port 1022 after printer removal

6. Mailing List without a Listserv?

7. Ld: fatal: file /path/libXt.a: unknown type, unable to process using elf libraries

8. connecting two dialup machines

9. S1692DL Tiger 2 unknown PCI bridge :unknown Host bridge :unknown PCI Device

10. Problem adding tty port using 128 Port Async Adapter with 16 port Async Node

11. How to determine which process uses which tcp/udp port?

12. S9: which process uses the port ?

13. finding out which process is using a particular port#