UDP packets dropping

UDP packets dropping

Post by Visha » Sat, 11 Nov 2006 21:58:31



Hi,
Its a very simple problem which anyone who has worked on any UDP
application might have faced.
UDP by itself is unreliable. If packets are lost, no acknowledgements
are sent.

I have a UDP application which is receiving packets on a standard port.
It receives packets asynchronously. At any time any packet can arrive.
The problem is that sometimes a number of packets arrive and our UDP
socket buffer ( being of finite size ) is not able
to handle them and simply drops the packet.
The UDP socket buffer cannot be increased beyond a finite size(
dynamically it cannot be increased, it is fixed for an OS) .

So a number of packets simply drop with out any intimation to the
receiver/sender.

I need to have checkpoint ( some kind of intimation ) that the packets
are being dropped.

I have implemented polling with poll parameter as POLLIN or
POLLRDNORM.( I have tried both parameters)
Both implementation works  that is gives me a poll status  implying
socket buffer is full.

In my test set up I can control the rate at which packets are
generated.
When the packets are sent at a very high rate and my socket buffer is
not able  to handle them, polling status tells me that
buffer is full.

when the rate of sending packets is slow, I get an intimation that
buffer space is available.

But in some cases ( when the rate of sending packets is slow ), I get a
poll status that buffer is full ( which is wrong ). Even though
these are corner cases but it made me think.( and caused worry) .

I have following queries..

a) Do we have any mechanism/ way in which we can find that udp socket
buffer is full and packets are being dropped.
b) Can polling be actually be used. My corner cases made me think that.
Is my approach totally wrong.
c) If not Polling then what ?
d) If polling can be used, which other parameters I can use to avoid
these corner case.

e) Is my implementation of polling wrong, Is it possible that I am
misunderstanding the polling status to tell whether
  buffer is full.

if you have some idea about  this issue please let me know.
If you need some more input from my end let me know.

 
 
 

UDP packets dropping

Post by Jacob Bunk Nielse » Sat, 11 Nov 2006 22:26:25



> I need to have checkpoint ( some kind of intimation ) that the packets
> are being dropped.

Why not just use TCP that provides reliable transport?

How does your application handle if packets arrive out of order? If it
doesn't it's yet another reason to look at TCP.

Quote:> I have implemented polling with poll parameter as POLLIN or
> POLLRDNORM.( I have tried both parameters)
> Both implementation works  that is gives me a poll status  implying
> socket buffer is full.

What if your buffer is almost full? E.g. there is room for a small
packet, but not for a bigger one?

Quote:> a) Do we have any mechanism/ way in which we can find that udp socket
> buffer is full and packets are being dropped.

What makes you think that packets are only being dropped when your
buffer is full? What will happen if packets are dropped elsewhere in
the network?

Quote:> b) Can polling be actually be used. My corner cases made me think that.
> Is my approach totally wrong.

I believe it is.

Quote:> c) If not Polling then what ?

TCP or some other transport protocol that provides reliable transport
if that is really what you need?

--
Jacob

 
 
 

UDP packets dropping

Post by Rick Jone » Sun, 12 Nov 2006 00:29:17



> I have implemented polling with poll parameter as POLLIN or
> POLLRDNORM.( I have tried both parameters)
> Both implementation works  that is gives me a poll status  implying
> socket buffer is full.
> In my test set up I can control the rate at which packets are
> generated.
> When the packets are sent at a very high rate and my socket buffer is
> not able  to handle them, polling status tells me that
> buffer is full.
> when the rate of sending packets is slow, I get an intimation that
> buffer space is available.
> But in some cases ( when the rate of sending packets is slow ), I get a
> poll status that buffer is full ( which is wrong ). Even though
> these are corner cases but it made me think.( and caused worry) .

unless there is a watermark invovled, poll POLLIN/POLLRDNORM only
means there is _some_ data in the buffer.  It does not mean the socket
buffer is full.

Quote:> I have following queries..
> a) Do we have any mechanism/ way in which we can find that udp socket
> buffer is full and packets are being dropped.

Platform-specific

Quote:> b) Can polling be actually be used. My corner cases made me think that.
> Is my approach totally wrong.

It is, polling via poll can tell you nothing about dropped packets

Quote:> c) If not Polling then what ?

Platform-specific

Quote:> d) If polling can be used, which other parameters I can use to avoid
> these corner case.

none, save perhaps being able to set watermarks on the socket, which
isn't a portable thing.

Quote:> e) Is my implementation of polling wrong, Is it possible that I am
> misunderstanding the polling status to tell whether buffer is full.

Yes, you are misunderstanding what poll returns.

I think others have pointed-out that it isn't just a full socket
buffer which can lead to your UDP datagrams being lost.  The only way
to know that UDP datagrams have been lost is to have your own
application-level acknowledgement mechanism.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

 
 
 

UDP packets dropping

Post by gudd » Tue, 14 Nov 2006 02:51:36




> > I have implemented polling with poll parameter as POLLIN or
> > POLLRDNORM.( I have tried both parameters)
> > Both implementation works  that is gives me a poll status  implying
> > socket buffer is full.

> > In my test set up I can control the rate at which packets are
> > generated.
> > When the packets are sent at a very high rate and my socket buffer is
> > not able  to handle them, polling status tells me that
> > buffer is full.

> > when the rate of sending packets is slow, I get an intimation that
> > buffer space is available.

> > But in some cases ( when the rate of sending packets is slow ), I get a
> > poll status that buffer is full ( which is wrong ). Even though
> > these are corner cases but it made me think.( and caused worry) .

> unless there is a watermark invovled, poll POLLIN/POLLRDNORM only
> means there is _some_ data in the buffer.  It does not mean the socket
> buffer is full.

> > I have following queries..

> > a) Do we have any mechanism/ way in which we can find that udp socket
> > buffer is full and packets are being dropped.

> Platform-specific

When you say platform specific how does it matter whether it it Unix or
windows? Or do you mean to different processors( thats unlikely)

- Show quoted text -

Quote:

> > b) Can polling be actually be used. My corner cases made me think that.
> > Is my approach totally wrong.

> It is, polling via poll can tell you nothing about dropped packets

> > c) If not Polling then what ?

> Platform-specific

> > d) If polling can be used, which other parameters I can use to avoid
> > these corner case.

> none, save perhaps being able to set watermarks on the socket, which
> isn't a portable thing.

What are watermarks ? And how to use them ? Can you give insights .
May if we can something for one OS( unix or Windows ) that will be
great help ..

- Show quoted text -

Quote:> > e) Is my implementation of polling wrong, Is it possible that I am
> > misunderstanding the polling status to tell whether buffer is full.

> Yes, you are misunderstanding what poll returns.

> I think others have pointed-out that it isn't just a full socket
> buffer which can lead to your UDP datagrams being lost.  The only way
> to know that UDP datagrams have been lost is to have your own
> application-level acknowledgement mechanism.

> rick jones
> --
> Process shall set you free from the need for rational thought.
> these opinions are mine, all mine; HP might not want them anyway... :)
> feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

 
 
 

UDP packets dropping

Post by Dan » Tue, 14 Nov 2006 10:15:00




>> I need to have checkpoint ( some kind of intimation ) that the packets
>> are being dropped.

> Why not just use TCP that provides reliable transport?

That seems the most obvious solution.  The op wants reliable
transport from UDP, which is the wrong approach.

Dan

 
 
 

UDP packets dropping

Post by gudd » Tue, 14 Nov 2006 13:41:06


Hi Jacob,
As i had mentioned, we are receving at a standard port( UDP port) . The
software has to receive on UDP port as it is based on a standard. So
TCP is totally ruled out in my case. as we have to stuck to a specific
standard.

Regards,
Vishal



> > I need to have checkpoint ( some kind of intimation ) that the packets
> > are being dropped.

> Why not just use TCP that provides reliable transport?

> How does your application handle if packets arrive out of order? If it
> doesn't it's yet another reason to look at TCP.

> > I have implemented polling with poll parameter as POLLIN or
> > POLLRDNORM.( I have tried both parameters)
> > Both implementation works  that is gives me a poll status  implying
> > socket buffer is full.

> What if your buffer is almost full? E.g. there is room for a small
> packet, but not for a bigger one?

> > a) Do we have any mechanism/ way in which we can find that udp socket
> > buffer is full and packets are being dropped.

> What makes you think that packets are only being dropped when your
> buffer is full? What will happen if packets are dropped elsewhere in
> the network?

> > b) Can polling be actually be used. My corner cases made me think that.
> > Is my approach totally wrong.

> I believe it is.

> > c) If not Polling then what ?

> TCP or some other transport protocol that provides reliable transport
> if that is really what you need?

> --
> Jacob

 
 
 

UDP packets dropping

Post by Jacob Bunk Nielse » Tue, 14 Nov 2006 18:02:24



> As i had mentioned, we are receving at a standard port( UDP port) . The
> software has to receive on UDP port as it is based on a standard. So
> TCP is totally ruled out in my case. as we have to stuck to a specific
> standard.

Doesn't the standard describe some way for handling dropped packets?

If you tell us which standard you are working with someone here might
know something about implementations of this specific standard.

--
Jacob

 
 
 

UDP packets dropping

Post by gudd » Tue, 14 Nov 2006 18:15:20


Quote:

> Doesn't the standard describe some way for handling dropped packets?

> If you tell us which standard you are working with someone here might
> know something about implementations of this specific standard.

> --
> Jacob

Hi Jacob,
I have been working on a SNMP application which is based on UDP.

regards,
Vishal

 
 
 

UDP packets dropping

Post by Jacob Bunk Nielse » Tue, 14 Nov 2006 18:36:48



> I have been working on a SNMP application which is based on UDP.

I just browsed RFC 1157 (which describes SNMPv1). It looks like it's a
design choice to keep SNMP as simple as possible and therefor reliable
transport of SNMP-messages has been deselected.

So basically you have no way of knowing if SNMP-packets are lost (in
the network or in a buffer or whatever). It's by design.

I'm not sure that SNMPv3 can't run over TCP. At least this is what my
/etc/services says:

$ grep -i snmp /etc/services
snmp            161/tcp                         # Simple Net Mgmt Protocol
snmp            161/udp                         # Simple Net Mgmt Protocol
snmp-trap       162/tcp         snmptrap        # Traps for SNMP
snmp-trap       162/udp         snmptrap        # Traps for SNMP
smux            199/tcp                         # SNMP Unix Multiplexer

... and also what RFC 1157 hints at in section 3.2.4.

--
Jacob

 
 
 

UDP packets dropping

Post by Joe Pfeiffe » Tue, 14 Nov 2006 23:48:34



Quote:

> I'm not sure that SNMPv3 can't run over TCP. At least this is what my
> /etc/services says:

> $ grep -i snmp /etc/services
> snmp            161/tcp                         # Simple Net Mgmt Protocol
> snmp            161/udp                         # Simple Net Mgmt Protocol
> snmp-trap       162/tcp         snmptrap        # Traps for SNMP
> snmp-trap       162/udp         snmptrap        # Traps for SNMP
> smux            199/tcp                         # SNMP Unix Multiplexer

> ... and also what RFC 1157 hints at in section 3.2.4.

The contents of /etc/services are no help at all for this -- as it
says at the top of the file,

# Note that it is presently the policy of IANA to assign a single well-known
# port number for both TCP and UDP; hence, officially ports have two entries
# even if the protocol doesn't support UDP operations.
(it doesn't say it, but it's also true of protocols that don't support
TCP operations).

I haven't read the RFP, but if the network devices are sending UDP
packets, no amount of telling them they could send TCP packets as well
is going to help them.  And since they aren't likely to support
requests to retransmit, it sounds like the OP is simply going to have
to write an application that is robust in the presence of dropped
packets.
--
Joseph J. Pfeiffer, Jr., Ph.D.       Phone -- (505) 646-1605
Department of Computer Science       FAX   -- (505) 646-1002
New Mexico State University          http://www.cs.nmsu.edu/~pfeiffer

 
 
 

UDP packets dropping

Post by Rick Jone » Wed, 15 Nov 2006 02:40:04




> When you say platform specific how does it matter whether it it Unix or
> windows?

yes.

Quote:> Or do you mean to different processors( thats unlikely)

no.

Quote:> What are watermarks ? And how to use them ? Can you give insights .
> May if we can something for one OS( unix or Windows ) that will be
> great help ..

I supsect the platform specifics of setting watermarks will be covered
in the manpage for setsockopt on each. There are also the excellent
works of the late W. Richard Stevens et al such as 'Unix Network
Programming." On some platforms, the option exists but is a no-op.

rick jones
--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

 
 
 

1. UDP packets dropped

Hi,

I have noticed that when using UDP at high bitrates in Linux, many
packets are dropped in the network stack before beeing transmitted!! I
read that this problem is related to the lack of socket buffering when
using UDP.

Is there any way to solve this problem?

Thanks,

Carles

2. nVidia, Xfree86 and Solaris 7

3. UDP packet drops when using 2 interfaces

4. Small Office Nwtwork Question

5. Firewall dropping Kazaa UDP packets it shouldn't -- Need help

6. Linux=bad attitude

7. PKERNEL: Warning: RPC/UDP receive queue are full, packet dropping

8. Why can't I switch back to X console?

9. Linux maybe dropping UDP packets?

10. dropped udp packets - due to no socket

11. UDP Tuning to prevent dropped packets

12. UDP dropping packets

13. Dropped UDP packets