Traffic Control traffic unexpectedly dipping

Traffic Control traffic unexpectedly dipping

Post by Washington Rats » Thu, 02 Dec 2010 11:54:47



I am using traffic control with a 2.6.26 kernel. During my testing, I
am intentionally sending in more best effort traffic than my hardware
can handle and I am seeing that the traffic control is dropping it, as
it should.  Occasionally, I see a dip in the traffic in the queues I
have defined for higher priority traffic.  This traffic is running
below the minimum rate.  When this happens, I see the following qdisc
statistics (sampling every 3 seconds):

qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0
 Sent 299188596 bytes 197913 pkt (dropped 162746, overlimits 377808
requeues 0)
 rate 0bit 0pps backlog 0b 999p requeues 0

qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0
 Sent 300330198 bytes 198669 pkt (dropped 163401, overlimits 379300
requeues 0)
 rate 0bit 0pps backlog 0b 1000p requeues 0

qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0
 Sent 300818616 bytes 198993 pkt (dropped 163684, overlimits 380146
requeues 0)
 rate 0bit 0pps backlog 0b 571p requeues 0

qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0
 Sent 301102914 bytes 199182 pkt (dropped 163684, overlimits 380626
requeues 0)
 rate 0bit 0pps backlog 0b 81p requeues 0

qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0
 Sent 303284730 bytes 200625 pkt (dropped 163684, overlimits 382126
requeues 0)
 rate 0bit 0pps backlog 0b 753p requeues 0

qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0
 Sent 304795218 bytes 201624 pkt (dropped 164095, overlimits 383623
requeues 0)
 rate 0bit 0pps backlog 0b 1000p requeues 0

As you can see, backlog goes from  999p to 1000p to 571p to 81p to
753p to 1000p.  Is this dip in the backlog to be expected?  Does
traffic control occasionally flush the backlog if it is staying around
1000p? I have verified that the incoming data, generated by iperf, is
constant and does not dip like this.  Note: while the backlog stays
well under 1000p, the number dropped stays at 163684.  Also, how can
backlog be 1000p but 0b?  (I assume that means 1000 packets and 0
bits.)

 
 
 

Traffic Control traffic unexpectedly dipping

Post by Andy Furnis » Sat, 04 Dec 2010 00:16:39



> I am using traffic control with a 2.6.26 kernel. During my testing, I
> am intentionally sending in more best effort traffic than my hardware
> can handle and I am seeing that the traffic control is dropping it, as
> it should.  Occasionally, I see a dip in the traffic in the queues I
> have defined for higher priority traffic.  This traffic is running
> below the minimum rate.  When this happens, I see the following qdisc
> statistics (sampling every 3 seconds):

> qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0

Not sure what the problem is (if it is a problem).

Remember when you set a default with htb that (unless you filter to
elsewhere) arp will also go to that class, which if low prio may cause
you problems.

 
 
 

Traffic Control traffic unexpectedly dipping

Post by Washington Rats » Sat, 04 Dec 2010 02:02:35




> > I am using traffic control with a 2.6.26 kernel. During my testing, I
> > am intentionally sending in more best effort traffic than my hardware
> > can handle and I am seeing that the traffic control is dropping it, as
> > it should. ?Occasionally, I see a dip in the traffic in the queues I
> > have defined for higher priority traffic. ?This traffic is running
> > below the minimum rate. ?When this happens, I see the following qdisc
> > statistics (sampling every 3 seconds):

> > qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0

> Not sure what the problem is (if it is a problem).

> Remember when you set a default with htb that (unless you filter to
> elsewhere) arp will also go to that class, which if low prio may cause
> you problems.

The problem is, for example, that a non-default htb queue with a rate
defined as 100000bits/sec which is getting 90000bits/sec of data only
lets about 20000bits/sec through while this behavior is seen.  Could
arp have this effect?
 
 
 

Traffic Control traffic unexpectedly dipping

Post by Washington Rats » Sat, 04 Dec 2010 02:48:10





> > > I am using traffic control with a 2.6.26 kernel. During my testing, I
> > > am intentionally sending in more best effort traffic than my hardware
> > > can handle and I am seeing that the traffic control is dropping it, as
> > > it should. ?Occasionally, I see a dip in the traffic in the queues I
> > > have defined for higher priority traffic. ?This traffic is running
> > > below the minimum rate. ?When this happens, I see the following qdisc
> > > statistics (sampling every 3 seconds):

> > > qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0

> > Not sure what the problem is (if it is a problem).

> > Remember when you set a default with htb that (unless you filter to
> > elsewhere) arp will also go to that class, which if low prio may cause
> > you problems.

> The problem is, for example, that a non-default htb queue with a rate
> defined as 100000bits/sec which is getting 90000bits/sec of data only
> lets about 20000bits/sec through while this behavior is seen. ?Could
> arp have this effect?

I ran the test again.  Actually the traffic from the non-default
queues goes to 0 at the same time that the backlog number goes down
and the number of packets dropped stays constant.
 
 
 

Traffic Control traffic unexpectedly dipping

Post by Washington Rats » Sat, 04 Dec 2010 04:02:41






> > > > I am using traffic control with a 2.6.26 kernel. During my testing, I
> > > > am intentionally sending in more best effort traffic than my hardware
> > > > can handle and I am seeing that the traffic control is dropping it, as
> > > > it should. ?Occasionally, I see a dip in the traffic in the queues I
> > > > have defined for higher priority traffic. ?This traffic is running
> > > > below the minimum rate. ?When this happens, I see the following qdisc
> > > > statistics (sampling every 3 seconds):

> > > > qdisc htb 9997: root r2q 10 default 9999 direct_packets_stat 0

> > > Not sure what the problem is (if it is a problem).

> > > Remember when you set a default with htb that (unless you filter to
> > > elsewhere) arp will also go to that class, which if low prio may cause
> > > you problems.

> > The problem is, for example, that a non-default htb queue with a rate
> > defined as 100000bits/sec which is getting 90000bits/sec of data only
> > lets about 20000bits/sec through while this behavior is seen. ?Could
> > arp have this effect?

> I ran the test again. ?Actually the traffic from the non-default
> queues goes to 0 at the same time that the backlog number goes down
> and the number of packets dropped stays constant.

Looking at the statistics for the classes, I can see that all of the
queues, including the default queue, stop sending data during this
period.  The data I am seeing from the default traffic was from data
previously buffered up in our system.
 
 
 

Traffic Control traffic unexpectedly dipping

Post by Andy Furnis » Sat, 04 Dec 2010 05:01:23



> Looking at the statistics for the classes, I can see that all of the
> queues, including the default queue, stop sending data during this
> period.  The data I am seeing from the default traffic was from data
> previously buffered up in our system.

I would test without using a default class. Just make a catch all tc
filter with a prio/pref higher than any others (1 is first for filters
not 0) to direct ip traffic to 9999.
 
 
 

Traffic Control traffic unexpectedly dipping

Post by Washington Rats » Sat, 04 Dec 2010 08:45:03




> > Looking at the statistics for the classes, I can see that all of the
> > queues, including the default queue, stop sending data during this
> > period. ?The data I am seeing from the default traffic was from data
> > previously buffered up in our system.

> I would test without using a default class. Just make a catch all tc
> filter with a prio/pref higher than any others (1 is first for filters
> not 0) to direct ip traffic to 9999.

I just tried that and the higher priority classes end up losing
packets.
 
 
 

Traffic Control traffic unexpectedly dipping

Post by Andy Furnis » Sun, 05 Dec 2010 04:18:06





>>> Looking at the statistics for the classes, I can see that all of the
>>> queues, including the default queue, stop sending data during this
>>> period.  The data I am seeing from the default traffic was from data
>>> previously buffered up in our system.

>> I would test without using a default class. Just make a catch all tc
>> filter with a prio/pref higher than any others (1 is first for filters
>> not 0) to direct ip traffic to 9999.

> I just tried that and the higher priority classes end up losing
> packets.

But does it fix the everything stops problem?

Maybe there is some other issue with the setup/tests that are causing this.

Posting more details of the hardware, tests, results and the most
simplified/stripped down htb setup that shows the problem would help.

 
 
 

Traffic Control traffic unexpectedly dipping

Post by Washington Rats » Sun, 05 Dec 2010 07:58:34






> >>> Looking at the statistics for the classes, I can see that all of the
> >>> queues, including the default queue, stop sending data during this
> >>> period. ?The data I am seeing from the default traffic was from data
> >>> previously buffered up in our system.

> >> I would test without using a default class. Just make a catch all tc
> >> filter with a prio/pref higher than any others (1 is first for filters
> >> not 0) to direct ip traffic to 9999.

> > I just tried that and the higher priority classes end up losing
> > packets.

> But does it fix the everything stops problem?

> Maybe there is some other issue with the setup/tests that are causing this.

> Posting more details of the hardware, tests, results and the most
> simplified/stripped down htb setup that shows the problem would help.

I added SFQ beneath the classes, as described in Section 9.5.5.1 on
page 65 of the Linux Advanced Routing & Traffic Control HOWTO (Version
1.43), and have not seen the problem occur again.  I watched it for 30
minutes.  I was seeing it happen in a few minutes.  I will continue to
test to verify it is gone, but this looks promising.
 
 
 

1. Should traffic control root qdisc & child class limit traffic?

I noticed that before I add client leaf classes that the value I set
as the ceiling has no effect. For example, after putting in the
following commands:

tc qdisc add dev sat0 root handle 9997 htb default 9999 r2q 2
tc class add dev sat0 parent 9997: classid 9997:9998 htb rate
500000bit ceil 500000bit

I can send over 3000000bits/sec with none of it being dropped. It
looks like the 500000bit ceiling I defined is ignored. After I add
client leaf classes below the root, I then see that packets are
dropped when I try to exceed the ceiling.

Is this behavior expected?

2. Some X weirdness

3. looking for a traffic LAN sniffer, able to archive traffic

4. pnptools-1.8

5. PPP: Does outgoing traffic slow incoming traffic?

6. Alpha always boots single user

7. setting an interface for up traffic and a second for down traffic

8. printer/network problem, please help!

9. DIP killed by heavy traffic ?!?

10. traffic control on host

11. netlink: traffic control and NETLINK_FIREWALL sockets

12. Traffic control of input ip packets

13. Answer: - Traffic Control, Bandwidth Throttling, Qos READ THIS!!