Very Computer

by **William Par** » Sun, 14 Jul 2002 00:31:59

Can anyone give me some insight as to why
- if I copy 20MB over NFS, I get 12MB/s (no complain here), but
- if I copy 500MB over NFS, I get 5MB/s ?
- this happens whether I copy single 500MB tarball or directory
tree worth 500MB.

Hardware is 2 identical Linux boxes with
- Abit VP6 motherboard (VIA 694X/686B), dual-P3, 3c905CX,
- cheap 100Mbit switch (SmartLink 5-port)

Any insight or direction to search would be appreciated. 'hdparm -t' gives
26MB/s and 35MB/s, so that looks normal.

--

8-CPU Cluster, Hosting, NAS, Linux, LaTeX, python, vim, mutt, tin

by **Cyril BONT** » Sun, 14 Jul 2002 00:54:03

Hello,

> Can anyone give me some insight as to why
> - if I copy 20MB over NFS, I get 12MB/s (no complain here), but
> - if I copy 500MB over NFS, I get 5MB/s ?
> - this happens whether I copy single 500MB tarball or directory
> tree worth 500MB.

how do you measure this speed rate ? is this with a low-level network
analyzer or from your shell (or with the command you use to copy your
datas)?

If you measure it from your copy command, it's probably normal :
the speed rate is calculated with 2 elements :
1. your datas' size
2. the time it takes to copy

But it doesn't take care of the extra-datas needed by each level of protocol
to communicate. The more you have datas, the more you have extra-datas.
So, your speed rate decreases to approach a lower limit value.

--

by **Karl Heye** » Sun, 14 Jul 2002 01:35:17

> Can anyone give me some insight as to why
> - if I copy 20MB over NFS, I get 12MB/s (no complain here), but
> - if I copy 500MB over NFS, I get 5MB/s ?
> - this happens whether I copy single 500MB tarball or directory
> tree worth 500MB.

> Hardware is 2 identical Linux boxes with
> - Abit VP6 motherboard (VIA 694X/686B), dual-P3, 3c905CX,
> - cheap 100Mbit switch (SmartLink 5-port)

The 20Mb one will get cache, so your probably maxing out the link.

run a vmstat 1 on both machines during the copy.

karl.

by **Scott Duckwort** » Sun, 14 Jul 2002 01:44:51

>> Can anyone give me some insight as to why
>> - if I copy 20MB over NFS, I get 12MB/s (no complain here), but
>> - if I copy 500MB over NFS, I get 5MB/s ?
>> - this happens whether I copy single 500MB tarball or directory
>> tree worth 500MB.

>> Hardware is 2 identical Linux boxes with
>> - Abit VP6 motherboard (VIA 694X/686B), dual-P3, 3c905CX,
>> - cheap 100Mbit switch (SmartLink 5-port)

> The 20Mb one will get cache, so your probably maxing out the link.

> run a vmstat 1 on both machines during the copy.

> karl.

Yes, the 20mb file might get cached, but make sure you have DMA enabled on
your computers. If it is not done with the Linux drivers you can do it
with the hdparm program.
--
Scott Duckworth
Computer engineering student and wanna-be know-it-all. ;)

by **William Par** » Sun, 14 Jul 2002 04:14:11

>> run a vmstat 1 on both machines during the copy.
> Yes, the 20mb file might get cached, but make sure you have DMA enabled on
> your computers. If it is not done with the Linux drivers you can do it
> with the hdparm program.

hdparm -- Yes, DMA is enabled.

/dev/hda:
multcount = 16 (on)
I/O support = 1 (32-bit)
unmaskirq = 1 (on)
using_dma = 1 (on)
keepsettings = 0 (off)
nowerr = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 1860/255/63, sectors = 29888820, start = 0

ifconfig -- It looks okey on both side. No errors or collisions.

eth0 Link encap:Ethernet HWaddr 00:01:03:27:12:FA
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1893479 errors:0 dropped:0 overruns:0 frame:0
TX packets:4685765 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:329656451 (314.3 Mb) TX bytes:753288958 (718.3 Mb)
Interrupt:17 Base address:0xdc00

vmstat -- Typical printout of 'vmstat 1' on the receiving side looks like
below. Looks okey, but I'm don't have much experience at this level.

0 1 0 36 3884 2456 76480 0 0 16 0 8793 5470 0 21 78
0 1 2 36 3844 2476 76516 0 0 0 0 9062 5711 0 22 78
0 1 0 36 3836 2508 76412 0 0 8 0 5094 4118 0 7 93
0 1 0 36 3912 2520 76276 0 0 4 0 5647 3934 0 15 85
0 1 1 36 3856 2604 76160 0 0 48 0 4324 3781 0 8 92
0 1 0 36 3852 2608 76040 0 0 4 0 8310 5708 0 18 81
0 1 0 36 3896 2548 76128 0 0 4 12104 6201 3816 0 14 86
0 1 0 36 3904 2480 76132 0 0 4 0 3609 2682 0 6 94
0 1 1 36 3888 2444 76176 0 0 0 20840 1511 805 0 6 94
0 1 2 36 3900 2448 76168 0 0 4 12944 246 26 0 2 98
0 1 1 36 3876 2480 76120 0 0 12 10652 1713 1274 0 6 94
0 1 0 36 3936 2480 76020 0 0 8 0 8054 5209 0 15 85
0 1 0 36 3884 2468 76052 0 0 0 0 9109 5671 0 22 78
0 1 0 36 3844 2440 76132 0 0 4 0 9776 6098 0 21 79
0 1 0 36 3944 2340 76124 0 0 0 0 9265 5897 0 23 77
0 1 2 36 3920 2356 76132 0 0 0 5376 7823 4811 0 18 81
0 1 1 36 3948 2360 76092 0 0 0 25668 711 216 0 5 95
0 1 1 36 3948 2360 76092 0 0 0 19476 273 17 0 2 98
0 0 0 36 6416 2388 76100 0 0 8 1164 5244 3552 0 9 91

After copying several times from both sides, I've noticed that
- 990MB directory tree -> 3MB/s if run on sending box
-> 5MB/s if run on receiving box
I'll chuck this as NFS problem...

--

8-CPU Cluster, Hosting, NAS, Linux, LaTeX, python, vim, mutt, tin

by **Karl Heye** » Sun, 14 Jul 2002 05:18:08

> 0 1 0 36 3936 2480 76020 0 0 8 0 8054 5209 0 15 85
> 0 1 0 36 3884 2468 76052 0 0 0 0 9109 5671 0 22 78
> 0 1 0 36 3844 2440 76132 0 0 4 0 9776 6098 0 21 79
> 0 1 0 36 3944 2340 76124 0 0 0 0 9265 5897 0 23 77
> 0 1 2 36 3920 2356 76132 0 0 0 5376 7823 4811 0 18 81
> 0 1 1 36 3948 2360 76092 0 0 0 25668 711 216 0 5 95
> 0 1 1 36 3948 2360 76092 0 0 0 19476 273 17 0 2 98
> 0 0 0 36 6416 2388 76100 0 0 8 1164 5244 3552 0 9 91

> After copying several times from both sides, I've noticed that
> - 990MB directory tree -> 3MB/s if run on sending box
> -> 5MB/s if run on receiving box
> I'll chuck this as NFS problem...

The CPU is mainly idle. It looks to be very bursty in the block out column
which maybe because your on a journalled fs. If your running on ext3
can you try again with the same fs mounted as ext2. It maybe just an
interaction between NFS and journalling.

karl.

by **William Par** » Sun, 14 Jul 2002 05:39:25

>> 0 1 0 36 3936 2480 76020 0 0 8 0 8054 5209 0 15 85
>> 0 1 0 36 3884 2468 76052 0 0 0 0 9109 5671 0 22 78
>> 0 1 0 36 3844 2440 76132 0 0 4 0 9776 6098 0 21 79
>> 0 1 0 36 3944 2340 76124 0 0 0 0 9265 5897 0 23 77
>> 0 1 2 36 3920 2356 76132 0 0 0 5376 7823 4811 0 18 81
>> 0 1 1 36 3948 2360 76092 0 0 0 25668 711 216 0 5 95
>> 0 1 1 36 3948 2360 76092 0 0 0 19476 273 17 0 2 98
>> 0 0 0 36 6416 2388 76100 0 0 8 1164 5244 3552 0 9 91

>> After copying several times from both sides, I've noticed that
>> - 990MB directory tree -> 3MB/s if run on sending box
>> -> 5MB/s if run on receiving box
>> I'll chuck this as NFS problem...

> The CPU is mainly idle. It looks to be very bursty in the block out column
> which maybe because your on a journalled fs. If your running on ext3
> can you try again with the same fs mounted as ext2. It maybe just an
> interaction between NFS and journalling.

All my filesystem is ext2. Bursty "bi" and "bo" on the receiving side does
trouble me, because it's very uniform on the sending side. Perhaps, it's
IRQ thing...

Thanks for responding, Karl. I appreciate that.

--

8-CPU Cluster, Hosting, NAS, Linux, LaTeX, python, vim, mutt, tin

by **Karl Heye** » Sun, 14 Jul 2002 07:29:15

> All my filesystem is ext2. Bursty "bi" and "bo" on the receiving side does
> trouble me, because it's very uniform on the sending side. Perhaps, it's
> IRQ thing...

possibly, is that like thousands of interrupts a second then. Can you
check /proc/interrupts see where most interrupts are showing up, at
that rate it should be easy to see which IRQ shows up most (besides
timers). Then see what devices are using that IRQ.

karl.

by **William Par** » Sun, 14 Jul 2002 08:36:53

>> All my filesystem is ext2. Bursty "bi" and "bo" on the receiving side does
>> trouble me, because it's very uniform on the sending side. Perhaps, it's
>> IRQ thing...

> possibly, is that like thousands of interrupts a second then. Can you
> check /proc/interrupts see where most interrupts are showing up, at
> that rate it should be easy to see which IRQ shows up most (besides
> timers). Then see what devices are using that IRQ.

As expected, eth0 (on both machines) are the most used, and interrupts are
equally divided between the 2 CPUs. 'eth0' is not shared with any other
devices. Oh well, at least it works...

--

8-CPU Cluster, Hosting, NAS, Linux, LaTeX, python, vim, mutt, tin

by **Karl Heye** » Sun, 14 Jul 2002 08:56:08

>>> All my filesystem is ext2. Bursty "bi" and "bo" on the receiving side does
>>> trouble me, because it's very uniform on the sending side. Perhaps, it's
>>> IRQ thing...

>> possibly, is that like thousands of interrupts a second then. Can you
>> check /proc/interrupts see where most interrupts are showing up, at
>> that rate it should be easy to see which IRQ shows up most (besides
>> timers). Then see what devices are using that IRQ.

> As expected, eth0 (on both machines) are the most used, and interrupts are
> equally divided between the 2 CPUs. 'eth0' is not shared with any other
> devices. Oh well, at least it works...

Make sure that you check all your devices (/proc/pci) but if maybe an
issue with the NFS daemon, you could check the rc1 release or the -aa
release to see if there are updates there.

karl.

by **William Par** » Sun, 14 Jul 2002 09:00:58

>> All my filesystem is ext2. Bursty "bi" and "bo" on the receiving side does
>> trouble me, because it's very uniform on the sending side. Perhaps, it's
>> IRQ thing...

> possibly, is that like thousands of interrupts a second then. Can you
> check /proc/interrupts see where most interrupts are showing up, at
> that rate it should be easy to see which IRQ shows up most (besides
> timers). Then see what devices are using that IRQ.

As expected, eth0 (on both machines) is the most used, and interrupts are
equally divided between the 2 CPUs. 'eth0' is not sharing with any other
devices.

It seems that the machines are not capable of handling harddisk and
ethernet, at the same time. According to HD-LED on the front panel,
- when receiving machine is writing to disk (LED is on steady), the
sending machine blinks, indicating it's not reading at full speed.
- when receiving machine has finished writing (LED off or blinks
occasionally), then the sending machine is reading from disk (LED on
steady).

So, the HD-LED essentially alternates between the 2 machines. This would
explain throughput of only 50%.

--

8-CPU Cluster, Hosting, NAS, Linux, LaTeX, python, vim, mutt, tin

by **Michael Heimin** » Sun, 14 Jul 2002 17:46:20

Quote:> Can anyone give me some insight as to why
> - if I copy 20MB over NFS, I get 12MB/s (no complain here),
> but - if I copy 500MB over NFS, I get 5MB/s ?
> - this happens whether I copy single 500MB tarball or directory
> tree worth 500MB.

> Hardware is 2 identical Linux boxes with
> - Abit VP6 motherboard (VIA 694X/686B), dual-P3, 3c905CX,
> - cheap 100Mbit switch (SmartLink 5-port)

> Any insight or direction to search would be appreciated. 'hdparm
> -t' gives 26MB/s and 35MB/s, so that looks normal.

What about the mount options for the nfs share?

Michael Heiming
--
Remove the +SIGNS case mail bounces.

Very Computer

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??

copy 20MB -> 12MB/s; copy 500MB -> 5MB/s ??