NFS performance til 1.3.97

NFS performance til 1.3.97

Post by Stephan Trebel » Fri, 03 May 1996 04:00:00



Hi,

I recently followed some discussion in this newsgroup, where people
said, the NFS performance of the linux client wasn't soo bad
nowadays. so I upgraded to the 1.3.97 kernel, newest libc, etc.
no performance boost or the like.

1. iozone test with hp-ux server 512 byte blocks

        11193 bytes/second for writing the file
        961996 bytes/second for reading the file

2. same test, with 4096 byte blocks rsize,wsize=4096

        73843 bytes/second for writing the file
        970903 bytes/second for reading the file

Yes, this is faster, than it used to be, but still
it's not really fast. read performance is good, though.
why this difference.

Ciao, Stephan
--
Task:  System Administration Unix (HP-UX, Linux, FreeBSD)

Mail:  Abt. 081, MPI bpC, Am Fassberg, 37077 Goettingen, Germany
Tel:   +49 551 201 1 454

 
 
 

NFS performance til 1.3.97

Post by Klaus Steinberg » Fri, 03 May 1996 04:00:00


Hi,


|> 1. iozone test with hp-ux server 512 byte blocks
|>
|>         11193 bytes/second for writing the file
|>         961996 bytes/second for reading the file
|>
|> 2. same test, with 4096 byte blocks rsize,wsize=4096
|>
|>         73843 bytes/second for writing the file
|>         970903 bytes/second for reading the file
|>
|> Yes, this is faster, than it used to be, but still
|> it's not really fast. read performance is good, though.
|> why this difference.

Linux currently does not merge write requests on the writing side. So if
you write in 512 byte chunks (or even smaller), the write performance goes
down to hell. Any application which uses write calls with small size
chunks will suffer from that. The above results are with iozone with
its default 512byte record size I bet. If you try iozone with a record size
of up to 8192, you will see good NFS write performance. In this case I got
betweeen 500 and 700 kByte/sec from a Linux client with various servers
(HP, MIPS, Linux). On the read side I saw even up to 1000000 Bytes/sec
with an Linux server.

Also you should up the rsize and wsize parameters to 8192, as long as you
have an good ethernet board.

I hope this deficiency will be fixed somewhere in the future.

Sincerly,
Klaus

--
Klaus Steinberger               Beschleunigerlabor der TU und LMU Muenchen
Phone: (+49 89)3209 4287        Hochschulgelaende, D-85748 Garching, Germany

URL: http://www.bl.physik.tu-muenchen.de/~k2/k2.html

 
 
 

NFS performance til 1.3.97

Post by Juergen Ho » Sat, 04 May 1996 04:00:00



> Hi,


> |> 1. iozone test with hp-ux server 512 byte blocks
> |>
> |>         11193 bytes/second for writing the file
> |>         961996 bytes/second for reading the file
> |>
> |> 2. same test, with 4096 byte blocks rsize,wsize=4096
> |>
> |>         73843 bytes/second for writing the file
> |>         970903 bytes/second for reading the file
> |>
> |> Yes, this is faster, than it used to be, but still
> |> it's not really fast. read performance is good, though.
> |> why this difference.

> Linux currently does not merge write requests on the writing side. So if
> you write in 512 byte chunks (or even smaller), the write performance goes
> down to hell. Any application which uses write calls with small size
> chunks will suffer from that.

Yes, and the most important application is ....
           gcc

Because gcc calls the assembler, and as writes blocks << 512 bytes, your
compiler performance goes down to hell!!

So, don't use linux as a development workstation in a network.

Quote:> ...
> I hope this deficiency will be fixed somewhere in the future.

ditto

Sincerly,
jh

------ Institut fuer Theoretische Physik, Universitaet Goettingen ------

 _______________________   ________.--'-`--._____   Tel:  +49 551 399363
|____==================_)  \_'===================`  FAX:  +49 551 399263
       _,--___.-|__|-.______|=====/  `---'          Post: Dr. J. A. Holm
       `---------._          ~~~~~|                       Bunsenstr. 9
                   `-._ -  -  - ,'                        37073 Goettingen
                       \_____,-'                          Deutschland

                        "Tee. Earl Grey. Heiss!"

 
 
 

NFS performance til 1.3.97

Post by Wolfram Glog » Sat, 04 May 1996 04:00:00



>Yes, and the most important application is ....
>           gcc

>Because gcc calls the assembler, and as writes blocks << 512 bytes, your
>compiler performance goes down to hell!!

I don't have the quote handy, but Ian Taylor mentioned that this was
fixed in GNU binutils some time ago -- it doesn't write() every
instruction seperately any more.

Quote:>So, don't use linux as a development workstation in a network.

Well, I have no performance problems in this scenario.

Regards,
Wolfram.

 
 
 

NFS performance til 1.3.97

Post by Mark Ha » Sun, 05 May 1996 04:00:00


just to be different, I'll start out with some hard, cold data.
following is a table of the relevant parts of bonnie, run on
a p5/90 client against a p4/80 server, both using 3c509 cards
on the same non-busy but non-idle local segment.  1.3.90 or so.

        b wr    rewr    b in    seeks
1k      252     128     264     142.6
2k      334     170     361     173.7
4k      424     273     679     226.9
8k      478     286     719     56.6

(character columns are by definition mostly libc performance and useless;
seeks are highly dubious here because it's only an 8M file I'm testing.)

anyway, it's pretty obvious that you want to be at least at rsize/wsize=4096.
note that the server is fairly lame: non-PCI, non-EIDE, delivers ~1.5 MB/s
locally, though it's fully capable of saturating the wire.

Quote:> Because gcc calls the assembler, and as writes blocks << 512 bytes, your
> compiler performance goes down to hell!!

uh, why would you put TMPDIR on NFS?  or fail to notice the -pipe option?

Quote:> So, don't use linux as a development workstation in a network.

NONSENSE!  never expect good performance out of a diskless workstation.
real, live disks are very fast and very cheap; use them!  fighting the
market is an excellent sign of ideology-overdose.

regards, mark hahn.
--

                                        http://neurocog.lrdc.pitt.edu/~hahn/

 
 
 

NFS performance til 1.3.97

Post by Stephan Trebel » Sun, 05 May 1996 04:00:00


Hi,

Quote:>>Because gcc calls the assembler, and as writes blocks << 512 bytes, your
>>compiler performance goes down to hell!!

>I don't have the quote handy, but Ian Taylor mentioned that this was
>fixed in GNU binutils some time ago -- it doesn't write() every
>instruction seperately any more.

So the current development tools in the distributions don't use it?
And why doesn't the same behaeviour occur with a linux NFS server?
I tried FreeBSD, and the client didn't show this behaviour on the
_same_ machine.

What I say is, small block sizes get the performance down, but it
goes to hell with some servers. other client's don't do it. why?
No, I wouldn't say that's the server's fault, if other clients can
handle it.

Quote:>>So, don't use linux as a development workstation in a network.
>Well, I have no performance problems in this scenario.

This sounds interesting, so which kind of setup do you use?
certainly no HP or OSF1 NFS server!

Ciao, Stephan
--
Task:  System Administration Unix (HP-UX, Linux, FreeBSD)

Mail:  Abt. 081, MPI bpC, Am Fassberg, 37077 Goettingen, Germany
Tel:   +49 551 201 1 454

 
 
 

NFS performance til 1.3.97

Post by Joern Rennec » Sun, 05 May 1996 04:00:00



>Yes, and the most important application is ....
>           gcc

>Because gcc calls the assembler, and as writes blocks << 512 bytes, your
>compiler performance goes down to hell!!
>So, don't use linux as a development workstation in a network.

This assumes that /tmp is NFS-mounted. Not exactly a high-performance
setup. Maybe you swap over NFS too?
 
 
 

NFS performance til 1.3.97

Post by Stephan Trebel » Mon, 06 May 1996 04:00:00


Hi,

Quote:>just to be different, I'll start out with some hard, cold data.
>following is a table of the relevant parts of bonnie, run on
>a p5/90 client against a p4/80 server, both using 3c509 cards

   -----                  -----

Quote:>on the same non-busy but non-idle local segment.  1.3.90 or so.

What I complained about, is a Linux NFS client has worse performance
than other OS's on the same machine (newest kernel, newest libc)
with a server on HP-UX or OSF/1. A linux server works, and doesn't
show the same behaviour.

If I exchange the server, it works, but alas, I have got these HP
workstations here, and I want to use them.

Quote:>anyway, it's pretty obvious that you want to be at least at rsize/wsize=4096.

That's what I use, because it increases performance. it's still slower
than other OS's. It has become better within the 1.3.* kernels, but FreeBSD
is still faster.

Quote:>> Because gcc calls the assembler, and as writes blocks << 512 bytes, your
>> compiler performance goes down to hell!!
>uh, why would you put TMPDIR on NFS?  or fail to notice the -pipe option?

I think, he meant, it's a symptom, nothing you couldn't get around.
the problem is not TMPDIR or -pipe, and it doesn't cure the problem.
the assembler e.g. (the last I tried) doesn't use buffered io, so it
writes the resulting object files on the NFS homedir of the user, and
the performance is away. Okay, you can get around it changing all the
rules, to create object files in $TMPDIR, and copying them back
afterwards, but that's a kludge IMHO.

Regards, Stephan
--
Task:  System Administration Unix (HP-UX, Linux, FreeBSD)

Mail:  Abt. 081, MPI bpC, Am Fassberg, 37077 Goettingen, Germany
Tel:   +49 551 201 1 454

 
 
 

NFS performance til 1.3.97

Post by James H Cloos » Wed, 08 May 1996 04:00:00



>> Linux currently does not merge write requests on the writing
>> side. So if you write in 512 byte chunks (or even smaller), the
>> write performance goes down to hell. Any application which uses
>> write calls with small size chunks will suffer from that.

Juergen> Yes, and the most important application is ....  gcc

Then use the -pipe option.  I always do, especially when the src is
NFS mounted....

-JimC

--
James H. Cloos, Jr.     <URL:http://www.jhcloos.com/~cloos/>

LPF,Usenix,SAGE,ISOC,ACLU

 
 
 

NFS performance til 1.3.97

Post by Or » Sun, 12 May 1996 04:00:00



  .
  .
  .

Quote:>Because gcc calls the assembler, and as writes blocks << 512 bytes, your
>compiler performance goes down to hell!!

>So, don't use linux as a development workstation in a network.

   Okay, so maybe I'm spoiled by having the big iron on the server,
but I've seen more slowdown from little processors than from NFS
sloth when I've been doing compiles over the net (with the unhappy
side effect that my Ascend 25 spontaneously reboots from the stress
of dealing with a meg or 10 of traffic.)

   I'd certainly not want to run a relational database over an NFS
connection from a Linux workstation, but for code development it's
not that bad.

                 ____
   david parsons \bi/ A P75 over NFS is faster than a 486/100 on local
                  \/                                             disks.

 
 
 

NFS performance til 1.3.97

Post by Miguel de Icaz » Tue, 14 May 1996 04:00:00


Quote:>Because gcc calls the assembler, and as writes blocks << 512 bytes, your
>compiler performance goes down to hell!!

>So, don't use linux as a development workstation in a network.

Someone from Cygnus reported that this was the case a couple of years
ago, now they use stdio to buffer out output in st.st_blksize chunks.

Miguel.

 
 
 

NFS performance til 1.3.97

Post by Stephan Trebel » Wed, 15 May 1996 04:00:00



Quote:>>So, don't use linux as a development workstation in a network.

>Someone from Cygnus reported that this was the case a couple of years
>ago, now they use stdio to buffer out output in st.st_blksize chunks.

Yes, but it doesn't work, as current libc flushes the stdio buffers on
any seek op. (there are LOTS of seek ops in the bfd code!) Therefore
buffered io doesn't help at all.

Sigh! Stephan
--
Task:  System Administration Unix (HP-UX, Linux, FreeBSD)

Mail:  Abt. 081, MPI bpC, Am Fassberg, 37077 Goettingen, Germany
Tel:   +49 551 201 1 454

 
 
 

NFS performance til 1.3.97

Post by Ronald F. Guilmet » Thu, 30 May 1996 04:00:00




Quote:

>>Because gcc calls the assembler, and as writes blocks << 512 bytes, your
>>compiler performance goes down to hell!!

>>So, don't use linux as a development workstation in a network.

>Someone from Cygnus reported that this was the case a couple of years
>ago, now they use stdio to buffer out output in st.st_blksize chunks.

See also how the macro IO_BUFFER_SIZE is used in GCC's toplev.c file.
(I was the one who asked Stallman to put this in.)

I never have figured out why more use isn't made of this in more of
the system-dependent configuration files for gcc.
--

-- Ron Guilmette, Roseville, CA -------- Infinite Monkeys & Co. ------------

------ Copyright (c) 1996 by Ronald F. Guilmette; All rights reserved. -----

 
 
 

NFS performance til 1.3.97

Post by Stephan Trebel » Fri, 31 May 1996 04:00:00



[performance of gcc/gas on NFS mounted disks.]

Quote:

>>Someone from Cygnus reported that this was the case a couple of years
>>ago, now they use stdio to buffer out output in st.st_blksize chunks.

>See also how the macro IO_BUFFER_SIZE is used in GCC's toplev.c file.
>(I was the one who asked Stallman to put this in.)

>I never have figured out why more use isn't made of this in more of
>the system-dependent configuration files for gcc.

I'm afraid this won't help. gas seems to cause the problem, as it uses
lots of seeks and libc flushes alll buffers on seek, sigh. at the moment
it's just unusable.

Ciao, Stephan

P.S. As I said before, we need either a libc correction, so a seek within
the current io buffer doesn't invalidate the buffer, even if it exceeds the
end of the file, OR a kernel nfs client that does write op merging.
--
Task:  System Administration Unix (HP-UX, Linux, FreeBSD)

Mail:  Abt. 081, MPI bpC, Am Fassberg, 37077 Goettingen, Germany
Tel:   +49 551 201 1 454