remote vs local vs NFS kernel compile (long)

remote vs local vs NFS kernel compile (long)

Post by Tim Moor » Fri, 19 May 2000 04:00:00



Someone posted a question about compiling a kernel by exporting a slow
machine's kernel build tree to a faster machine.  For me the NFS export
wasn't very efficient so I began using a remote build/copy back process as
follows.  Both systems are RH 6.2 base with 2.2.16 kernels.

Architecture
------------
The slow machine is a DELL 486/DX-100, ISA bus with 2-NE2000 10bT NICs,
2-DTC fast parallel ports, a 2GB IDE drive and 40MB memory used as a
firewall, IP masquerade gateway and print server.  Performance is pretty
well summed up by its hdparm(8) output:

/dev/hda:
 Timing buffer-cache reads:   128 MB in  7.34 seconds =17.44 MB/sec
 Timing buffered disk reads:  64 MB in 27.40 seconds = 2.34 MB/sec

The fast machine is a slightly more modern ASUS P3B, 37MHz PCI bus, 525MHz
Celeron, 256MB/CL2/8ns SDRAM memory and a NE2000 10bT connected by a switch
to the slow machine.  Compared with the DELL:

/dev/hda:
 Timing buffer-cache reads:   128 MB in  1.28 seconds =100.00 MB/sec
 Timing buffered disk reads:  64 MB in  3.38 seconds = 18.93 MB/sec

Key Point: The fast machine is significantly faster than the slow machine.
6x memory, 8x disk, 10x CPU in this case.  Most likely the point of
diminishing return is about an overall 2x difference.

Compilation Method
------------------
1. The remote kernel source tree is created on the fast machine's local
disk.  Ideally the location is not the local machine's /usr/src.  In my
case a partition /big/dell/usr/src is mounted on a separate disk far from
the local system's /usr/src.  Think "preemptive risk management".

2. The remote kernel is configured in the usual way:  # cd
/big/dell/usr/src/linux; make menuconfig.  Remember to select kernel
options for the remote machine.  It's easy to context slip and think in
terms of what's in your visual field (eg- the local machine) and get
confused between the two.

3. The 'make dep; make clean; make bzImage; make modules' steps are also
executed in the usual way.

4. The 'make modules_install' step is different.  Use the INSTALL_MOD_PATH
flag to pass the remote machine's root to make:

# make modules_install INSTALL_MOD_PATH=/net/dell

The modules end up on the DELL's /lib/modules rather than locally.  Use
automount, amd or an NFS export from the remote machine.  With a 10Mb
network they are all equally slow.  Keep in mind any security issues in
your particular environment given that root needs to do the writing.

5. To move the new kernel, I again use automount to copy bzImage (as
vmlinuz) and System.map to /net/dell/boot

6. Rerun lilo on the remote system.

7. Optionally tar the source tree up and move it to the remote system.
This keeps the kernel and source in sync in case you do need a local slow
machine compile.  rsync, a selective tar or 'find -newer' would be faster
but more complex to set up.  In general go for the least complex solution.

Results
-------
Native compile on the slow machine, local disk: ~37m
Remote compile on the fast machine via NFS:     ~11m30s

Native compile on the fast machine, local disk: ~04m45s
Copy the source tree back via NFS (optional):   ~03m45s

Examples
--------
1. Build script running on the fast machine on local disk.  kernel, modules
moved to slow machine via NFS.


building 2.2.16pre3 for dell...
dep =======
13.03user 1.62system 0:14.91elapsed 98%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (63572major+21178minor)pagefaults 0swaps
clean =======
0.24user 0.21system 0:00.44elapsed 101%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (5046major+1046minor)pagefaults 0swaps
bzImage =======
Root device is (3, 5)
Boot sector 512 bytes.
Setup is 3420 bytes.
System is 502 kB
217.52user 12.72system 4:00.27elapsed 95%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (295439major+328841minor)pagefaults 0swaps
modules =======
26.85user 1.78system 0:30.24elapsed 94%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (56054major+47341minor)pagefaults 0swaps
modules_install =======
Installing modules under /net/dell/lib/modules/2.2.16pre3/block
Installing modules under /net/dell/lib/modules/2.2.16pre3/net
Installing modules under /net/dell/lib/modules/2.2.16pre3/ipv4
Installing modules under /net/dell/lib/modules/2.2.16pre3/fs
0.20user 0.09system 0:00.76elapsed 37%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (3653major+1292minor)pagefaults 0swaps
./arch/i386/boot/bzImage -> /net/dell/boot/vmlinuz-2.2.16pre3
./System.map -> /net/dell/boot/System.map-2.2.16pre3
/net/dell/boot /big/dell/linux
vmlinuz -> vmlinuz.old
System.map -> System.map.old

fix up any soft links in /boot and rerun lilo on dell

started:        Wed May 17 23:34:19 PDT 2000
finished:       Wed May 17 23:39:07 PDT 2000

2. copy back the source tree.


|\
? (cd /dell/usr/src; tar xpBbf 32 -)
0.20user 1.26system 3:42.78elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (21099major+34minor)pagefaults 0swaps

3. Same build script running on the fast machine with slow machine NFS
export.  kernel, modules moved to slow machine via NFS.


building 2.2.16pre3 for dell...
dep =======
14.12user 6.82system 4:03.33elapsed 8%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (63588major+21535minor)pagefaults 0swaps
clean =======
0.36user 0.51system 0:18.36elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (5051major+1060minor)pagefaults 0swaps
bzImage =======
Root device is (3, 5)
Boot sector 512 bytes.
Setup is 3420 bytes.
System is 502 kB
219.22user 33.49system 5:19.55elapsed 79%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (293529major+329136minor)pagefaults 0swaps
modules =======
27.30user 5.25system 0:42.79elapsed 76%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (55836major+47423minor)pagefaults 0swaps
modules_install =======
Installing modules under /net/dell/lib/modules/2.2.16pre3/block
Installing modules under /net/dell/lib/modules/2.2.16pre3/net
Installing modules under /net/dell/lib/modules/2.2.16pre3/ipv4
Installing modules under /net/dell/lib/modules/2.2.16pre3/fs
0.27user 0.17system 0:01.68elapsed 26%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (3630major+1284minor)pagefaults 0swaps
./arch/i386/boot/bzImage -> /net/dell/boot/vmlinuz-2.2.16pre3
./System.map -> /net/dell/boot/System.map-2.2.16pre3
/net/dell/boot /dell/usr/src/linux
vmlinuz -> vmlinuz.old
System.map -> System.map.old

fix up any soft links in /boot and rerun lilo on dell

started:        Thu May 18 02:35:47 PDT 2000
finished:       Thu May 18 02:46:17 PDT 2000

4. General kernel compile speed hint.  Redirect stdout to /dev/null for
make dep, clean, bzImage and modules (eg- 'make bzImage > /dev/null').
Warnings and fatal errors will still be visible.
--
timothymoore    bigfoot    com

 
 
 

1. Linux vs OS2 vs NT vs Win95 vs Multics vs PDP11 vs BSD geeks

        Every machine and operating system has got its useful
purpose...

        I see no point in argueing with people which OS is better, and
which is worse, and what will survive and what wont...

        The bottom line is obviously the best OS is the one that make
the end user most productive.    Ive used quite a variety of software
from intel, ibm, MS, sun, GNU, DEC/compaq, etc,   and everything OS
has got its UPz and DOWnz, so depending on what you want to do with it
yer machine, probably determines what OS you run.

        So lets cut to the chase -  OS bashing is a waste of time,
and most of the time I'd say the person putting it down just hasn't
seen that particular OS's potential,  or should I say speciality....

      Hell,  Plan 9 has even got some interesting features.. <snicker>

       And all PC users know,  that no matter what use on a day to day
basis on the PC, that one day you will need to boot good ole ancient
DOS to do something...

2. SEARCH IT!!! http://users.why.net/flam/search.html

3. Perfomance: tar vs ftp vs rsync vs cp vs ?

4. sharing a Qic 80 with windows

5. Slackware vs SuSE vs Debian vs Redhat vs ....

6. lp under Solaris2.5

7. DOS vs. Windows vs. Mac vs. Unix vs. NS

8. How do you repair the MBR after uninstalling linux?

9. KDE vs. Openlook vs. Xfree86 vs. MetroX vs. CDE

10. Redhat vs Debian vs Yggdrasil vs Caldera vs ...

11. Linux/alpha vs. 2.4.20 and ISO9660 vs long file names

12. DBM vs. flatfile vs. LONG line in /etc/group?

13. NFS vs SAMBA vs ??: what's the right approach?