How does one produce a CD set from a sources mirror?

How does one produce a CD set from a sources mirror?

Post by Tim Shuttlewor » Sat, 08 Nov 1997 04:00:00



Hi,

Someone out there must have experience of this, so maybe they can help me
out. I'm trying to create a 2 CD set of FreeBSD 2.2.5 on a machine which has
a full mirror of the sources (it's tracking the 2.2-stable branch). It also
has full NFS access to the CVS repository mirror, the ports collection
etc. However, I can't find anything that gives me a clear description of what
I should be doing to make a CD release from all this stuff. I want the CDs
for my home machine which doesn't have a fast network connection (and I don't
fancy installing FreeBSD over a 28.8k modem link!).

What I've tried so far is: free up just over 600 megs of disk space on a
machine which NFS exports it to the machine I want to make the release
on as /tmp/release. Then I've done:

     cd /usr/src/release
     make -k release BUILDNAME=2.2.5-RELEASE CHROOTDIR=/tmp/release \
             CVSROOT=/usr/local/cvs > ~/release.out 2>&1 &
     tail -f ~/release.out

The make process runs for two days (!!!), creating a directory tree, checking
out all of the sources into /tmp/release/usr/src, apparently compiling
everything (including the packages). I then end up with a directory in
/tmp/release looking like:

.cshrc          bin             lkm             root            tmp
.profile        bootstrap       mk              sbin            usr
COPYRIGHT       dev             mnt             stand           var
R               etc             proc            sys

Glancing at the Makefile (in /usr/src/release) I assumed that I was supposed
to end up with a /tmp/release/R/cdrom directory containing the CD images
(ready to have mkisofs run on them), but /usr/tmp/release only contains a
"stage" sub-directory. I tried "make ftp.1" and "make cdrom.1", but it
doesn't do the chroot to /tmp/release that I expected, and bombs out
complaining that it can't cd into /R/stage. The Makefile is rather daunting,
and although if no-one will help me out I'm prepared to tackle it, I thought
I'd ask here and hopefully avoid that nightmare!

There were a few things that failed to compile (partly because I didn't have
the full ports collection mounted), but there was plenty of disk space
available. Also, the machine which is exporting the disk space is running a
rather flaky version of 3.0, and fell over a couple of times during the
process (although the NFS mounts recovered after the reboot); I assume that
this lead to some files being skipped/corrupted. Unfortunately, there's so
much output from the make command that it's hard for me to know where the
critical failure is. I assume that the CDs should be made as part of the
normal make process?

So, before I subject myself to another two days of irritating disk clicking,
might I ask,

    a) Am doing approximately the right thing so far? If not, where am I
       going wrong.

    b) After I get a proper build of /tmp/release/R/cdrom, should I just
       be able to do something like:

            /usr/share/examples/worm/makecdfs.sh -b "FreeBSD 2.2.5" \
                  /tmp/release/R/cdrom/disc1 /place_i_want_the_image

       for each of the two discs in the set?

    c) Is there some way to "short-cut" the process, given I also have
       ftp and/or NFS access to a mirror of the FreeBSD ftp repository? It
       seems a bit pointless re-compiling all the sources when there's a copy
       of the compiled and packaged binaries just a mount command away.

    d) Is this documented somewhere? I have grepped the handbook and the
       FAQ to no avail. I'm sure I'm not the only one who wants to create
       their own CDs...

        Thanks,
                Tim.
--
                          Dr Tim Shuttleworth
                       Network Support Programmer
                          Manchester Computing

 
 
 

How does one produce a CD set from a sources mirror?

Post by Jordan K. Hubbar » Sat, 08 Nov 1997 04:00:00


[posted and mailed]


>      cd /usr/src/release
>      make -k release BUILDNAME=2.2.5-RELEASE CHROOTDIR=/tmp/release \
>              CVSROOT=/usr/local/cvs > ~/release.out 2>&1 &
>      tail -f ~/release.out

> The make process runs for two days (!!!), creating a directory tree, checking
> out all of the sources into /tmp/release/usr/src, apparently compiling
> everything (including the packages). I then end up with a directory in
> /tmp/release looking like:

Actually, it doesn't build the packages.  That's done manually still, as
is the XFree86 distribution.  Anyone seeking to make their own FTP/CDROM
distribution will have to copy those components manually into the
relevant CD build directories.  I used to put both on CD #1 but the
increasing size of the packages collection has made it necessary to
split it across multiple CDs in 2.2.5.

Quote:> Glancing at the Makefile (in /usr/src/release) I assumed that I was supposed
> to end up with a /tmp/release/R/cdrom directory containing the CD images
> (ready to have mkisofs run on them), but /usr/tmp/release only contains a
> "stage" sub-directory. I tried "make ftp.1" and "make cdrom.1", but it

Well, since 600MB isn't nearly enough for a full release, I can only
assume that you simply ran out of space during the build and didn't see
it somehow.  This directory should actually look like this:


cdrom   ftp     stage

disc1   disc2

Quote:> There were a few things that failed to compile (partly because I didn't have
> the full ports collection mounted), but there was plenty of disk space

You don't need the ports collection, you simply need a *full* CVS
repository which contains $CVSROOT/ports as well as $CVSROOT/src.

Quote:> available. Also, the machine which is exporting the disk space is running a
> rather flaky version of 3.0, and fell over a couple of times during the
> process (although the NFS mounts recovered after the reboot); I assume that

Ah.  That would explain a few things.  When that happens, you really
have no choice but to hand-edit ${CHROOTDIR}/mk and eliminate all the
stuff which has been run already - usually you just have to remove the
stuff for moving the libraries away into another area and edit out the
"make world" pass if it's already been run successfully, leaving the
final "make doRELEASE" intact (and it will keep track of its progress by
touching files as each "stage" completes, so it can be safely resumed
unlike the previous stuff).

Quote:>     a) Am doing approximately the right thing so far? If not, where am I
>        going wrong.

Well, modulo the above advice, I'd say you're also doing it on far too
slow a machine if it's taking you 5 days. :-)  I can generate a full
release here in just over 8 hours, 4.5 hours if I use a CCD array of 5
quantum drives as /usr/src and /usr/obj.  The problem may also simply be
NFS, which I defintely wouldn't use to do this kind of thing - life is
too short and 4GB drives are too cheap to make that anything I'd ever
consider doing. :)

--
- Jordan Hubbard
  FreeBSD core team / Walnut Creek CDROM.

 
 
 

How does one produce a CD set from a sources mirror?

Post by J Wuns » Thu, 20 Nov 1997 04:00:00



>  Then I've done:

>      cd /usr/src/release
>      make -k release BUILDNAME=2.2.5-RELEASE CHROOTDIR=/tmp/release \
>         CVSROOT=/usr/local/cvs > ~/release.out 2>&1 &
>      tail -f ~/release.out

That's basically right.  Well, you need to set the CVS tag, too, IIRC.
(I'm normally editing the Makefile, as opposed to set the variables
from the command line, but that's merely a matter of taste.)

Quote:> The make process runs for two days (!!!), ...

Oh, that's rather *ic then.  On my P133 with local disks, it
usually takes ~ 6 hours.  (That's without the ports tree though, due
to the numerous small files a checkout of the ports itself pokes
around for more than an hour.)

Quote:> Glancing at the Makefile (in /usr/src/release) I assumed that I was supposed
> to end up with a /tmp/release/R/cdrom directory containing the CD images
> (ready to have mkisofs run on them), but /usr/tmp/release only contains a
> "stage" sub-directory.

That means the build process ended up with some sort of an error, and
you need to spot this in "release.out".  It's OK to run make -k (i'm
also doing it this way), but you gotta care for the errors yourself
then.

Quote:>  Unfortunately, there's so
> much output from the make command that it's hard for me to know where the
> critical failure is.

Well, you can probably ignore all errors that are within the first 20
or so percent of the output, stemming from the bootstrapping `make
install' into the chroot tree.  Unless some utility failed to install
there that's really critical for the later build process, all that
goes wrong that early is bening.

Simply search for the word "continuing" in release.out, that's what
make drops there if it detects a fatal error that would normally abort
the build, but continues due to "make -k".

Quote:> I assume that the CDs should be made as part of the
> normal make process?

Yep.

Quote:>     b) After I get a proper build of /tmp/release/R/cdrom, should I just
>        be able to do something like:

>             /usr/share/examples/worm/makecdfs.sh -b "FreeBSD 2.2.5" \
>                   /tmp/release/R/cdrom/disc1 /place_i_want_the_image

I haven't used this script myself, but i think that looks right.

Quote:>     c) Is there some way to "short-cut" the process, given I also have
>        ftp and/or NFS access to a mirror of the FreeBSD ftp repository? It

No, without Makefile tweaking, you need the CVS repository.

There's a "rerelease" target, but this one also wants to CVS update
first.  Of course, you can chroot /tmp/release yourself, edit the /mk
script, perhaps remove the /usr/release/obj/release.[1-8] targets you
wanna force to rebuild, and run /mk manually.  That's how i normally
continue after a failed build attempt.

Quote:>     d) Is this documented somewhere?

In /usr/src/release/Makefile.  Sorry, this is mainly the dumping
ground for the release engineers, and has never been intended to be a
`supported' user interface.  In the course of a release preparation,
we (the release engineers, past and current :) usually cycle through
`make release' a number of times, fixing all the problems that pop up
since something broke the process, and commit changes to that area.
There's not guarantee that this process will work all the time.  This
area has mainly been integrated into CVS in order to allow sharing the
load of the release engineering work.  Remember, Walnut Creek is
sharing almost all of their secrets how to prepare a high-quality (and
controlled quality) FreeBSD release CD-ROM set.  It's not that common
that CD manufacturers allow sharing this secret... and it's IMHO also
a good proof of the flexibility of make(1).  Whenever someone claims
that something like a Borland IDE were better, ask him how to prepare
a complete CD-ROM with it, with all the documentation and stuff and
kitchensink. ;-)

--
cheers, J"org


Never trust an operating system you don't have sources for. ;-)

 
 
 

1. Why set read policy to "first" for ufsdump of mirror on Sol10?

In the Solaris Volume Manager Administration Guide (in the
Solaris 10 System Administrator Collection),
http://docs.sun.com/app/docs/doc/816-4520/6manpieqb?a=view
it states:

"You can use the ufsdump command to backup the files of a mounted
filesystem residing on a RAID-1 volume. Set the read policy on the volume
to "first" when the backup utility is ufsdump. This improves the rate at
which the backup is performed."

 And

"After the ufsdump command is done, set the read policy on the mirror to
"roundrobin.""

This may have been the same for previous versions of Solaris and I just
never noticed, but my question is why would this improve the rate?  Why is
it faster/better to limit reads to the first disk?

Matt S

2. ppp1 -> ppp0. Help.

3. CD-ROM mount needs source <--> source needs CD-ROM mount

4. FS: SCO Unix OpenServer Enterprise v5.0.0 5 user

5. lpr producing mirrored outputs.

6. /dev/* ownership and groupership?

7. No CD unless I use DOS first

8. Insuring sane /dev modes?

9. Setting up Linux/DOS/OS2 on one drive

10. Linux CD set , which one?

11. Linux CD sets , which one?

12. Smoke and Mirrors: mirrored concat / concat mirrors

13. CD set with source to everything