Breaking free of packaging systems (code + request for comments)

Breaking free of packaging systems (code + request for comments)

Post by Adam J. Richt » Thu, 14 May 1998 04:00:00



        Over the last couple of years, the GNU/Linux user community
has been divided and constrained by "packaging systems" that give
users the file installation tracking that they want, but require them
to install all of their software in a particular distributor's package
format if they do not want to break the software tracking system.

        There is really no need for users to pay such a high price for
file tracking.

        For this reason, we at Yggdrasil are working on a system that
puts some of the common functionality of file tracking systems in a
common software layer and can track installation of files by any
method, provided that the software that implements it is linked
against the standard shared GNU C library.  This is done by trapping
the system calls that modify files.  This way, we can track
installation of files installed by rpm, debian dpkg, stampede, tar
files, or even doing "make install" from a source tree.

        Under this scheme, the Red Hat tools would be aware of all of
the rpm components installed on the system, the Debian tools would be
aware of all of the dpkg components installed on the system, and our
tools would be aware of everything.  This would be a big step forward
for GNU/Linux users, but, clearly, it would be even better if all of
the tools could see all of the software.  Efforts in this direction
with regard to the current RPM central database format have led to a
separate program with a very simple interface that will probably be of
general use to current RPM users.  So, we are releasing a test version
of this separate program in its current form, in the hopes of getting
good technical advice and contributed improvements.

        Well, here it is:

        ftp://ftp.yggdrasil.com/pub/pkg/modify_rpmdb-0.2.tar.gz.

        With this program, you can modify your RPM database to
reflect unpacking and installation of, say, a tar format program,
by doing something like:

           cd /
           tar xfpvz frobnicator-5.3.bin.tar.gz | \
               modify_rpmdb --name=frobnicator --version=5.3 --mode=replace

        There is a manual page, and "modify_rpmdb --help" will show you
the command syntax.

        Please note that this software is in an incomplete, flakey,
experimental form.  Back up the relevant parts of your system, including
your RPM database, before experimenting with this program.

        If you are interested in the larger file installation tracking
system, a snapshot of that work in progress, currently separate from
modify_rpmdb, is available from
ftp://ftp.yggdrasil.com/pub/dist/pkg/filetrack-0.27.tar.gz.  However,
this main part of the software is more complex, and is probably only
of interest if you want to study the code.

        Anyhow, discussion of this design in this newsgroup, and any
other technical contributions would be welcome.

--
Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 205

+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Bernd Paysa » Fri, 15 May 1998 04:00:00



>         Anyhow, discussion of this design in this newsgroup, and any
> other technical contributions would be welcome.

Good idea. However, sometimes I wonder why RPM needs a database at all.
Libraries already have versioning informations attached to the file
name, so for the shared libraries an application needs, all you have to
do is to look into ld.so.cache, or lookup the files in the directories
listed in ld.so.conf.

Other dependencies can be scripting languages (this Tk script neads Tk
4.7.5 or newer, this Perl script needs Perl 5.003 or newer). I suggest
that scripting languages have to be installed the same way as libraries.
My own project, gforth, does. You get a file gforth-0.3.0, and a link
gforth, which links to the latest installed version. So checking for the
right version of the scripting language installed is a directory lookup,
too.

If all packages could also use a consistent way to store data,
installing and uninstalling could be a breeze, even without keeping a
data base (i.e. always use $PREFIX/$PROGRAM-$VERSION to install your
data files, where $PREFIX typically is something like /usr/local/share
or /opt). With a bit discipline, a database IMHO isn't necessary.

--
Bernd Paysan
"Late answers are wrong answers!"
http://www.jwdt.com/~paysan/

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Stefan Monnie » Fri, 15 May 1998 04:00:00



> or /opt). With a bit discipline, a database IMHO isn't necessary.

I've been complaining for a while now that there's no package management that
reduces the usage of a database to its minimum.  A database is not just
unnecessary, it's detrimental since you always end up woth problems where the
database is not consistent with the disk's content.  Typically, if I manually
add/remove packages (or if I add/remove them via the package manager but on
another machine that's locally NFS mounted).

        Stefan

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Kaz Kylhe » Fri, 15 May 1998 04:00:00



Quote:>    Over the last couple of years, the GNU/Linux user community
>has been divided and constrained by "packaging systems" that give
>users the file installation tracking that they want, but require them
>to install all of their software in a particular distributor's package
>format if they do not want to break the software tracking system.

I have never used them myself. They are nonsense. The true UNIX spirit
requires you to get the sources and compile it.

I have friends who have ended up with useless systems in which nothing
worthwhile can build, as a result of applying binary packages and
patches. They have all kinds of library problems as well and
can't seem to be able to compile anything.

Funny, I have no such problems with my three year old Slackware
installation. Everything runs, and I can get the latest version of
my favorite freeware and build it without problems.

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Bob Tenne » Fri, 15 May 1998 04:00:00



 >
 >>       Over the last couple of years, the GNU/Linux user community
 >>has been divided and constrained by "packaging systems" that give
 >>users the file installation tracking that they want, but require them
 >>to install all of their software in a particular distributor's package
 >>format if they do not want to break the software tracking system.
 >
 >I have never used them myself. They are nonsense. The true UNIX spirit
 >requires you to get the sources and compile it.
 >
 >I have friends who have ended up with useless systems in which nothing
 >worthwhile can build, as a result of applying binary packages and
 >patches. They have all kinds of library problems as well and
 >can't seem to be able to compile anything.
 >
 >Funny, I have no such problems with my three year old Slackware
 >installation. Everything runs, and I can get the latest version of
 >my favorite freeware and build it without problems.

You can indeed get into trouble if, for example, you try to
upgrade an rpm package by a newer tgz version.  But if one
is careful not to mix-and-match systems, both approaches can be
used.  When rpms are available, that system is very convenient and
reliable.  

Bob T.

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by David F » Fri, 15 May 1998 04:00:00



> If all packages could also use a consistent way to store data,
> installing and uninstalling could be a breeze, even without keeping a
> data base (i.e. always use $PREFIX/$PROGRAM-$VERSION to install your
> data files, where $PREFIX typically is something like /usr/local/share
> or /opt). With a bit discipline, a database IMHO isn't necessary.

Package managers were invented precisely because not all packages use
a consistent way to store data.
--
David Fox          http://www.cat.nyu.edu/fox            xoF divaD

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Christopher Brow » Sat, 16 May 1998 04:00:00


On Thu, 14 May 1998 13:04:20 +0200, Bernd Paysan



>>         Anyhow, discussion of this design in this newsgroup, and any
>> other technical contributions would be welcome.

>Good idea. However, sometimes I wonder why RPM needs a database at all.
>Libraries already have versioning informations attached to the file
>name, so for the shared libraries an application needs, all you have to
>do is to look into ld.so.cache, or lookup the files in the directories
>listed in ld.so.conf.

>Other dependencies can be scripting languages (this Tk script neads Tk
>4.7.5 or newer, this Perl script needs Perl 5.003 or newer). I suggest
>that scripting languages have to be installed the same way as libraries.
>My own project, gforth, does. You get a file gforth-0.3.0, and a link
>gforth, which links to the latest installed version. So checking for the
>right version of the scripting language installed is a directory lookup,
>too.

>If all packages could also use a consistent way to store data,
>installing and uninstalling could be a breeze, even without keeping a
>data base (i.e. always use $PREFIX/$PROGRAM-$VERSION to install your
>data files, where $PREFIX typically is something like /usr/local/share
>or /opt). With a bit discipline, a database IMHO isn't necessary.

These are all well and nice things; they require that certain
disciplines be imposed onto software packages.

Unfortunately, there is not a uniform set of disciplines used by
different packages.

- Some software puts configuration information in /etc
- Other software puts configuration information in /usr/lib
- Still other software puts configuration information in /var (qmail
comes to mind)

- Binaries might be in /usr/bin, /bin, /usr/sbin.
- Utility components *THAT ARE STILL PROGRAMS* might hide in /usr/lib.
(MH comes to mind)

- Documentation might be in /usr/doc
- Info files might be in /usr/info

It is an attractive idea to be able to query software to see where *all*
of the relevant pieces are, including configuration, programs, and
documentation.  

And that's not something that pops obviously out of any one place.
Unless we create the "configuration database/registry."

The other benefit to having the information in a database is that since
a database can have multiple indices, we can readily build efficient
queries relating to dependancies.
--
"On the Internet, no one knows you're using Windows NT"


 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Stefan Monnie » Sat, 16 May 1998 04:00:00



> It is an attractive idea to be able to query software to see where *all*
> of the relevant pieces are, including configuration, programs, and
> documentation.  

> And that's not something that pops obviously out of any one place.
> Unless we create the "configuration database/registry."

Well, on my depot system, I can see all the packages installed with:

        % ls /softs/

I can then see all the files of a particular package with:

        % cd /softs/<pkg>; find -type f

and I can see where a specific file comes from with

        % ls -l /usr/local/bin/emacsclient
        lrwxrwxrwx   1 root     root           33 Apr  3 14:13 /usr/local/bin/emacsclient -> /softs/emacs-20.2/bin/emacsclient
        %

Look, ma, no database !

Quote:> The other benefit to having the information in a database is that since
> a database can have multiple indices, we can readily build efficient
> queries relating to dependancies.

But the database should be tightly bound to the actual package so that it
cannot get out of sync unless you really mess up.  My /softs thing doesn't give
me any dependency information, but that doesn't mean that I need to have a
separate /var/lib/rpm database.  Why not have /softs/<pkg>/.depend instead ?

Actually, I could live with /var/lib/rpm as long as it's only used for speed
purposes rather than for correctness: it might allow to quickly list everything
that depends on `perl-5.004', but it should be possible to completely
regenerate the database from the filesystem itself (from the horse's mouth).

        Stefan

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Sam Trenhol » Sat, 16 May 1998 04:00:00


It's good to see Yggdrasil is still doing things.  I know people that
considered the Plug-n-Play Yggdrasil Linux the best distro out there, and
I am looking forward to a version that is current with where Linux is in
1998.

As for tar.gz-vs-a packaging system, this is an endless thread [1].

A packaging system is a step upwards, since it allows easier installing
and removal of programs.  If you need to install .tar.gz files, a little
disclipline with installing such files in /usr/local instead of /usr or
wherever else will stop things from being a mess.

- Sam

[1] http://linux.samiam.org/linux/endless_threads.html

--
Unique Linux information: http://linux-rules.samiam.org/linux/linux_links.html
  I reserve the right to make public any mail I receive in response to this

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Tim Smi » Sun, 17 May 1998 04:00:00



>I can then see all the files of a particular package with:

>    % cd /softs/<pkg>; find -type f

What if the package puts some files outside of /softs, such as man
pages?

--Tim Smith

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Stefan Monnie » Sun, 17 May 1998 04:00:00




> >       % cd /softs/<pkg>; find -type f

> What if the package puts some files outside of /softs, such as man
> pages?

Well, packages *don't* put things outside of /softs/<pkg>.
Manpages are installed in /softs/<pkg>/man and available via
/usr/local/man, as usual.

        % l /usr/local/man/man1/depot.1
        lrwxrwxrwx   1 root     root           34 Mar 25 13:42 /usr/local/man/man1/depot.1 -> /softs/depot-5.13/man/man1/depot.1
        %

There are some subtle issues, but manpages are not among them.
Config and info files are more messy.

        Stefan

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Chris Sai » Sun, 17 May 1998 04:00:00



> There are some subtle issues, but manpages are not among them.
> Config and info files are more messy.

An packaging methodology similar to yours is employed by the College of
Computer Science at Northeastern University (in Boston, MA).  Information
on their SoftInst project can be found at
  * http://www.ccs.neu.edu/groups/systems/proj/SoftInst/

Cheers,
=C=

--

PGP Public Key is available at http://www.wtower.com/~csaia/personal/pubkey.asc
| A!JW223 YK+++!/+++^/+++"/+++'/ir WK++ DT+ PI+ BR++^ MM++  PFeather XSire P+ |
|    I++++ Dswiad $+++dap{c} Vrm{jt} T212/Pinky E77 EeH5 EeeH2 H17 Ay75 M     |

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Christopher Brow » Wed, 20 May 1998 04:00:00


On 16 May 1998 16:28:01 -0400, Stefan Monnier




>> >   % cd /softs/<pkg>; find -type f

>> What if the package puts some files outside of /softs, such as man
>> pages?

>Well, packages *don't* put things outside of /softs/<pkg>.
>Manpages are installed in /softs/<pkg>/man and available via
>/usr/local/man, as usual.

And manual pages are of course not the "subtle" issue.

Quote:>There are some subtle issues, but manpages are not among them.
>Config and info files are more messy.

Precisely, and this is the unfortunate weakness with Depot and anything
like it.

It's like "belling the cat;" if you can force everyone to clean up all
their software so that there are no messy bits, (e.g. - "packages
*don't* put things outside of /softs/<pkg>"), then Depot works.

Unfortunately, there are enough useful pieces of software that *don't*
follow the rules as to break the forcible assumption that "Packages put
all their components in /softs/<pkg>".

Well-behaved software shouldn't break that rule, but not all software is
"well-behaved software."

--
"[In 'Doctor' mode], I spent a good ten minutes telling Emacs what I
thought of it.  (The response was, 'Perhaps you could try to be less
abusive.')" (By Matt Welsh)

 
 
 

Breaking free of packaging systems (code + request for comments)

Post by Christopher Brow » Wed, 20 May 1998 04:00:00


On 16 May 1998 16:28:01 -0400, Stefan Monnier




>> >   % cd /softs/<pkg>; find -type f

>> What if the package puts some files outside of /softs, such as man
>> pages?

>Well, packages *don't* put things outside of /softs/<pkg>.
>Manpages are installed in /softs/<pkg>/man and available via
>/usr/local/man, as usual.

And manual pages are of course not the "subtle" issue.

Quote:>There are some subtle issues, but manpages are not among them.
>Config and info files are more messy.

Precisely, and this is the unfortunate weakness with Depot and anything
like it.

It's like "belling the cat;" if you can force everyone to clean up all
their software so that there are no messy bits, (e.g. - "packages
*don't* put things outside of /softs/<pkg>"), then Depot works.

Unfortunately, there are enough useful pieces of software that *don't*
follow the rules as to break the forcible assumption that "Packages put
all their components in /softs/<pkg>".

Well-behaved software shouldn't break that rule, but not all software is
"well-behaved software."

--
"[In 'Doctor' mode], I spent a good ten minutes telling Emacs what I
thought of it.  (The response was, 'Perhaps you could try to be less
abusive.')" (By Matt Welsh)

 
 
 

1. 64-bit packing code: request for comments

Hi.  I am trying to come up with a way to pack and unpack integers of
unknown size into a cross-platform format.  I plan to use a 64-bit
big-endian format, but it could be little-endian too.  It's 64-bits
because that's as big as the data should get.

I've come up with a couple of routines that can be used like in this
example, where a struct stat's member is packed and unpacked:

    pack_int(istat->size, sb->st_size);

    sb->st_size = unpack_int(istat->size);

Here's the code I've come up with:

  #include <limits.h>
  #define ELC_INT64_SIZ (64 / CHAR_BIT) /* eight for eight-bit char systems */

  typedef unsigned char packed[ELC_INT64_SIZ];

  static unsigned long unpack_int(packed p)
  {
      int i, end;
      unsigned long n = 0;

      for (i = 0, end = sizeof(p); i < end; ++i)
        n |= (((unsigned long) p[i]) >> i) & UCHAR_MAX;

      return n;
  }

  static void pack_int(packed p, unsigned long n)
  {
      int i, end;

      for (i = 0, end = sizeof(p); i < end; ++i)
        p[i] = (n >> i) & UCHAR_MAX;
  }

I'm interested in correctness first, then simplicity, and then
efficiency.  I'm hoping that the compiler will significantly optimize
the above code such that it performs well, but since I just came up
with it, I haven't profiled it or anything.  My question is, does
anyone have any comments on this code, suggestions, or ideas?Z

--
--Ed L Cashin            |   PGP public key:

2. How retire hard isk

3. alternate payment method for "free solaris" package requested

4. netcfg Help for DSL Access

5. Request Comments on Legacy System compatibility

6. HOWTO change special device label?

7. mail-order system from netexpress - request comments

8. changing resolution in redhat 6.0

9. Request for comments on system specs...

10. File request: /etc/rc.config for a broken Redhat 6.0 system

11. Request for TR tracking system code

12. Source code for System-V packaging.

13. CFS 1.1.2 Unix encrypting file system source code available (free)