Deeply integrating package management?

Deeply integrating package management?

Post by John Jense » Wed, 02 Sep 1998 04:00:00



I've had these ideas lately ... always a bad sign ... about the problems I
have maintaing a rolling Linux system.  Maybe I'll get the hang of it, but
right now I'm having a hard time upgrading libraries and keeping my
applications running.

The ideas that keep popping up are about integrating package management at
a deeper level of the OS.  I see the problem as one of fan-out, in which
files from a package are spread out across the system. One hard way to
manage things would be to keep files in the same places they are now and
track them in a database.  An easier way might be to reorganize the
filesystem, and keep files for a package together and access them in
place.  I'm intrigued by the second solution, in which libraries would be
kept with their h-files and man pages.  I think this would make it easier
to keep multiple revisions of libraries on-line, and ease automation of
the upgrade process.

I wonder if anyone is thinking along these lines, and if any projects are
in progress.  Conversely, perhaps some of you think the current systems
can be extended smoothly to handle upgrade and compatibility problems?

John

 
 
 

Deeply integrating package management?

Post by Peter A Fei » Wed, 02 Sep 1998 04:00:00



> The ideas that keep popping up are about integrating package management at
> a deeper level of the OS.  I see the problem as one of fan-out, in which
> files from a package are spread out across the system. One hard way to
> manage things would be to keep files in the same places they are now and
> track them in a database.  An easier way might be to reorganize the
> filesystem, and keep files for a package together and access them in
> place.  I'm intrigued by the second solution, in which libraries would be
> kept with their h-files and man pages.  I think this would make it easier
> to keep multiple revisions of libraries on-line, and ease automation of
> the upgrade process.

Dpkg (the Debian package manager) does the database approach. Quite
successfully, too. http://www.debian.org/

--
Peter A Fein                                           Summering in SF!
Home: 650-571-6476                                   Work: 650-628-2172

Who's going to pay for this!?!

 
 
 

Deeply integrating package management?

Post by Tristan Wibberle » Wed, 02 Sep 1998 04:00:00




> > The ideas that keep popping up are about integrating package management at
> > a deeper level of the OS.  I see the problem as one of fan-out, in which
> > files from a package are spread out across the system. One hard way to
> > manage things would be to keep files in the same places they are now and
> > track them in a database.  An easier way might be to reorganize the
> > filesystem, and keep files for a package together and access them in
> > place.  I'm intrigued by the second solution, in which libraries would be
> > kept with their h-files and man pages.  I think this would make it easier
> > to keep multiple revisions of libraries on-line, and ease automation of
> > the upgrade process.

That's not difficult, have a system of symbolic links and a file
containing a list of those belonging to each packages (which would be
kept in that packages installation directory).

--
Tristan Wibberley

 
 
 

Deeply integrating package management?

Post by John Jense » Wed, 02 Sep 1998 04:00:00



: >
: >The ideas that keep popping up are about integrating package management at
: >a deeper level of the OS.  I see the problem as one of fan-out, in which
: >files from a package are spread out across the system. One hard way to
: >manage things would be to keep files in the same places they are now and
: >track them in a database.  An easier way might be to reorganize the
: >filesystem, and keep files for a package together and access them in
: >place.  I'm intrigued by the second solution, in which libraries would be
: >kept with their h-files and man pages.  I think this would make it easier
: >to keep multiple revisions of libraries on-line, and ease automation of
: >the upgrade process.

: That's not difficult, have a system of symbolic links and a file
: containing a list of those belonging to each packages (which would be
: kept in that packages installation directory).

I was concerned with h-file collision, and compile-time selection of
h-files/lib combinations.  From the little bit I've seen, people try to
manage multiple library revisions by managing include/lib/man paths at
application-complile-time.  I haven't looked at the debian package manager
(suggested by another poster), but with RPMs it doesn't seem that you can
really have multiple library generations fully on-line, which is what I'm
talking about.

John

 
 
 

Deeply integrating package management?

Post by Craig Kell » Wed, 02 Sep 1998 04:00:00




->The ideas that keep popping up are about integrating package management at
->a deeper level of the OS.  I see the problem as one of fan-out, in which
->files from a package are spread out across the system. One hard way to
->manage things would be to keep files in the same places they are now and
->track them in a database.

This is already done with rpm and the Debian package manger.

->An easier way might be to reorganize the
->filesystem, and keep files for a package together and access them in
->place.  I'm intrigued by the second solution, in which libraries would be
->kept with their h-files and man pages.  I think this would make it easier
->to keep multiple revisions of libraries on-line, and ease automation of
->the upgrade process.

That sounds a lot like Windows.  Everything has it's own directory.
Every (real) application wants you to add it's directory to your PATH
(which then becomes a huge mess).

No thanks.  Take a look at the man page for rpm and you'll find some
pretty cool stuff.

--
"Internet Explorer 4.0 Brings the Web to UNIX"  http://www.windows.com/ie/unix


 
 
 

Deeply integrating package management?

Post by John Jense » Wed, 02 Sep 1998 04:00:00




: This is already done with rpm and the Debian package manger.

I haven't looked at that one yet.

: ->An easier way might be to reorganize the
: ->filesystem, and keep files for a package together and access them in
: ->place.  I'm intrigued by the second solution, in which libraries would be
: ->kept with their h-files and man pages.  I think this would make it easier
: ->to keep multiple revisions of libraries on-line, and ease automation of
: ->the upgrade process.

: That sounds a lot like Windows.  Everything has it's own directory.
: Every (real) application wants you to add it's directory to your PATH
: (which then becomes a huge mess).

I'm not sure that Windows has tackled incremental update and change. Their
approach in a service pack is to replace pretty much everything. I think
the mechanism to to tie h-files to DLLs is pretty much a one-shot.  You
can compile against the "latest" headers and then test at runtime if you
have the "right" DLL.  I also think this parallels the current Linux
relationship between h-files and shared libraries.

What I'm thinking about is a mechanism to use multiple libraries and their
associated files without collision.  I'm not proposing a PATH based
solution, but rather a package look-up solution.

: No thanks.  Take a look at the man page for rpm and you'll find some
: pretty cool stuff.

Maybe I'm just using it wrong ... can I tell rpm to keep two libjpeg's
online so that I can compile against, link against, or run with either
one?

John

 
 
 

Deeply integrating package management?

Post by Craig Kell » Wed, 02 Sep 1998 04:00:00




->I was concerned with h-file collision, and compile-time selection of
->h-files/lib combinations.  From the little bit I've seen, people try to
->manage multiple library revisions by managing include/lib/man paths at
->application-complile-time.  I haven't looked at the debian package manager
->(suggested by another poster), but with RPMs it doesn't seem that you can
->really have multiple library generations fully on-line, which is what I'm
->talking about.

Sure you can, in fact both RedHat 5.0 and RedHat 5.1 come with the
ability to install the older libc5 C libraries as packages.

--
"Internet Explorer 4.0 Brings the Web to UNIX"  http://www.windows.com/ie/unix


 
 
 

Deeply integrating package management?

Post by Christopher Brow » Wed, 02 Sep 1998 04:00:00


On Tue, 01 Sep 1998 19:38:48 +0000, Tristan Wibberley




>> > The ideas that keep popping up are about integrating package management at
>> > a deeper level of the OS.  I see the problem as one of fan-out, in which
>> > files from a package are spread out across the system. One hard way to
>> > manage things would be to keep files in the same places they are now and
>> > track them in a database.  An easier way might be to reorganize the
>> > filesystem, and keep files for a package together and access them in
>> > place.  I'm intrigued by the second solution, in which libraries would be
>> > kept with their h-files and man pages.  I think this would make it easier
>> > to keep multiple revisions of libraries on-line, and ease automation of
>> > the upgrade process.

>That's not difficult, have a system of symbolic links and a file
>containing a list of those belonging to each packages (which would be
>kept in that packages installation directory).

If memory serves, BSD "Ports" <url
url="http://www.freebsd.org/handbook/ports.html" name="Installing
Applications: The Ports collection"> works in somewhat this fashion.

<url url="http://www.ccs.neu.edu/groups/systems/proj/SoftInst/"
name="The SoftInst project"> also does this.

I'm not sure about <url url="http://www.bell-labs.com/nsbd" name="NSBD -
Not-So-Bad Distribution">; not sure what mechanism it uses.  It is more
oriented towards authentication.

And there's a FSF packaging system too; GNUS Bulletin is sitting
elsewhere so I can't verify the name.

There is a *slight* problem with using symbolic-link-based systems; the
extra indirection does assess some additional costs in that you'll have
to do extra disk accesses to get at the actual files.

--
/*
 * [...] Note that 120 sec is defined in the protocol as the maximum
 * possible RTT.  I guess we'll have to use something other than TCP
 * to talk to the University of Mars.
 * PAWS allows us longer timeouts and large windows, so once implemented
 * ftp to mars will work nicely.
 */
(from /usr/src/linux/net/inet/tcp.c, concerning RTT [retransmission timeout])

 
 
 

Deeply integrating package management?

Post by Christopher Brow » Wed, 02 Sep 1998 04:00:00





>->The ideas that keep popping up are about integrating package management at
>->a deeper level of the OS.  I see the problem as one of fan-out, in which
>->files from a package are spread out across the system. One hard way to
>->manage things would be to keep files in the same places they are now and
>->track them in a database.

>This is already done with rpm and the Debian package manger.

An alternative view is that "the same places they are now" ought to be
the directories in which the source code compiled them to be.

For instance, this might mean that the *real* copy of "make" might
reside in /usr/src/GNU-Make-3.77/bin/make.  There would be a symbolic
link in /usr/bin, linking /usr/bin/make to
/usr/src/GNU-Make-3.77/bin/make.  Similarly, the manual pages might
truly reside in /usr/src/GNU-Make-3.77/man/man1, and be linked to
/usr/man/man1.  

And so forth.

To be sure, this view favors the "source" versions of packages.  

That's fairly different from the view that RPM/dpkg have, where one
compiles a bundle of software, and then "compile" a package that only
contains the pieces that actually need to be installed.

Quote:>->An easier way might be to reorganize the
>->filesystem, and keep files for a package together and access them in
>->place.  I'm intrigued by the second solution, in which libraries would be
>->kept with their h-files and man pages.  I think this would make it easier
>->to keep multiple revisions of libraries on-line, and ease automation of
>->the upgrade process.

>That sounds a lot like Windows.  Everything has it's own directory.
>Every (real) application wants you to add it's directory to your PATH
>(which then becomes a huge mess).

>No thanks.  Take a look at the man page for rpm and you'll find some
>pretty cool stuff.

This is *not* the same as Windows, and even to the extent to which there
is some similarity, it doesn't indicate that it's wrong.

Windows has the *really* serious problem that DLL files and other such
stuff gets all dropped into place in C:\WINDOWS as a giant jumble of
stuff.  The notion of having a couple of versions of some equivalent to
GLIBC as well as LIBC5 as well as LIBC4 present is a problem that is
presently making the heads of the NT5.0 developers explode.

With Linux, not only can you have a bunch of versions of libraries, you
can even have libraries that are directed towards use in conjunction
with cross-compilers for when you're compiling Alpha code on your Intel
box, or ARM code on your Alpha box.

And further, you can get the "best" of all worlds by the use of symbolic
links, which is to permit multiple simultaneous views on the same data.

Via sym-links, you can have:
a) Source code for the system libraries off in /usr/src,
b) Multiple targets available by referencing the symlinks to
/usr/lib/libc/[version]-[architecture], and
c) Default behaviour, that being to compile for the machine's preferred
libraries, by referencing through /usr/lib/libc.so

In effect, you can "have your cake and eat it too."

rpm and dpkg are well and good, but don't always take *full* advantage
of this.

--
"Let me get this straight:
 A company that dominates the desktop, and can afford to hire an army of
the world's best programmers, markets what is arguably the world's LEAST
reliable operating system?


 
 
 

Deeply integrating package management?

Post by Craig Kell » Wed, 02 Sep 1998 04:00:00



 [snip about keeping a database of installed packages]

->>This is already done with rpm and the Debian package manger.
->
->An alternative view is that "the same places they are now" ought to be
->the directories in which the source code compiled them to be.
->
->For instance, this might mean that the *real* copy of "make" might
->reside in /usr/src/GNU-Make-3.77/bin/make.  There would be a symbolic
->link in /usr/bin, linking /usr/bin/make to
->/usr/src/GNU-Make-3.77/bin/make.  Similarly, the manual pages might
->truly reside in /usr/src/GNU-Make-3.77/man/man1, and be linked to
->/usr/man/man1.  
->
->And so forth.
->
->To be sure, this view favors the "source" versions of packages.  
->
->That's fairly different from the view that RPM/dpkg have, where one
->compiles a bundle of software, and then "compile" a package that only
->contains the pieces that actually need to be installed.

But don't forget that each debian and redhat package comes with the
corresponding _source_ package as well.  Each system installs the
source packages in consistent locations, so I guess I still don't
understand what you want to do differently on this point.

->Windows has the *really* serious problem that DLL files and other such
->stuff gets all dropped into place in C:\WINDOWS as a giant jumble of
->stuff.  The notion of having a couple of versions of some equivalent to
->GLIBC as well as LIBC5 as well as LIBC4 present is a problem that is
->presently making the heads of the NT5.0 developers explode.

True true true.

->With Linux, not only can you have a bunch of versions of libraries, you
->can even have libraries that are directed towards use in conjunction
->with cross-compilers for when you're compiling Alpha code on your Intel
->box, or ARM code on your Alpha box.
->
->And further, you can get the "best" of all worlds by the use of symbolic
->links, which is to permit multiple simultaneous views on the same data.
->
->Via sym-links, you can have:
->a) Source code for the system libraries off in /usr/src,

Can be done w/o symbolic links (in fact this is exactly where RedHat's
srpms are put).

->b) Multiple targets available by referencing the symlinks to
->/usr/lib/libc/[version]-[architecture], and

Or, you can see which are installed with

  rpm -qa | grep libc

which even has an architecture, a version number and a build number
(in case you need to re-create the package at the same version).

->c) Default behaviour, that being to compile for the machine's preferred
->libraries, by referencing through /usr/lib/libc.so
->
->In effect, you can "have your cake and eat it too."
->
->rpm and dpkg are well and good, but don't always take *full* advantage
->of this.

I disagree, I think that they solve all the same problems that you are
trying to solve; they do it differently (and without many confusing
symbolic links) to be sure.

--
"Internet Explorer 4.0 Brings the Web to UNIX"  http://www.windows.com/ie/unix


 
 
 

Deeply integrating package management?

Post by David F » Wed, 02 Sep 1998 04:00:00



> I've had these ideas lately ... always a bad sign ... about the problems I
> have maintaing a rolling Linux system.  Maybe I'll get the hang of it, but
> right now I'm having a hard time upgrading libraries and keeping my
> applications running.

> The ideas that keep popping up are about integrating package management at
> a deeper level of the OS.  I see the problem as one of fan-out, in which
> files from a package are spread out across the system. One hard way to
> manage things would be to keep files in the same places they are now and
> track them in a database.  An easier way might be to reorganize the
> filesystem, and keep files for a package together and access them in
> place.

I think you've got it backwards.  Solution 1 is the easy one, and is
an integral part of various Linux packages such as RedHat and Debian.
Solution 2 is the *e - it basically means going into each
package in the operating system and painstakingly changing it all
around.  This means many person-years of work.
--
David Fox          http://www.veryComputer.com/;          xoF divaD

 
 
 

Deeply integrating package management?

Post by Kalle Niemital » Thu, 03 Sep 1998 04:00:00


GNU Stow is a utility for maintaining /usr/local.  You install each
package in a separate directory under /usr/local/stow (or some other
place) and Stow maintains symlinks in /usr/local that point there.

It has no databases.  This means you don't have to tell it if you add
or remove files.  On the other hand, uninstalling packages is rather
slow, as Stow has to scan the entire /usr/local tree for symlinks
pointing to the package.

http://www.gnu.org/software/stow/

--

666 MHz getting closer

 
 
 

Deeply integrating package management?

Post by John Jense » Thu, 03 Sep 1998 04:00:00



: > I've had these ideas lately ... always a bad sign ... about the problems I
: > have maintaing a rolling Linux system.  Maybe I'll get the hang of it, but
: > right now I'm having a hard time upgrading libraries and keeping my
: > applications running.
: >
: > The ideas that keep popping up are about integrating package management at
: > a deeper level of the OS.  I see the problem as one of fan-out, in which
: > files from a package are spread out across the system. One hard way to
: > manage things would be to keep files in the same places they are now and
: > track them in a database.  An easier way might be to reorganize the
: > filesystem, and keep files for a package together and access them in
: > place.

: I think you've got it backwards.  Solution 1 is the easy one, and is
: an integral part of various Linux packages such as RedHat and Debian.
: Solution 2 is the *e - it basically means going into each
: package in the operating system and painstakingly changing it all
: around.  This means many person-years of work.

I think an OS which keeps its resources in packages is conceptually
simple, but I agree that a mass migration to such a system would be many
person-years of work.  It would be fun to create a package-OS to test the
concept, but I understand that Linux must take an evolutionary approach.

I long time ago I read that engineers consistently underestimate the
ability of current technolgies to be extended, and consistently
overestimate the ability of new technologies to come on-line.  I've seen
that proven again and again.  So, I think I'll try to look a little closer
at the current technolgies before I recommend revolution.

John

 
 
 

Deeply integrating package management?

Post by Peter Samuels » Thu, 03 Sep 1998 04:00:00



Quote:> > > The ideas that keep popping up are about integrating package
> > > management at a deeper level of the OS.  I see the problem as one
> > > of fan-out, in which files from a package are spread out across
> > > the system. One hard way to manage things would be to keep files
> > > in the same places they are now and track them in a database.  An
> > > easier way might be to reorganize the filesystem, and keep files
> > > for a package together and access them in place.

Oh, right, the MS-DOS solution: make a directory, C:\PACKAGENAME, and
put everything in it.  Then edit AUTOEXEC.BAT to add the directory to
your path, the end result of which is that $PATH has six* thousand
directories in it, half of which are for packages you no longer have
around.  (I'm exaggerating the problem, of course, but this is
definitely one of the (many) things I don't like about the DOS world.)

$PATH management is one problem and it is tripled if you bring header
files and manpages into it.  Another is functional difference between
different files.  For instance: /etc traditionally stores config files.
/var stores files with relatively volatile contents.  You might want to
mount your root filesystem read-only -- or even share it across several
machines -- and have separate read-write partitions for /etc, /var and
/tmp.  If each package has a directory tree in which it keeps its own
config files, you pretty much can't share those trees around via NFS.
You also have to dump everything to tape, not just /etc, in order to
cover what can't be trivially reinstalled (i.e. your config files).

Quote:> > > I'm intrigued by the second solution, in which libraries would be
> > > kept with their h-files and man pages.  I think this would make
> > > it easier to keep multiple revisions of libraries on-line, and
> > > ease automation of the upgrade process.

Do you often need multiple versions of a library?  For run-time libs,
yes, perhaps, so as not to break applications.  That's why .so's are
versioned.  But when you're using a library for _development_, i.e. you
need the .h and .3 files around, what do you use multiple versions for?
(Not a rhetorical question, BTW.)  I just pick a recent stable version.

Yes, there are times when two versions of a library have significant
functional differences: Berkeley db 1.85 and 2.x, for instance.  Maybe
this is what you're thinking of?


Quote:> have a system of symbolic links and a file containing a list of those
> belonging to each packages (which would be kept in that packages
> installation directory).

If you're going to pollute your main directory trees with symlinks all
over the place, why not pollute them with the original files?  In other
words, as far as I can tell, the symlink system is very like the
database system except with an extra layer of complexity.

I have to vote for the database system: the Debian Project's dpkg does
a great job of keeping track of whose files are whose, and it upgrades
packages very smoothly.  (It even lets the pkg maintainer make a list
of "config files" then md5's them so it'll know not to overwrite one if
you've edited it.  Whoever came up with that concept is a genius.)  I
assume (though I haven't used it) that rpm is just as smooth.  As for
multiple library versions, I don't know offhand what Debian packaging
does for, say, db 1.85 and db 2.0 header file/manpage conflicts (if
there be any).  With minor library versions, that is to say, anything
the package maintainer doesn't feel to be an important enough
distinction, it just doesn't let you install more than one version....
This limitation, frankly, hasn't bothered me yet.

--
Peter Samuelson
<sampo.creighton.edu ! psamuels>

 
 
 

Deeply integrating package management?

Post by Peter Samuels » Thu, 03 Sep 1998 04:00:00



Quote:> > But don't forget that each debian and redhat package comes with the
> > corresponding _source_ package as well.  Each system installs the
> > source packages in consistent locations, so I guess I still don't
> > understand what you want to do differently on this point.


Quote:> RPM and DPKG are designed to build packages in a way that creates
> binaries so as to make the sources irrelevant.

Yes ... if you install the .deb/.i386.rpm, you don't need the source.
This makes old Unix people understandably suspicious, but me, I'd just
as soon not have maintain disk space for built sources of everything.
And call me lazy but it's very handy to let someone else make sure each
program uses the right mail locking scheme, puts config files in a sane
place, makes sure vim and nvi don't both try to install a
/usr/man/man1/vi.1, etc.  In short, the binary-package-based approach
serves me well, for the most part.

Occasionally I feel like hacking a program.  No problem, I just
download the source (and the Debian patch to see what they've already
decided was worth hacking up) and dig in.  I don't find this
inconvenient at all.

Quote:> In the case of the Stow/Depot ("link to source") approach, there is a
> direct isomorphism between the sources and the places where the
> binaries that are used get targeted.  One can *always* trace back to
> the sources, given the apparent location of one of the "target"
> files.
[...]
> but the source files are normally omitted in the .rpm/.deb file.  It
> requires extra effort to get at the source code.

dpkg's "-L" and "-S" print out files owned by a package and package
owning a file, respectively.  I assume rpm has similar options.  As for
tracing files back to where they appeared in the source code -- as has
been mentioned it is not hard to download the source for yourself.
Extra effort, yes, but not _that_ much extra effort.  Debian packages
also usually install any interesting READMEs and/or example files
somewhere in /usr/doc/packagename/ (except where said documentation is
particularly large, in which case it'll get its own package).

--
Peter Samuelson
<sampo.creighton.edu ! psamuels>

 
 
 

1. Visionary in Integrated Document and Output Management

Media Alert: Cypress Corporation Recognized as a Visionary in the
Integrated Document and Output Management Market

Cypress Corporation announced today that it has earned recognition in
the visionary quadrant of the Integrated Document and Output Management
(IDOM) market.

"Integrated document and output management focuses on the technologies,
architectures, organizational support and user behaviors that allow
users to transcend the constraints of today's limited document
structures and directly apply the value of information content to e-
business processes."; GartnerGroup's Integrated Document Scenario,
October 1999.

To read the full article:
http://www.cypressdelivers.com/pr_idom_00.htm

Sent via Deja.com http://www.deja.com/
Before you buy.

2. Why is unix so large part II

3. Free Integrated Web Site Management Software for NT 4.0

4. DISPLAY Env Var

5. Integrated Unix and VMS data management

6. Svgalib programming (tutorial?)

7. Free Integrated Web Site Management Software for NT 4.0

8. Initio 9100W

9. Free Integrated Web Site Management Software (Beta 3)

10. integrated package

11. any integrated C/C++ programming package on UNIX?

12. Setting VPE on integrated Trident CyberBlade/i1 for use with integrated RealMagic EM8470 (re: NetStream 2000)