Blastwave.org packages with insane dependencies

Blastwave.org packages with insane dependencies

Post by UNIX admi » Wed, 14 Jan 2004 23:25:47



I noticed peculiarities about CSW* packages, like totally insane package
dependencies.  For example, why in blue blazes would aRts need qt???  And of
course, a closer look reveals that it doesn't need it at all.

Phil, if you're reading this, you need to contact me, pronto.  I recently
developed the technology for generating the exact runtime dependencies in
the "depend" file format for `pkgmk`, and I'm willing to release it to
Blastwave for day-to-day use and further testing/development.

 
 
 

Blastwave.org packages with insane dependencies

Post by Thomas Glanzman » Thu, 15 Jan 2004 04:14:08


Hello UNIX admin, :-))


Quote:> Phil, if you're reading this, you need to contact me, pronto.  I recently
> developed the technology for generating the exact runtime dependencies in
> the "depend" file format for `pkgmk`, and I'm willing to release it to
> Blastwave for day-to-day use and further testing/development.

I am using the following perl script to determine deps automatically.
But it doesn't work yet for perl dependencies. - But shouldn't be that
hard todo.

sub
find_dependencies
{
        my $r = shift || die("one reference expected");


        chdir(${buildroot}) || die("can't change to ${buildroot}: $!");
        # look for shared libaries

        # look for bangs



        }

                chomp($file);
                open(FILE, $file) || die("can't open ${file}: $!");
                my $firstline = <FILE>;
                if ($firstline =~ /^#!\s?([^\s]+)/) {

                }
                close(FILE);
        }

        # resolve symlinks / substitute


                # /bin and /lib are packages in /usr/{bin,lib}
                $element =~ s#^/bin#/usr/bin#;
                $element =~ s#^/lib#/usr/lib#;
                # Resolve links if necessary
                $element = resolve_link($element);
        }

        # get dependencies

                # </usr/lib/../openwin/lib/libX11.so.4>
                $dep =~ s#\w+\/\.\.##g;


        }

        # parse out package names

                if ($line =~ /^\t([^\s]+)/) {

                }
        }

        # make them uniq and don't include a dependency to the packet itself

        write_dependencies($r);

Quote:}

write_dependencies
{
        my $r = shift || die("one reference expected");


        my %pkg = ();

                if ($line =~ /^[^\s]+\s+([^\s]+)\s+([^\s].*)/) {
                        $pkg{$1} = "$2";
                }
        }

        open(DEP, '> depend') || die("can't open depend file: $!");


                print DEP "P $dep $pkg{$dep}\n";
        }

        if (defined($r->{incompatibilities})) {

                        print DEP "I $inc $pkg{$inc}\n";
                }
        }

        close(DEP);

Quote:}

If you're interested in my whole packaging envrionment drop me an eMail.
But I basically tried to have something like rpm.

        Thomas

 
 
 

Blastwave.org packages with insane dependencies

Post by Drazen Kaca » Thu, 15 Jan 2004 05:15:11



>  I am using the following perl script to determine deps automatically.
>  But it doesn't work yet for perl dependencies. - But shouldn't be that
>  hard todo.

>  sub
>  find_dependencies
>  {
>          my $r = shift || die("one reference expected");


>          chdir(${buildroot}) || die("can't change to ${buildroot}: $!");
>          # look for shared libaries


Hmm. I have some questions for the discussion. Should a program like this:

  1. treat lazy loaded libraries the samo way as the normal ones?
  2. shouldn't it "reset" crle configuration to the default?
  3. shouldn't it ignore libraries which are not direct dependancies?

--
 .-.   .-.    Yes, I am an agent of Satan, but my duties are largely
(_  \ /  _)   ceremonial.
     |

 
 
 

Blastwave.org packages with insane dependencies

Post by Thomas Glanzman » Thu, 15 Jan 2004 06:56:43


Hi,

Quote:>   1. treat lazy loaded libraries the same way as the normal ones?

it should, but doesn't. But works for most of the packages because
dlopen is rarely used.

Quote:>   2. shouldn't it "reset" crle configuration to the default?

it should, but we don't touch crle - so no need for us in our
envrionment on the build machines of blastwave.

Quote:>   3. shouldn't it ignore libraries which are not direct dependancies?

It does, doesn't it?

        Thomas

 
 
 

Blastwave.org packages with insane dependencies

Post by Markus Gyg » Thu, 15 Jan 2004 09:00:08



> I am using the following perl script to determine deps automatically.

You might also take a look at /opt/csw/bin/checkpkg.
Most likely you'd need to replace ldd by dump -Lv to
not get the indirect dependencies.

Markus

 
 
 

Blastwave.org packages with insane dependencies

Post by Thomas Glanzman » Thu, 15 Jan 2004 09:37:24


Hi,


Quote:> You might also take a look at /opt/csw/bin/checkpkg.  Most likely
> you'd need to replace ldd by dump -Lv to not get the indirect
> dependencies.

I already had a look at this (and patched the script twice). But I don't
like it that much. And it doesn't detect the bang thing.

        Thomas

 
 
 

Blastwave.org packages with insane dependencies

Post by Drazen Kaca » Thu, 15 Jan 2004 17:28:30



> >   1. treat lazy loaded libraries the same way as the normal ones?

>  it should, but doesn't. But works for most of the packages because
>  dlopen is rarely used.

I didn't mean dlopen. You can use positional flag -zlazyload when linking
(-znolazyload to revert to default) and libraries marked in that way will
not be loaded when the application attempts to use the first function from
it, but not on the startup (unless you're unlucky). So you could, for
example, lazy load wget with the SSL library, transfer wget binary to the
system which doesn't have SSL library installed, and wget would work for
non-SSL connections.

Quote:> >   2. shouldn't it "reset" crle configuration to the default?

>  it should, but we don't touch crle - so no need for us in our
>  envrionment on the build machines of blastwave.

It's just a question of setting LD_NOCONFIG environment variable.

Quote:> >   3. shouldn't it ignore libraries which are not direct dependancies?

>  It does, doesn't it?

It's using ldd, and ldd lists all libraries which are used by the
libraries that the executable is using, meaning it's listing indirect
dependencis. But I could be wrong, I didn't start the program.

--
 .-.   .-.    Yes, I am an agent of Satan, but my duties are largely
(_  \ /  _)   ceremonial.
     |

 
 
 

Blastwave.org packages with insane dependencies

Post by Thomas Glanzman » Thu, 15 Jan 2004 19:43:22


Hi,

Quote:> I didn't mean dlopen. You can use positional flag -zlazyload when linking
> (-znolazyload to revert to default) and libraries marked in that way will
> not be loaded when the application attempts to use the first function from
> it, but not on the startup (unless you're unlucky). So you could, for
> example, lazy load wget with the SSL library, transfer wget binary to the
> system which doesn't have SSL library installed, and wget would work for
> non-SSL connections.

Ic. Do you know a way to detect this dependencies? Is there any
application you know about which uses this actually?

Quote:> It's just a question of setting LD_NOCONFIG environment variable.

I will add this to my script.

Quote:> It's using ldd, and ldd lists all libraries which are used by the
> libraries that the executable is using, meaning it's listing indirect
> dependencis. But I could be wrong, I didn't start the program.

I think you're right, but it never happend in the real world for me
because for example X dpendencies had a ../ in ther path to the library
which was my mechanism unable to resolve (but I fixed it as you can sse
in the perlscript a few weeks ago).

        Thomas

 
 
 

Blastwave.org packages with insane dependencies

Post by James Le » Thu, 15 Jan 2004 19:49:33



> I noticed peculiarities about CSW* packages, like totally insane package
> dependencies.  For example, why in blue blazes would aRts need qt??? And
> of course, a closer look reveals that it doesn't need it at all.

I can't say why but I can confirm that it does. (KDE integration perhaps.)

If you think it shouldn't, please make the appropriate representation via
the bug reporting mechanism.

 
 
 

Blastwave.org packages with insane dependencies

Post by UNIX admi » Thu, 15 Jan 2004 19:52:22




> Hello UNIX admin, :-))


> > Phil, if you're reading this, you need to contact me, pronto.  I
recently
> > developed the technology for generating the exact runtime dependencies
in
> > the "depend" file format for `pkgmk`, and I'm willing to release it to
> > Blastwave for day-to-day use and further testing/development.

> I am using the following perl script to determine deps automatically.
> But it doesn't work yet for perl dependencies. - But shouldn't be that
> hard todo.

> sub
> find_dependencies
> {
>         my $r = shift || die("one reference expected");


>         chdir(${buildroot}) || die("can't change to ${buildroot}: $!");
>         # look for shared libaries


2> /dev/null | grep -v 'file not found' 2> /dev/null | grep '=>' | awk
'{print \$3}'`;

>         # look for bangs




"${buildroot}/${file}");

- Show quoted text -

>         }

>                 chomp($file);
>                 open(FILE, $file) || die("can't open ${file}: $!");
>                 my $firstline = <FILE>;
>                 if ($firstline =~ /^#!\s?([^\s]+)/) {

>                 }
>                 close(FILE);
>         }

>         # resolve symlinks / substitute


>                 # /bin and /lib are packages in /usr/{bin,lib}
>                 $element =~ s#^/bin#/usr/bin#;
>                 $element =~ s#^/lib#/usr/lib#;
>                 # Resolve links if necessary
>                 $element = resolve_link($element);
>         }

>         # get dependencies

>                 # </usr/lib/../openwin/lib/libX11.so.4>
>                 $dep =~ s#\w+\/\.\.##g;


>         }

>         # parse out package names

>                 if ($line =~ /^\t([^\s]+)/) {

>                 }
>         }

>         # make them uniq and don't include a dependency to the packet
itself



- Show quoted text -

>         write_dependencies($r);
> }

> write_dependencies
> {
>         my $r = shift || die("one reference expected");


>         my %pkg = ();

>                 if ($line =~ /^[^\s]+\s+([^\s]+)\s+([^\s].*)/) {
>                         $pkg{$1} = "$2";
>                 }
>         }

>         open(DEP, '> depend') || die("can't open depend file: $!");


>                 print DEP "P $dep $pkg{$dep}\n";
>         }

>         if (defined($r->{incompatibilities})) {

>                         print DEP "I $inc $pkg{$inc}\n";
>                 }
>         }

>         close(DEP);
> }

> If you're interested in my whole packaging envrionment drop me an eMail.
> But I basically tried to have something like rpm.

This isn't what I meant.  If I understand this code correctly, you check
package references, i.e. which package(s) reference(s) the package in which
the file in question is included.  This is how you could potentially end up
with CSWarts package needing CSWqt, which is not a sane state of affairs.

If that is indeed so, then there is a logic error, because now you're
assuming that whoever packaged the software knew exactly which libraries and
files the package needed.

On the other hand, I look at run-time dependencies, meaning I generate a
depend file with a list of packages that contain the run-time .so libraries
that you would normally see opened with the open() call when looking at the
binary with `truss` under Solaris or `par` under IRIX.
I don't actually use `truss` to do that, but I have verified that the depend
file is correct by cross referencing the output from `truss`, i.e. looking
at all the open() calls for .so's.

This way, I assume NOTHING.  I do not assume that the packager (in this
case, me!) knew which packages will be needed.  I have the computer churn
out which packages are *really* going to be needed by the dynamically linked
libraries and binaries in my own package.

 
 
 

Blastwave.org packages with insane dependencies

Post by Markus Gyg » Thu, 15 Jan 2004 21:02:45



>> I didn't mean dlopen. You can use positional flag -zlazyload when linking
>> (-znolazyload to revert to default) and libraries marked in that way will
>> not be loaded when the application attempts to use the first function from
>> it, but not on the startup (unless you're unlucky). So you could, for
>> example, lazy load wget with the SSL library, transfer wget binary to the
>> system which doesn't have SSL library installed, and wget would work for
>> non-SSL connections.

I would guess one would have to decide to include these in
the dependencies on a case by case basis. The not so nice
thing is, that you don't get intuitive error messages if e.g.
on above example a user uses an HTTPS URL and OpenSSL isn't
installed (see pkgadd below for the same situation).

Quote:>Ic. Do you know a way to detect this dependencies? Is there any
>application you know about which uses this actually?

See e.g. the output of dump -Lv /opt/csw/bin/ksh: libsecdb is only
loaded if you invoke it as pfksh (same with /usr/bin/ksh). Another
example is pkgadd (recent Solaris 9 recommended patches installed
but not upgraded to Solaris 9 12/03) when using a URL.

Markus

 
 
 

Blastwave.org packages with insane dependencies

Post by Markus Gyg » Thu, 15 Jan 2004 21:07:38



>On the other hand, I look at run-time dependencies, meaning I generate a
>depend file with a list of packages that contain the run-time .so libraries
>that you would normally see opened with the open() call when looking at the
>binary with `truss` under Solaris or `par` under IRIX.

Or pldd on recent versions of Solaris. But it
also includes indirectly referenced objects.

Markus

 
 
 

Blastwave.org packages with insane dependencies

Post by Drazen Kaca » Thu, 15 Jan 2004 23:23:19



> > I didn't mean dlopen. You can use positional flag -zlazyload when linking
> > (-znolazyload to revert to default) and libraries marked in that way will
> > not be loaded when the application attempts to use the first function from
> > it, but not on the startup (unless you're unlucky). So you could, for
> > example, lazy load wget with the SSL library, transfer wget binary to the
> > system which doesn't have SSL library installed, and wget would work for
> > non-SSL connections.

>  Ic. Do you know a way to detect this dependencies? Is there any
>  application you know about which uses this actually?

Lazy loading is currently not very usable for applications which want to
change behavior in run-time. If it happens that the library can't be
found, ld.so.1 will by default send SIGKILL. The application can ask for a
different signal, say one it can catch and then do something about the
situation, but the whole mechanism is currently too crude to be really
useful.

AIX can do better, for example. If the lazy library is not found, dynamic
loader can call user supplied function which can return the address of
a function which will be called instead of the one which the application
originally called. I suppose Solaris will get something like this in
the future.

I tend to use lazy loading a lot when building binaries, because sometimes
I have to run them in the hostile environments, so it's very nice if the
basic functionality requires only minimal environment. Hence the wget and
SSL example.

I think Mozilla will assert unconditional -zlazyload for all libraries
when being built on Solaris with Sun's compilers. I don't know of any
other examples and I'm not even sure that what Mozilla is doing actually
helps. A lot of programs are written in a way which disables lazy loading
benefits, so you end up with bigger binaries which start slower in case
you try to use lazy loading.

So the whole thing is something that might be used more in the future. You
can detect lazyload flags with dump or elfdump, but they will be on the
line preceeding the one with the library, so grepping might be a bit
complicated.

--
 .-.   .-.    Yes, I am an agent of Satan, but my duties are largely
(_  \ /  _)   ceremonial.
     |

 
 
 

Blastwave.org packages with insane dependencies

Post by Thomas Glanzman » Thu, 15 Jan 2004 23:44:18


Hi Drazen,


Quote:> So the whole thing is something that might be used more in the future. You
> can detect lazyload flags with dump or elfdump, but they will be on the
> line preceeding the one with the library, so grepping might be a bit
> complicated.

Thnx for the awareness training. And I will implement as soon as I hit
the first application using it.

        Thomas

 
 
 

Blastwave.org packages with insane dependencies

Post by Phil Brow » Sat, 17 Jan 2004 15:12:52




>> I noticed peculiarities about CSW* packages, like totally insane package
>> dependencies.  For example, why in blue blazes would aRts need qt??? And
>> of course, a closer look reveals that it doesn't need it at all.

> I can't say why but I can confirm that it does. (KDE integration perhaps.)

arts, contains libartskde.so, which requires libqtmcop and libqt-mt

Yes, I think this is stupid too. This doesnt belong in arts. But the
writers of arts, decided that they would have it in their software, rather
than have kde sort things out at a higher, cleaner level.
If you (tripivceta) dont like this, complain to the aRts/KDE developers