using gdb with dynamically loaded .so libraries

using gdb with dynamically loaded .so libraries

Post by Lee Jenkin » Thu, 13 Dec 2001 08:05:38



I'm looking for someone who has some experience using gdb (or another
linux de*) to debug .so code.

My application loads "plug-in" .so files at run-time using dlopen(...).
The libraries load and execute as they should, but I can't seem to
convince kdbg (or gdb, for that matter) to let me step through the
source code of the loaded module. I've even gone so far as creating a
bare bones test case where the executable and library are each build
from cpp files in the same directory, but gdb doesn't recognize the
library's cpp file.

I've included more details below. Any ideas or helpful links would be
rather, well, er, helpful.

thanks,
Lee Jenkins
Corporate Diagnostic Software
Compaq Computer Corporation

I've reduced this problem to such a simple example that either I must be
missing some crucial but perhaps badly documented option, or else gdb
does not support the type of operation I desire. The code is very short,
and both files are located (and both targets built) in the same
directory:

------------------------------ [ testso.cpp ]
------------------------------
extern "C"
{
  int soAdd( int a, int b )
  {
    return( a + b );
  }

Quote:}

------------------------------ [ tsx.cpp ]
------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>
extern "C"
{
  typedef int (*intmathsofun)( int a, int b );
  intmathsofun Add;

Quote:}

int main( int argc, char* argv[] )
{
  char* soname  =  "./testso.so";
  void* tso  =  dlopen( soname, RTLD_LAZY );
  if( tso == NULL )
    printf( "unable to open %s.\n", soname );
  else
  {
    if( (Add=(intmathsofun)dlsym( tso, "soAdd" )) == NULL )
      printf( "unable to get function address for soAdd().\n" );
    else
    {
      int a = 7, b = 2;
      int s = Add( a, b );
      printf( "a: %d   b: %d  a + b: %d\n", a, b, s );
    }
    dlclose( tso );
  }

  return( 0 );

Quote:}

------------------------------ [ EOF ] ------------------------------

Compiling the above code (with -g for debug) and loading it into gdb, I
have observed the following:

1st Run
* start the debug session and set a breakpoint at line 23 of tsx.cpp
(the call to the Add ( ) function)
* I run the program, and it stops at line 23.
* if I try to step into the function call, gdb tells me that the
execution point is in eval.c, which it cannot find.
* I tell gdb to continue, and the program runs correctly and exits.

2nd Run
* this time I try a more direct route, and gdb happily accepts a
breakpoint in testso.cpp (at the return statement) with this command:
          break testso.cpp:5
* if I run the program with the above breakpoint, gdb runs, then claims
to stop in function soAdd ( ), but (again) in a file named eval.c, which
it cannot find.
* the disassemble command shows that gdb really did stop in the middle
of the soAdd ( ) function. however, the list command doesn't seem to
know which source code file it should use. it's looking for eval.c here,
too.

On the other hand, if I add testso.so to the link command for creating
tsx and change the Add= statement to assign directly from the
"statically" linked testso.so, gdb will step into the soAdd ( ) function
with no problem (and no recompile required for testso.so).

The problem is obviously with how (or if) gdb recognizes source code
files for .so files loaded using the dlopen ( ) function.

I would sincerely appreciate any insight from someone who can tell me
"no, gdb can't do that", or, "yes, gdb can, but first..."

 
 
 

using gdb with dynamically loaded .so libraries

Post by Eugene Kuznetso » Thu, 13 Dec 2001 12:03:39



> I'm looking for someone who has some experience using gdb (or another
> linux de*) to debug .so code.

> My application loads "plug-in" .so files at run-time using dlopen(...).
> The libraries load and execute as they should, but I can't seem to
> convince kdbg (or gdb, for that matter) to let me step through the
> source code of the loaded module. I've even gone so far as creating a
> bare bones test case where the executable and library are each build
> from cpp files in the same directory, but gdb doesn't recognize the
> library's cpp file.

> I've included more details below. Any ideas or helpful links would be
> rather, well, er, helpful.

> thanks,
> Lee Jenkins
> Corporate Diagnostic Software
> Compaq Computer Corporation

> I've reduced this problem to such a simple example that either I must be
> missing some crucial but perhaps badly documented option, or else gdb
> does not support the type of operation I desire. The code is very short,
> and both files are located (and both targets built) in the same
> directory:

> ------------------------------ [ testso.cpp ]
> ------------------------------
> extern "C"
> {
>   int soAdd( int a, int b )
>   {
>     return( a + b );
>   }
> }
> ------------------------------ [ tsx.cpp ]
> ------------------------------
> #include <stdio.h>
> #include <stdlib.h>
> #include <dlfcn.h>
> extern "C"
> {
>   typedef int (*intmathsofun)( int a, int b );
>   intmathsofun Add;
> }

> int main( int argc, char* argv[] )
> {
>   char* soname  =  "./testso.so";
>   void* tso  =  dlopen( soname, RTLD_LAZY );
>   if( tso == NULL )
>     printf( "unable to open %s.\n", soname );
>   else
>   {
>     if( (Add=(intmathsofun)dlsym( tso, "soAdd" )) == NULL )
>       printf( "unable to get function address for soAdd().\n" );
>     else
>     {
>       int a = 7, b = 2;
>       int s = Add( a, b );
>       printf( "a: %d   b: %d  a + b: %d\n", a, b, s );
>     }
>     dlclose( tso );
>   }

>   return( 0 );
> }
> ------------------------------ [ EOF ] ------------------------------

> Compiling the above code (with -g for debug) and loading it into gdb, I
> have observed the following:

> 1st Run
> * start the debug session and set a breakpoint at line 23 of tsx.cpp
> (the call to the Add ( ) function)
> * I run the program, and it stops at line 23.
> * if I try to step into the function call, gdb tells me that the
> execution point is in eval.c, which it cannot find.
> * I tell gdb to continue, and the program runs correctly and exits.

> 2nd Run
> * this time I try a more direct route, and gdb happily accepts a
> breakpoint in testso.cpp (at the return statement) with this command:
>           break testso.cpp:5
> * if I run the program with the above breakpoint, gdb runs, then claims
> to stop in function soAdd ( ), but (again) in a file named eval.c, which
> it cannot find.
> * the disassemble command shows that gdb really did stop in the middle
> of the soAdd ( ) function. however, the list command doesn't seem to
> know which source code file it should use. it's looking for eval.c here,
> too.

> On the other hand, if I add testso.so to the link command for creating
> tsx and change the Add= statement to assign directly from the
> "statically" linked testso.so, gdb will step into the soAdd ( ) function
> with no problem (and no recompile required for testso.so).

> The problem is obviously with how (or if) gdb recognizes source code
> files for .so files loaded using the dlopen ( ) function.

> I would sincerely appreciate any insight from someone who can tell me
> "no, gdb can't do that", or, "yes, gdb can, but first..."

gdb can do that with no problems ( your sample works just fine here ).
eval() stuff sometimes indicates that current code ( in your case,
testso.cpp ) is compiled or linked without -g option.
a) what are the commands you use to link the library and executable?
b) what version of gdb is it?

 
 
 

using gdb with dynamically loaded .so libraries

Post by Jens.Toerr.. » Thu, 13 Dec 2001 12:22:07



> I'm looking for someone who has some experience using gdb (or another
> linux de*) to debug .so code.
> My application loads "plug-in" .so files at run-time using dlopen(...).
> The libraries load and execute as they should, but I can't seem to
> convince kdbg (or gdb, for that matter) to let me step through the
> source code of the loaded module. I've even gone so far as creating a
> bare bones test case where the executable and library are each build
> from cpp files in the same directory, but gdb doesn't recognize the
> library's cpp file.

[ snipped ]

Quote:> I would sincerely appreciate any insight from someone who can tell me
> "no, gdb can't do that", or, "yes, gdb can, but first..."

Well, at least I can tell you that it can be done - I never had any
real problems (but I have tried it only with C code, not C++). Just a
two things which I have a feeling might help:
a) Compile without optimization, possibly most of your rather simple
   code gets optimized away so that there isn't really any meaningful
   line of code left...
b) Use RTLD_NOW instead of RTLD_LAZY (I have the feeling that in this
   case there's no need to invoke code from eval.c, so stepping directly
   into your library function should be possible.)
c) Since I don't know how you build the libraries: are you sure that
   the library doesn't get stripped? If you're using a overly 'helpful'
   tool that does this behind your back to reduce the library's size
   gdb obviously will have it's problems ;-)

                                          Regards, Jens
--
      _  _____  _____

  _  | |  | |    | |          AG Moebius, Institut fuer Molekuelphysik
 | |_| |  | |    | |          Fachbereich Physik, Freie Universitaet Berlin
  \___/ens|_|homs|_|oerring   Tel: ++49 (0)30 838 - 53394 / FAX: - 56046

 
 
 

using gdb with dynamically loaded .so libraries

Post by Lee Jenkin » Fri, 14 Dec 2001 07:14:33



> gdb can do that with no problems ( your sample works just fine here ).
> eval() stuff sometimes indicates that current code ( in your case,
> testso.cpp ) is compiled or linked without -g option.
> a) what are the commands you use to link the library and executable?
> b) what version of gdb is it?

Here's the makefile:

------------------------------
all: tsx testso.so

tsx: tsx.o
 g++ -g -o tsx tsx.o -ldl

tsx.o: tsx.cpp
 g++ -fPIC -g -c -Wall tsx.cpp

testso.so: testso.o
 g++ -g -shared -Wl,-soname,testso.so -o testso.so -ldl testso.o

testso.o: testso.cpp
 g++ -fPIC -g -c -Wall -o testso.o testso.cpp

------------------------------

These are the relevant versions:
* GNU gdb 5.0rh-5 Red Hat Linux 7.1
* gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-81)
* glibc -> 2.1.2

The really, really, strange thing is that if I put 'testso.so' in the link
when I build 'tsx', and (of course) call the Add ( ) function directly, gdb
steps into the .so file without any problems.

If others aren't having the same trouble with my example, then it may very
well be an issue with my installed version of gdb or gcc.

Thanks for your help.

leej

 
 
 

using gdb with dynamically loaded .so libraries

Post by Eugene Kuznetso » Fri, 14 Dec 2001 08:24:42




>> gdb can do that with no problems ( your sample works just fine here ).
>> eval() stuff sometimes indicates that current code ( in your case,
>> testso.cpp ) is compiled or linked without -g option.
>> a) what are the commands you use to link the library and executable?
>> b) what version of gdb is it?

> Here's the makefile:

> ------------------------------
> all: tsx testso.so

> tsx: tsx.o
>  g++ -g -o tsx tsx.o -ldl

> tsx.o: tsx.cpp
>  g++ -fPIC -g -c -Wall tsx.cpp

> testso.so: testso.o
>  g++ -g -shared -Wl,-soname,testso.so -o testso.so -ldl testso.o

> testso.o: testso.cpp
>  g++ -fPIC -g -c -Wall -o testso.o testso.cpp

> ------------------------------

Looks OK.

Quote:

> These are the relevant versions:
> * GNU gdb 5.0rh-5 Red Hat Linux 7.1
> * gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-81)
> * glibc -> 2.1.2

The problem must be somewhere in here. How come you have Red Hat Linux 7.1
and glibc 2.1.2? RH 7.1 comes with 2.2.2.

What happens if you type 'sharedlib' in gdb before stepping into Add()?

 
 
 

using gdb with dynamically loaded .so libraries

Post by Lee Jenkin » Fri, 14 Dec 2001 09:25:48



> The problem must be somewhere in here. How come you have Red Hat Linux 7.1
> and glibc 2.1.2? RH 7.1 comes with 2.2.2.

oops. my bad. it is 2.2.2.

Quote:> What happens if you type 'sharedlib' in gdb before stepping into Add()?

sharedlib says it has loaded symbols for several libraries, including
testso.so

I've also tried Jens Toerring's suggestion of changing RTLD_LAZY to RTLD_NOW,
but gdb still doesn't seem to know how to find the source code for the shared
library.

By now you must surely think I am crazy or clueless or perhaps something far
worse. Let me show you a log of an actual gdb session, perhaps it will help
clarify:

------------------------- [ begin gdb session ] -------------------------
GNU gdb 5.0rh-5 Red Hat Linux 7.1
Copyright 2001 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux"...
(gdb) break 29
Breakpoint 1 at 0x804873a: file tsx.cpp, line 29.
(gdb) run
Starting program: /home/leej/source/sodebug/exec/tsx

Breakpoint 1, main (argc=1, argv=0xbffff99c) at tsx.cpp:29
29       int s = Add( a, b );
Current language:  auto; currently c++
(gdb) sharedlib
Symbols already loaded for /lib/libdl.so.2
Symbols already loaded for /usr/lib/libstdc++-libc6.2-2.so.3
Symbols already loaded for /lib/i686/libm.so.6
Symbols already loaded for /lib/i686/libc.so.6
Symbols already loaded for /lib/ld-linux.so.2
Symbols already loaded for ./testso.so
(gdb) step
0x40018743 in soAdd () at eval.c:41
41          eval.c: No such file or directory
 in eval.c
Current language:  auto; currently c
(gdb) quit
The program is running.  Exit anyway? (y or n) y
------------------------- [ end gdb session ] -------------------------

You can see that gdb prints the name of ths function 'soAdd ( )' correctly. It
must have accessed the debug symbols from the .so file to do so. So why
doesn't it read the name of the source code file correctly?

Is anyone else running Red Hat 7.1? Maybe there's something dodgey with the
combination of gcc, gdb, etc.

perplexed,
leej

 
 
 

using gdb with dynamically loaded .so libraries

Post by Jens.Toerr.. » Fri, 14 Dec 2001 10:35:33


Hi Lee,

I compiled your program exactly as you posted it - source text (and with
RTLD_LAZY and not RTLD_NOW) as well as the Makefile - and I don't have a
problem stepping into the shared library. The versions I'm using:


Reading specs from /usr/lib/gcc-lib/i486-suse-linux/2.95.2/specs
gcc version 2.95.2 19991024 (release)


GNU C Library stable release version 2.2, by Roland McGrath et al.
Copyright (C) 1992,93,94,95,96,97,98,99,2000 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 2.95.2 19991024 (release).
Compiled on a Linux 2.4.0 system on 2001-01-19.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        Berkeley DB glibc 2.1 compat library by Thorsten Kukuk
        linuxthreads-0.9 by Xavier Leroy
        NoVersion patch for broken glibc 2.0 binaries
        BIND-8.2.3-T5B
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        NSS V1 modules 2.1 by Thorsten Kukuk
        libthread_db work sponsored by Alpha Processor Inc


GNU gdb 5.0
Copyright 2000 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-suse-linux".

When I run it in under gdb I get:


GNU gdb 5.0
Copyright 2000 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-suse-linux"...
(gdb) b 23
Breakpoint 1 at 0x80486f4: file tsx.cpp, line 23.
(gdb) r
Starting program: /home/jens/TESTS/tsx

Breakpoint 1, main (argc=1, argv=0xbffff3d4) at tsx.cpp:23
23            int s = Add( a, b );
(gdb) s
soAdd (a=7, b=2) at testso.cpp:5
5           return( a + b );
(gdb) s
6         }
(gdb) s
main (argc=1, argv=0xbffff3d4) at tsx.cpp:24
24            printf( "a: %d   b: %d  a + b: %d\n", a, b, s );
(gdb) s
a: 7   b: 2  a + b: 9
26          dlclose( tso );
(gdb) s
29        return( 0 );
(gdb) s
30      }
(gdb) s
0x400b2baf in __libc_start_main () from /lib/libc.so.6
(gdb) s
Single stepping until exit from function __libc_start_main,
which has no line number information.

Program exited normally.
(gdb)

The only major difference is that I don't run Red Hat but SuSE 7.1. So,
one might suspect there's something strange with the version of gdb Red
Hat distributes...
                                       Regards, Jens
--
      _  _____  _____

  _  | |  | |    | |          AG Moebius, Institut fuer Molekuelphysik
 | |_| |  | |    | |          Fachbereich Physik, Freie Universitaet Berlin
  \___/ens|_|homs|_|oerring   Tel: ++49 (0)30 838 - 53394 / FAX: - 56046

 
 
 

using gdb with dynamically loaded .so libraries

Post by Lee Jenkin » Sat, 15 Dec 2001 06:49:27



> The only major difference is that I don't run Red Hat but SuSE 7.1. So,
> one might suspect there's something strange with the version of gdb Red
> Hat distributes...
>                                        Regards, Jens

Jens,

Thanks for taking the time to run my example. At least I know that it should work
and that I haven't completely lost my mind. We've considering moving to Red Hat
7.2 anyway, but one way or another I should be able to figure it out now.

Thanks also to Eugene Kuznetsov for testing my example.

cheers,
Lee Jenkins
Compaq Computer Corp.

 
 
 

1. Using GDB on dynamically loaded code

How can I use Gnu gdb to step into my dynamically loaded code? I'm using
glib's g_module_* functions, which I think wrap dlopen etc. When I run my
program from within gdb, I get this message:

warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Any help would be gratefully received!

Matthew R.

2. redirecting STDOUT of a bg process to STDOUT of a new xterm

3. Dynamically Loaded Shared Libraries and dbx

4. trouble

5. Dynamically loading a ****.so library

6. stty question

7. Unresolved references in dynamically loaded shared library (SunOS 4.1.3)

8. AIX Communicating With Outer World

9. Controlling exports in Dynamically loaded shared libraries

10. Proble: Dynamically loaded C++ libraries

11. dynamically loading shared libraries

12. what is the equivalent of dynamically loading libraries on Solaris

13. Getting a handle of a dynamically loaded library with dlopen() in _init() function.