Printing backtrace from core file

Printing backtrace from core file

Post by Paul Tombl » Sat, 28 Jun 1997 04:00:00



Our users frequently get core dumps from our huge application, and in order to
facilite our debugging they often feel the need to email us these 650Mb core
files, or more likely they just set "limit core 0" and forget about it.  All
we really need is the backtrace from the core file, but many of them don't
have dbx, and even the ones that do can't be bothered to start up dbx then
type "where".  So we end up with a lot of complaints about core dumps, but no
help in narrowing down where and why these are happening.

We are thinking about shipping gdb with our product in order to help the ones
who don't have dbx.  But first I thought I'd have a poke around to see if
there is an easy way to *just* get the backtrace from a core file without all
the overhead of a huge de*.  Is there a program that would do that for
me?  This is Irix 6.2 on various SGI platforms.  Alternatively, is there a
signal handler I could add to the program that would print out a stack trace
before dying *without* dumping core?

--
Paul Tomblin, Current Product Engineering team leader.
I don't speak for Kodak, they don't speak for me.

"You are in a twisty maze of Motif Widget resources, all inconsistent."

 
 
 

Printing backtrace from core file

Post by Justin Bank » Sat, 28 Jun 1997 04:00:00



> Alternatively, is there a signal handler I could add to the program
> that would print out a stack trace before dying *without* dumping
> core?

Paul,
        Try man trace_back_stack. If you set the signal hander for whatever
signal your program recieves before it dumps core to be a function that
implements trace_back_stack, I think you'll get the effect you're after.

--
Justin Banks
Silicon Graphics Cray Research
Eagan, Minnesota

 
 
 

Printing backtrace from core file

Post by Walter Brisco » Sat, 28 Jun 1997 04:00:00


Expires: 05 Jul 1997 05:45:03 -0000

comp.unix.programmer FAQ: <URL: http://www.veryComputer.com/;

6.5 How can I generate a stack dump from within a running program?

With the information in that reference, you should be able to reduce your
users' ability to produce core files by a couple of orders of magnitude.



Quote:>Our users frequently get core dumps from our huge application, and in order to
>facilite our debugging they often feel the need to email us these 650Mb core
>files, or more likely they just set "limit core 0" and forget about it.  All
>we really need is the backtrace from the core file, but many of them don't
>have dbx, and even the ones that do can't be bothered to start up dbx then
>type "where".  So we end up with a lot of complaints about core dumps, but no
>help in narrowing down where and why these are happening.

>We are thinking about shipping gdb with our product in order to help the ones
>who don't have dbx.  But first I thought I'd have a poke around to see if
>there is an easy way to *just* get the backtrace from a core file without all
>the overhead of a huge de*.  Is there a program that would do that for
>me?  This is Irix 6.2 on various SGI platforms.  Alternatively, is there a
>signal handler I could add to the program that would print out a stack trace
>before dying *without* dumping core?

--
Walter Briscoe
 
 
 

Printing backtrace from core file

Post by Bjorn Ree » Sat, 28 Jun 1997 04:00:00



> who don't have dbx.  But first I thought I'd have a poke around to see if
> there is an easy way to *just* get the backtrace from a core file without all
> the overhead of a huge de*.  Is there a program that would do that for
> me?  This is Irix 6.2 on various SGI platforms.  Alternatively, is there a
> signal handler I could add to the program that would print out a stack trace
> before dying *without* dumping core?

To the best of my knowledge it isn't easy without a de*. When
faced with the same problem, I decided to maintain my own trace stack.
Using C++ I would simply have a trace object, where I would add the
current function to the stack in the constructor, and remove it in the
destructor. This works very well and is platform independent (hell, it
even works on NT :) but unfortunately it has a little computational
overhead and you have to remember to add the trace object to all
functions yourself and it cannot trace system calls. In practice it
has provided me with most the detail I want.

If someone knows a better way I would be very eager to hear.

 
 
 

Printing backtrace from core file

Post by Paul Tombl » Sun, 29 Jun 1997 04:00:00



Quote:>comp.unix.programmer FAQ: <URL: http://www.erlenstar.demon.co.uk/unix/>

>6.5 How can I generate a stack dump from within a running program?

>With the information in that reference, you should be able to reduce your
>users' ability to produce core files by a couple of orders of magnitude.

To paraphrase what it says in the FAQ:  "Call some system specific thing if
you're lucky, otherwise spawn dbx".  To recap what I said when I originally
asked the question:



>>we really need is the backtrace from the core file, but many of them don't

                                                          ^^^^^^^^^^^^^^^^^^
Quote:>>have dbx, and even the ones that do can't be bothered to start up dbx then

  ^^^^^^^^

Which is why I asked the question AFTER READING THE FAQ in the first place.

Thanks but no thanks.
--
Paul Tomblin, Current Product Engineering team leader.
I don't speak for Kodak, they don't speak for me.

"You are in a twisty maze of Motif Widget resources, all inconsistent."

 
 
 

Printing backtrace from core file

Post by Paul Tombl » Sun, 29 Jun 1997 04:00:00




>> Alternatively, is there a signal handler I could add to the program
>> that would print out a stack trace before dying *without* dumping
>> core?
>Paul,
>    Try man trace_back_stack. If you set the signal hander for whatever
>signal your program recieves before it dumps core to be a function that
>implements trace_back_stack, I think you'll get the effect you're after.

This sounds promising.  Unfortunately it doesn't seem to be on my system (Irix
6.2).  Is there a subsystem I need to install?

--
Paul Tomblin, Current Product Engineering team leader.
I don't speak for Kodak, they don't speak for me.

"You are in a twisty maze of Motif Widget resources, all inconsistent."

 
 
 

Printing backtrace from core file

Post by Walter Brisco » Mon, 30 Jun 1997 04:00:00


I am sorry that my - fairly snide - response did not respond to every
word in your original posting. I NOW know that you have read the FAQ. In
your situation, I would be inclined to investigate the balance between
shipping gdb and a configuration decision as to which de* to use if
I had customers who had no de* and living with a much reduced
number of core files. (I infer from "Unix in a Nutshell" that sdb and
dbx should cover most ports.) Regardless, the FAQ's technique seems good
to me as it uses work from someone else in matching the real and
conceptual machines. You could always look at a stripped down gdb but
that is WORK and a support nightmare. MY experience of traceback reports
comes from VMS where they are a normal response to an illegal operation
and where I would have considered killing for a dump as useful as core!




>>comp.unix.programmer FAQ: <URL: http://www.veryComputer.com/;

>>6.5 How can I generate a stack dump from within a running program?

>>With the information in that reference, you should be able to reduce your
>>users' ability to produce core files by a couple of orders of magnitude.

>To paraphrase what it says in the FAQ:  "Call some system specific thing if
>you're lucky, otherwise spawn dbx".  To recap what I said when I originally
>asked the question:



>>>we really need is the backtrace from the core file, but many of them don't
>                                                          ^^^^^^^^^^^^^^^^^^
>>>have dbx, and even the ones that do can't be bothered to start up dbx then
>  ^^^^^^^^

>Which is why I asked the question AFTER READING THE FAQ in the first place.

>Thanks but no thanks.

--
Walter Briscoe
 
 
 

Printing backtrace from core file

Post by Richard Col » Fri, 04 Jul 1997 04:00:00






>>> Alternatively, is there a signal handler I could add to the program
>>> that would print out a stack trace before dying *without* dumping
>>> core?

>>Paul,
>>        Try man trace_back_stack. If you set the signal hander for whatever
>>signal your program recieves before it dumps core to be a function that
>>implements trace_back_stack, I think you'll get the effect you're after.

> This sounds promising.  Unfortunately it doesn't seem to be on my system (Irix
> 6.2).  Is there a subsystem I need to install?

How about this...

Look on the MIT X11R6 distribution. In the utils subdirectory there used
to be a malloc debugging library. This included source for mips,ix86 and
sparc to parse the stack and generate a numeric backtrace. There was an
awk script to that took these addreses and run nm on the binary to produce
a verbose back stack trace...  It is also possible to obtain the symbol
names without using nm by reading the symbol table from the executable.
Depends how much trouble you want to go to... I did post an library
on alt.sources a little while back for a utility that part of what it
did was generate a stack back trace...

cheers

Rich

 
 
 

Printing backtrace from core file

Post by Alan Hawrylysh » Wed, 09 Jul 1997 04:00:00



> To paraphrase what it says in the FAQ:  "Call some system specific thing if
> you're lucky, otherwise spawn dbx".  To recap what I said when I originally
> asked the question:

> [snip]
> many of them don't have dbx.

What about adb?

Alternatively, (and this is ugly), what about writing a small stack
traversal routine yourself? Look in frame.h.  Catch the offending
signal, and just walk the stack back, we do that here with a piece of
assembly code that returns an array of up to 25 stack frame
pointers. (We are working with HP-UX 10.20).

A shell script could be written around nm and a few other tools to
postprocess the stack trace and the core file, producing parameters
and at least some symbol names.  If you have decent run-time symbol
support from a library of some-sorts, you could likely get the symbols
at crash time.

--
Alan Hawrylyshen           | The opinions expressed above are my own.
Vancouver, BC, Canada      | ICBM :  N04910.466' W12304.347' WGS84


 
 
 

1. Interpreting the results from backtrace()/ backtrace() usability

I've got the following output from backtrace_symbols_fd():

(There are a ton of [0x0]s in the beginning of the file)

...
[0x0]
[0x0]
[0x14000000]
/usr/local/sbin/lockmgr.4.5
[0x42029188]
[0x2e320000]
/usr/local/sbin/lockmgr.4.5
[0x804a758]
[0x2000]
[0x0]
[0x0]
[0x0]
[0x0]
[0x0]
[0x0]

...

[0x0]
[0x0]
[0x0]
[0x0]
[0x0]
[0x0]
[0x0]
[0x10000000]
/lib/ld-linux.so.2[0x40000812]
/usr/local/sbin/lockmgr.4.5[0x4201033a]
[0x4002c1a0]
/lib/ld-linux.so.2(_dl_lookup_versioned_symbol+0x11)[0x40007641]
[0x55a36f5]
/usr/local/sbin/lockmgr.4.5[0x804a7c7]
[0x3]
[0xbfffdef0]
[0xbfffdc0c]
/usr/local/sbin/lockmgr.4.5[0x804a7b7]
[0xe]
/usr/local/sbin/lockmgr.4.5[0x804a758]
[0xbfffdc0c]
/usr/local/sbin/lockmgr.4.5[0x804a7a8]
[0xc]
/lib/ld-linux.so.2[0x400005b8]
/lib/ld-linux.so.2[0x40000218]
/lib/ld-linux.so.2(_rtld_global+0x1c8)[0x400131e8]
[0x4]
[0x4002c294]
[0xbfffdf08]
/usr/local/sbin/lockmgr.4.5
[0x42029188]

I seem to be able to crossreference some 0x8???????es to the output of
nm. But what about the rest? The executable is compiled with debug info.

2. raw device fro Database

3. postmortem analysis without core, but with backtrace information

4. net connect?

5. When a core file is not a core file

6. Fix kobject_get oopses triggered by i2c in 2.5.65-bk

7. Code to print function call backtrace wanted

8. running a shell and logging its input and output

9. Get "Segmentation fault (core dumped)" but no core file found

10. Mysterious core file from "CORE"

11. XF86_SVGA core dumped, but where is the core file?

12. How to configure a Print Server to print a file as a file

13. where are files to convert ez files to ps files for printing?