Getting a stack trace of a running Ruby program.

[Ron M]

Is there a way of displaying the stack trace of a running ruby program without killing it?

Java virtual machines(*) will show the stack trace(s) of a running program
if you send them a QUIT signal (or on windows <ctrl><break>) without stopping
the program. On Linux, the "pstack" program gives you the same capabilities
for C programs.

This is useful when a program runs for days and gets gradually slower. By
periodically sending QUIT signals you can essentally poll the program's
stack to see where it's getting slow.

Is there any way to do similar in Ruby? Even if I need to attach using
GDB, it would help me - but I'd need someone to point me to what data
structures I should look at with the debugger. Otherwise, I think it'd
be a really nice feature if, like java, a QUIT signal could show the
stack trace(s) of the existing nthread(s) without killing an application


Ron

* http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/
* http://linuxcommand.org/man_pages/pstack1.html

 

[Robert Klemme]

Is there a way of displaying the stack trace of a running ruby
program without killing it?
Java virtual machines(*) will show the stack trace(s) of a running
program if you send them a QUIT signal (or on windows <ctrl><break>)
without
stopping the program. On Linux, the "pstack" program gives you the same
capabilities for C programs.

This is useful when a program runs for days and gets gradually
slower. By periodically sending QUIT signals you can essentally poll the
program's stack to see where it's getting slow.

Is there any way to do similar in Ruby? Even if I need to attach
using GDB, it would help me - but I'd need someone to point me to what
data
structures I should look at with the debugger. Otherwise, I think
it'd be a really nice feature if, like java, a QUIT signal could show the
stack trace(s) of the existing nthread(s) without killing an
application

Ron

*
http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/
* http://linuxcommand.org/man_pages/pstack1.html

I prefer to use real profiling tools instead of looking at a single
stacktrace. In Java that would be -Xrunhprof, OptimizeIt et al. In Ruby
you can use -r profile for profiling. Also, there's module Benchmark which
is less intrusive than the profiler. Finally there's set_trace_func which
can be used to generate call stacks for each thread and print them to
console on a signal similarly to your Java approach.

HTH

robert

 

[Stephen Kellett]

Is there a way of displaying the stack trace of a running ruby program
without killing it?

Ruby Performance Validator and/or Ruby Bug Validator can do this.

http://www.softwareverify.com

Stephen

 

[Ron M]

Is there a way of displaying the stack trace of a running ruby
[like] Java QUIT signal
[or] the "pstack" program gives you the same capabilities for C programs.

I prefer to use real profiling tools instead of looking at a single
stacktrace. In Java that would be -Xrunhprof, OptimizeIt et al. In
Ruby you can use -r profile for profiling. Also, there's module
Benchmark which is less intrusive than the profiler. Finally there's
set_trace_func which can be used to generate call stacks for each thread
and print them to console on a signal similarly to your Java approach.

HTH

These all work fine for relatively short-lived programs; but
do little to help in a progam that runs for days and only gets
slow after many hours of running.

On short runs (few hours) my program works fine (processes
dozens of records per second) but after half a day or so
it slows to taking a few seconds per record. "-r profile"
and set_trace_func are pretty intrusive and I fear my
program that runs for days would take weeks. Benchmark
has the disadvantage of needing to know what piece of code
to measure; rather than exposing the slow part.

But the biggest advantage in my mind of sampling instantaneous
snapshots of the stack is that it works on any program witout
modification (well, linux's ptrace is most useful if you didn't
strip signals); and it's almost totally non-intrusive, and does
not skew the timing results at all.

 

[Eric Hodel]

Is there a way of displaying the stack trace of a running ruby
program without killing it?

Java virtual machines(*) will show the stack trace(s) of a running
program
if you send them a QUIT signal (or on windows <ctrl><break>)
without stopping
the program. On Linux, the "pstack" program gives you the same
capabilities
for C programs.

trap('QUIT') { puts caller.join("\n") }

Probably better to trap a signal other than QUIT, like USR1

 

[Ara.T.Howard]

Is there a way of displaying the stack trace of a running ruby
[like] Java QUIT signal
[or] the "pstack" program gives you the same capabilities for C programs.

I prefer to use real profiling tools instead of looking at a single
stacktrace. In Java that would be -Xrunhprof, OptimizeIt et al. In Ruby
you can use -r profile for profiling. Also, there's module Benchmark which
is less intrusive than the profiler. Finally there's set_trace_func which
can be used to generate call stacks for each thread and print them to
console on a signal similarly to your Java approach.

HTH

These all work fine for relatively short-lived programs; but
do little to help in a progam that runs for days and only gets
slow after many hours of running.

On short runs (few hours) my program works fine (processes
dozens of records per second) but after half a day or so
it slows to taking a few seconds per record. "-r profile"
and set_trace_func are pretty intrusive and I fear my
program that runs for days would take weeks. Benchmark
has the disadvantage of needing to know what piece of code
to measure; rather than exposing the slow part.

But the biggest advantage in my mind of sampling instantaneous
snapshots of the stack is that it works on any program witout
modification (well, linux's ptrace is most useful if you didn't
strip signals); and it's almost totally non-intrusive, and does
not skew the timing results at all.

so send it a signal that makes it fork and dump core - then you can do

gdb ruby core.1234

whamo.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| anything that contradicts experience and logic should be abandoned.
| -- h.h. the 14th dalai lama
===============================================================================

 

[Robert Klemme]

Is there a way of displaying the stack trace of a running ruby
[like] Java QUIT signal
[or] the "pstack" program gives you the same capabilities for C
programs.

I prefer to use real profiling tools instead of looking at a single
stacktrace. In Java that would be -Xrunhprof, OptimizeIt et al. In
Ruby you can use -r profile for profiling. Also, there's module
Benchmark which is less intrusive than the profiler. Finally there's
set_trace_func which can be used to generate call stacks for each
thread and print them to console on a signal similarly to your Java
approach.

HTH

These all work fine for relatively short-lived programs; but
do little to help in a progam that runs for days and only gets
slow after many hours of running.

On short runs (few hours) my program works fine (processes
dozens of records per second) but after half a day or so
it slows to taking a few seconds per record. "-r profile"
and set_trace_func are pretty intrusive and I fear my
program that runs for days would take weeks. Benchmark
has the disadvantage of needing to know what piece of code
to measure; rather than exposing the slow part.

But the biggest advantage in my mind of sampling instantaneous
snapshots of the stack is that it works on any program witout
modification (well, linux's ptrace is most useful if you didn't
strip signals); and it's almost totally non-intrusive, and does
not skew the timing results at all.

How about this?


Signal.trap "INT" do
fork do
ObjectSpace.each_object(Thread) do |th|
th.raise Exception, "Stack Dump" unless Thread.current == th
end
raise Exception, "Stack Dump"
end
end

Thread.new do
10.times do
puts "."
sleep 5
end
end

10.times do
puts "."
sleep 5
end


This is non intrusive.

robert

 

[Ron M]

How about this?
[cool code fragment omitted]
This is non intrusive.

Thanks! As always I'm impressed with Ruby that
every time I find something I wanted from another
platform you guys (you and Eric Hodel both in this case)
give me a handful of lines of code that enable Ruby with
everything I had needed.

 

[Joel VanderWerf]

Is there a way of displaying the stack trace of a running ruby program
without killing it?

Java virtual machines(*) will show the stack trace(s) of a running program
if you send them a QUIT signal (or on windows <ctrl><break>) without
stopping
the program. On Linux, the "pstack" program gives you the same
capabilities
for C programs.

This is useful when a program runs for days and gets gradually slower. By
periodically sending QUIT signals you can essentally poll the program's
stack to see where it's getting slow.

Is there any way to do similar in Ruby? Even if I need to attach using
GDB, it would help me - but I'd need someone to point me to what data
structures I should look at with the debugger. Otherwise, I think it'd
be a really nice feature if, like java, a QUIT signal could show the
stack trace(s) of the existing nthread(s) without killing an application


Ron

* http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/
* http://linuxcommand.org/man_pages/pstack1.html

Yes, this is a very late comment to this thread, but as it happens I
needed something like this last week. Unfortunately none of the proposed
solutions in this thread would have worked for me, because I was
debugging a multithreaded ruby process. (I was debugging from inside the
process using irb while other threads ran.)

Robert Klemme's suggestion to fork and raise an exception is nice, but
only works an a single thread. A forked ruby process keeps only one
thread from the parent (unless something has changed since 1.8.2):

threads = (0..3).map {Thread.new {sleep}}

p Thread.list.size # ==> 5

sleep 1

fork do
p Thread.list.size # ==> 1
end

sleep 1

In any case, #fork doesn't work on all platforms.

Eric Hodel's suggestion to trap some signal and use #caller is also
nice, but only gives you information about one thread (the main thread,
IIRC).

Robert also suggested set_trace_func. But that doesn't tell you where
your threads are sleeping, unless you've enabled the trace func before
the threads went to sleep. Keeping set_trace_func enabled can have
serious overhead costs.

It would be very nice to be able to ask a thread for its current backtrace.

 

[Stephen Kellett]

It would be very nice to be able to ask a thread for its current backtrace.

Yes. From what I've seen of the internals the current implementation
does not have this information, well not in a format that is retrievable
anyway. I'd like to be proved wrong on this point!

Stephen