memory leak

[Joe Van Dyk]

Got a memory leak in the following code.. not sure if DRb is necessary
in it or not.

Put the code into a file named test.rb, btw.

I think I tracked it down to the SystemVIPC::MessageQueue#recv method.
If I don't give it a high receive length (like 8192), I don't see the
leak. If I do give it a high length, the program takes up a very
large amount of space pretty quickly... about an increase of a meg
every five seconds.

Any ideas? (On my linux system, kernel.msgmax is set to 10000 if it matters)

# test.rb

require 'drb'
require 'sysvipc'

class Foo
def initialize
@key = SystemVIPC::ftok("test.rb", 1)
@msg = SystemVIPC::MessageQueue.new(@key, 0777 | SystemVIPC::IPC_CREAT)
end

def recv
continue = true
while continue
begin
@msg.recv(1, 8192, SystemVIPC::IPC_NOWAIT)
rescue
continue = false
end
end
end


def send
@msg.send(1, "HELLO WORLD HELLO WORLD")
end
end

f = Foo.new
some_thread = Thread.new(f) do |foo|
loop do
foo.recv
sleep 1
end
end
some_thread = Thread.new(f) do |foo|
loop do
foo.send
sleep 1
end
end

DRb.start_service('druby://localhost:8000', f)
DRb.thread.join


def send
@msg.send(1, "HELLO WORLD HELLO WORLD")
end
end

f = Foo.new
some_thread = Thread.new(f) do |foo|
loop do
foo.recv
sleep 1
end
end
some_thread = Thread.new(f) do |foo|
loop do
foo.send
sleep 1
end
end

DRb.start_service('druby://localhost:8000', f)
DRb.thread.join

 

[Joe Van Dyk]

eek. copy/paste borked.

See below for correct version.

Got a memory leak in the following code.. not sure if DRb is necessary
in it or not.

Put the code into a file named test.rb, btw.

I think I tracked it down to the SystemVIPC::MessageQueue#recv method.
If I don't give it a high receive length (like 8192), I don't see the
leak. If I do give it a high length, the program takes up a very
large amount of space pretty quickly... about an increase of a meg
every five seconds.

Any ideas? (On my linux system, kernel.msgmax is set to 10000 if it matters)

# test.rb

require 'drb'
require 'sysvipc'

class Foo
def initialize
@key = SystemVIPC::ftok("test.rb", 1)
@msg = SystemVIPC::MessageQueue.new(@key, 0777 | SystemVIPC::IPC_CREAT)
end

def recv
continue = true
while continue
begin
@msg.recv(1, 8192, SystemVIPC::IPC_NOWAIT)
rescue
continue = false
end
end
end


def send
@msg.send(1, "HELLO WORLD HELLO WORLD")
end
end

f = Foo.new
some_thread = Thread.new(f) do |foo|
loop do
foo.recv
sleep 1
end
end
some_thread = Thread.new(f) do |foo|
loop do
foo.send
sleep 1
end
end

DRb.start_service('druby://localhost:8000', f)
DRb.thread.join

 

[John Carter]

Got a memory leak in the following code.. not sure if DRb is necessary
in it or not.

Try
http://rubyforge.org/snippet/detail.php?type=snippet&id=70

It is really _very_ trivial to use. (it is also pretty brute force and
dumb code)

Save to snippet as MemoryProfile.rb and then run your code as

ruby -w -r MemoryProfile your_code.rb

On exit it reports simple counts and also traverses the object tree to
give detailed counts of what object is using what and prints the
largest object as well. The log file is appended to /tmp/memory_profile.log

It takes quite a lot of time and memory...

It also dumps /proc/pid/status to display your vm usage, so it is a
little Linux specific, but you can cure that with a # or two.

John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : (e-mail address removed)
New Zealand

Refactorers do it a little better every time.

 

[Ara.T.Howard]

Try
http://rubyforge.org/snippet/detail.php?type=snippet&id=70

It is really _very_ trivial to use. (it is also pretty brute force and
dumb code)

Save to snippet as MemoryProfile.rb and then run your code as

ruby -w -r MemoryProfile your_code.rb

On exit it reports simple counts and also traverses the object tree to
give detailed counts of what object is using what and prints the
largest object as well. The log file is appended to /tmp/memory_profile.log

It takes quite a lot of time and memory...

It also dumps /proc/pid/status to display your vm usage, so it is a
little Linux specific, but you can cure that with a # or two.

very cool. thanks for this.

cheers.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| although gold dust is precious, when it gets in your eyes, it obstructs
| your vision. --hsi-tang
===============================================================================

 

[Joe Van Dyk]

eek. copy/paste borked.

See below for correct version.

FYI, you'll need SysVIPC installed
(http://deisui.org/~ueno/ruby/sysvipc.html).

In essence, the code below starts a thread that uses a object to read
from a message queue repeatedly. If there's no messages left on the
message queue, the #recv method throws an exception, we catch that and
exit the loop. The reader thread then sleeps for a second and reads
more messages from the queue.

The problem doesn't seem to be with the sending of the message over
the message queue, rather its the receive. Somewhere, something isn't
being thrown away. I used ObjectSpace to monitor the number of
objects that existed. The total number of Objects in memory seem to
fluctuate, but don't increase over time (while memory is increasing).

I looked at the C code for the SysVIPC module but couldn't see any
obvious errors.

Any thoughts?

Joe

 

[Ara.T.Howard]

FYI, you'll need SysVIPC installed
(http://deisui.org/~ueno/ruby/sysvipc.html).

In essence, the code below starts a thread that uses a object to read
from a message queue repeatedly. If there's no messages left on the
message queue, the #recv method throws an exception, we catch that and
exit the loop. The reader thread then sleeps for a second and reads
more messages from the queue.

The problem doesn't seem to be with the sending of the message over
the message queue, rather its the receive. Somewhere, something isn't
being thrown away. I used ObjectSpace to monitor the number of
objects that existed. The total number of Objects in memory seem to
fluctuate, but don't increase over time (while memory is increasing).

I looked at the C code for the SysVIPC module but couldn't see any
obvious errors.

Any thoughts?

can you run it under electric fence?

Joe

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| although gold dust is precious, when it gets in your eyes, it obstructs
| your vision. --hsi-tang
===============================================================================

 

[ts]

J> I looked at the C code for the SysVIPC module but couldn't see any
J> obvious errors.

xcalloc() called without a free(), no ?


Guy Decoux

 

[Joe Van Dyk]

J> I looked at the C code for the SysVIPC module but couldn't see any
J> obvious errors.

xcalloc() called without a free(), no ?

Guy Decoux

I saw that. But I didn't see how it could be free'd (but my C-fu sucks).

Here's the relevant code:

============================
static VALUE
rb_msg_recv (argc, argv, obj)
int argc;
VALUE *argv, obj;
{
VALUE v_type, v_len, v_flags;
int flags = 0;
struct msgbuf *msgp;
struct ipcid_ds *msgid;
long type;
size_t rlen, len;

rb_scan_args (argc, argv, "21", &v_type, &v_len, &v_flags);
type = NUM2LONG (v_type);
len = NUM2INT (v_len);
if (!NIL_P (v_flags))
flags = NUM2INT (v_flags);

msgp = xcalloc (sizeof (long) + len, sizeof (char));
msgid = get_ipcid (obj);

retry:
TRAP_BEG;
rlen = msgrcv (msgid->id, msgp, len, type, flags);
TRAP_END;
if (rlen == (size_t)-1)
{
switch (errno)
{
case EINTR:
rb_thread_schedule ();
case EWOULDBLOCK:
#if EAGAIN != EWOULDBLOCK
case EAGAIN:
#endif
goto retry;
}
rb_sys_fail ("msgrcv(2)");
}

return rb_str_new (msgp->mtext, rlen);
}
============================

msgp is included in the return statement, so I'm not sure how it could
be free'd. Isn't its memory being used by the new Ruby string? And
shouldn't that memory be garbage collected at some point?

Joe

 

[ts]

J> I saw that. But I didn't see how it could be free'd (but my C-fu sucks).

perhaps use alloca() (with memzero()) rather than malloc().


Guy Decoux

 

[Joe Van Dyk]

J> I saw that. But I didn't see how it could be free'd (but my C-fu sucks).

perhaps use alloca() (with memzero()) rather than malloc().

Is that standard practice with Ruby extensions?

 

[ts]

J> Is that standard practice with Ruby extensions?

Only the author of the extension know the response to this question ...


Guy Decoux

 

[Joe Van Dyk]

J> Is that standard practice with Ruby extensions?

Only the author of the extension know the response to this question ...

Guy Decoux

Thanks for your help. I think I fixed it. I haven't given it hugely
extensive testing, but it fixes my problems. Essentially, I free'd
everything that was xcalloc'd.

Patch is below. I'm new to this whole patching/fixing thing business,
so not sure if its in the right format or what.

--- sysvipc.c.old 2005-04-27 09:54:17.000000000 -0700
+++ sysvipc.c 2005-04-27 10:03:52.000000000 -0700
@@ -209,6 +209,7 @@
rb_sys_fail ("msgsnd(2)");
}

+ free(msgp);
return obj;
}

@@ -223,6 +224,7 @@
struct ipcid_ds *msgid;
long type;
size_t rlen, len;
+ VALUE return_str;

rb_scan_args (argc, argv, "21", &v_type, &v_len, &v_flags);
type = NUM2LONG (v_type);
@@ -249,10 +251,13 @@
#endif
goto retry;
}
+ free(msgp);
rb_sys_fail ("msgrcv(2)");
}

- return rb_str_new (msgp->mtext, rlen);
+ return_str = rb_str_new(msgp->mtext, rlen);
+ free(msgp);
+ return return_str;
}

static void
@@ -329,6 +334,7 @@
for (i = 0; i < nsems; i++)
rb_ary_push (dst, INT2FIX (array));

+ free(array);
return dst;
}

@@ -351,6 +357,7 @@
array = NUM2INT (RARRAY(ary)->ptr);
semctl (semid->id, 0, SETALL, array);

+ free(array);
return obj;
}

@@ -463,6 +470,7 @@

if (semop (semid->id, array, nsops) == -1)
rb_sys_fail ("semop(2)");
+ free(array);
return obj;
}