GC error ?

[Piotr Sawicki]

Run this program and observe memory usage.

<code>

def leaker
a = ""
10000.times do
a = a + "buble"
end
a = ""
end


loop do
leaker
end

</code>

 

[Robert Klemme]

Run this program and observe memory usage.

<code>

def leaker
a = ""
10000.times do
a = a + "buble"
end
a = ""
end


loop do
leaker
end

</code>

Your point being?

robert

 

[Piotr Sawicki]

Your point being?

robert

I run this on ruby-1.8.6 (Linux and cygwin [WinXP]).
Why memory used by ruby process grow?
Is that normal behaviour ?

I wrote the same progam on python and perl and I have discover that that
this problem occur only in ruby.

 

[Gary Wright]

Your point being?

robert

I run this on ruby-1.8.6 (Linux and cygwin [WinXP]).
Why memory used by ruby process grow?
Is that normal behaviour ?

I ran that code via irb and compared the memory usage to running
'loop {}' in irb.

Your code used more memory but it oscillated around a fixed point
that was about 10 megabytes larger than the empty loop process, it
didn't grow.

This was on Mac OS X 10.4, ruby 1.8.6.

 

[Rados³aw Bu³at]

R3V5cywgaXQncyB3b3J0aCB0byBzZW5kIGl0IHRvIHJ1YnktY29yZSBhbmQgYXNrIHRoZXJlLiBJ
IGNoZWNrZWQKcnVieTEuOSBhbmQgaXQgaGFzIHRoZSBzYW1lIGlzc3VlIDovLgoKCgotLSAKUmFk
b3OzYXcgQnWzYXQKCmh0dHA6Ly9yYWRhcmVrLmpvZ2dlci5wbCAtIG3zaiBibG9nCg==

 

[Robert Klemme]

Your point being?

I run this on ruby-1.8.6 (Linux and cygwin [WinXP]).
Why memory used by ruby process grow?
Is that normal behaviour ?

I wrote the same progam on python and perl and I have discover that that
this problem occur only in ruby.

Well, concluding from the postings not all versions of Ruby on all
platforms suffer this phenomenon. As far as I can see so far you have
proven that

(1) on cygwin on Windows XP and on Linux
(2) Ruby version 1.8.6
(3) will allocate and free memory in a way that
(4) the OS shows continuous increasing size of used memory
(5) for a particular program, namely one that allocates memory at a high rate.

Whether that's a bug or not is probably really a question for ruby
core. There might be multiple causes for this observed behavior
including but not limited to Ruby's GC.

Kind regards

robert

 

[Piotr Sawicki]

I have simple, but ugly solution for this problem.

class String
alias plus +
def +(a)
res = self.plus(a)
GC.start
return res
end
end

 

[Robert Klemme]

I have simple, but ugly solution for this problem.

class String
alias plus +
def +(a)
res = self.plus(a)
GC.start
return res
end
end

That's not a solution. That's a bad workaround which might degrade
performance of other applications.

Cheers

robert

 

[Justin Collins]

I have simple, but ugly solution for this problem.

class String
alias plus +
def +(a)
res = self.plus(a)
GC.start
return res
end
end

Obviously this is a just an example to illustrate the issue, but

def leaker
a = ""
10000.times do
a << "buble"
end
a = ""
end


loop do
leaker
end


does not exhibit the same memory usage as the OP.

-Justin

 

[Christopher Dicely]

To perhaps underscore Robert's point, I did this in IRB on the OCI
version on WinXP, and its behavior appeared (just watching memory
usage in the Windows task manager) very similar to the with-gc variant
on cygwin, even though it was run without explicit GC calls.
(Interestingly, about 27 MB of the excess over what IRB was using
before running this remained in use afterward until an explicit
GC.start, and the explicit call to GC only dropped that by about 21
MB.)

Your point being?

I run this on ruby-1.8.6 (Linux and cygwin [WinXP]).
Why memory used by ruby process grow?
Is that normal behaviour ?

I wrote the same progam on python and perl and I have discover that that
this problem occur only in ruby.

Well, concluding from the postings not all versions of Ruby on all
platforms suffer this phenomenon. As far as I can see so far you have
proven that

(1) on cygwin on Windows XP and on Linux
(2) Ruby version 1.8.6
(3) will allocate and free memory in a way that
(4) the OS shows continuous increasing size of used memory
(5) for a particular program, namely one that allocates memory at a high rate.

Whether that's a bug or not is probably really a question for ruby
core. There might be multiple causes for this observed behavior
including but not limited to Ruby's GC.

Kind regards

robert

 

[evanwebb]

Run this program and observe memory usage.

<code>

def leaker
    a = ""
    10000.times do
         a = a + "buble"
    end
    a = ""
end

loop do
    leaker
end

</code>

Not a bug in any way. It's simply that the garbage collector has not
been trigger. It runs only when it needs to. As you realized you, you
can force it to run by doing 'GC.start'. If you run 'GC.start' at the
end of your program, you'll see the memory reclaimed.

This is a typical behavior for a mark/sweep garbage collector. Perl
and Python are reference count, freeing memory the instant it goes out
of scope. Ruby waits to free memory until it needs memory to free some.

 

[Piotr Sawicki]

Not a bug in any way. It's simply that the garbage collector has not
been trigger. It runs only when it needs to. As you realized you, you
can force it to run by doing 'GC.start'. If you run 'GC.start' at the
end of your program, you'll see the memory reclaimed.

This is a typical behavior for a mark/sweep garbage collector. Perl
and Python are reference count, freeing memory the instant it goes out
of scope. Ruby waits to free memory until it needs memory to free some.

Yes, I agree. I analyzed source code of gc.c. GC was triggered each time
when malloc_increase reach malloc_limit. Variable malloc_limit was
increased by factor proportional to (malloc_increase - malloc_limit). In
each loop cycle malloc_increase > malloc_limit because malloc_increase
is sum of arithmetic series of string length between GC calls.

 

[Piotr Sawicki]

<code>
def leaker
a = Array.new(1_000_000) {|i| "blah"}
end

loop do
printf "%10s", `ps h -o rss #{$$}`
leaker
end
</code>

Why this program doesn't cause memory leaks?

Beacuse in this program

<code>
a = ""
10000.times do
a = a + "buble"
end
</code>

in each cycle ruby creates a new string with length:
<No. of cycle> * "buble".length + 1 ('\0')

No. of cycle Heap
------------ ----------
1) a = "buble"
2) a = "bublebuble" , a' = "buble"
3) a = "bublebublebuble", a' = "bublebuble", a'' = "buble"
etc.

so after n-cycles we have
S = n * (("buble".length + 1) + (("buble".length * n + 1)) / 2
in my program:
n = 10000
"buble".size = 5
S = 10000 * (5 + 1 + 50000 + 1)/2 = 5000 * 50007 = 250_035_000 bytes

In your program the strings occupy only
1_000_000 * ("blah".length + 1) = 6_000_000 bytes

 

[Piotr Sawicki]

You refer to cycles inside method "leaker". But this method always
creates the same amount of objects so I don't see reason why ruby eats
more and more memory for each "leaker" call.

Haw gc works ? Here is pseudocode:

malloc_limit = 8_000_000
malloc_increase = 0;

malloc(size)
{
malloc_increase += size;
if (malloc_increase > malloc_limit)
garbage_collect()
}

garbage_collect()
{
malloc_limit = malloc_limit + (malloc_increase - malloc_limit) *
some_factor
malloc_increase = 0
}

 

[Eliot Miranda]

Haw gc works ? Here is pseudocode:

malloc_limit = 8_000_000
malloc_increase = 0;

malloc(size)
{
malloc_increase += size;
if (malloc_increase > malloc_limit)
garbage_collect()
}

garbage_collect()
{
malloc_limit = malloc_limit + (malloc_increase - malloc_limit) *
some_factor
malloc_increase = 0
}

In my day they were called libraries. Now its called "wikipedia".

http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)

But efore you read up, ask yourself how you *think* it might work. One
hint. Its paradoxical. To first collect garbage you could identify
everything you don't wan to throw away.

HTH

 

[Rick DeNatale]

In my day they were called libraries. Now its called "wikipedia".

http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)

I hadn't seen this article before, a quick skim and I'd say it looks
like a decent survey, although I'd like to see a little more credit
given to some of the GC innovators:

Henry Baker,
Dave Ungar
Eliot Miranda ?

But efore you read up, ask yourself how you *think* it might work. One
hint. Its paradoxical. To first collect garbage you could identify
everything you don't wan to throw away.

Yes indeed, unless you want a potentially unpleasant surprise! Far
worse for a GC to throw away non-garbage than not to throw away all
garbage.

 

[Paul Brannan]

Yes indeed, unless you want a potentially unpleasant surprise! Far
worse for a GC to throw away non-garbage than not to throw away all
garbage.

I've known people who have such a policy (in the real world). And then
they wonder why they can't find anything...

Paul

 

[Eliot Miranda]

I hadn't seen this article before, a quick skim and I'd say it looks
like a decent survey, although I'd like to see a little more credit
given to some of the GC innovators:

Henry Baker,
Dave Ungar
Eliot Miranda ?

I wish I've never done anything innovative in the GC field.
Just implemeted and/or enhanced a few.

People like Carl Hewitt, L. Peter Deutsch, Hans Boehm, Richard Hudson,
Eliot Moss, Urs Hölzle (and Ungar and Baker) have innovated.

Yes indeed, unless you want a potentially unpleasant surprise! Far
worse for a GC to throw away non-garbage than not to throw away all
garbage.

I meant that most algorithms identify non-garbage, leaving garbage to be
found implicitly (e.g. the space left behind after a scavenging/copying
GC). Only ref counting explicitly identifies garbage, and proves to be
expensive since in OO languages most objects die young.

 

[Rick DeNatale]

I meant that most algorithms identify non-garbage, leaving garbage to be
found implicitly (e.g. the space left behind after a scavenging/copying
GC). Only ref counting explicitly identifies garbage, and proves to be
expensive since in OO languages most objects die young.

Yes, and it also suffers from two facts that make it perform
sub-optimally on a virtual memory system (i.e. just about any system
these days)

1) It fails to compact the live objects
2) It pretty much is guaranteed to visit every memory page at least
twice per GC cycle.

This tends to lead to large working sets and the attendant swapping.