ruby memory allocation

[Emanuele Ricci]

I just discovered that ruby allocates memory in 8Mb chunks.
Forgive my ignorance.
Is there a way to change this behaviour? Since my app runs in an
environment with very strict memory constraints, a memory spike of 8Mb
can be lethal...
I would change it to a lower figure.
It doesn't matter if I have to patch my ruby source files in order to
achieve this goal.
Any clues?

Thank you in advance,
Emanuele Ricci.

 

[Chiyuan Zhang]

[Note: parts of this message were removed to make it a legal post.]

Here's an useful article by _why:
http://whytheluckystiff.net/articles/theFullyUpturnedBin.html

Maybe you can modify the parameters in gc.c to achieve your goals.

 

[M. Edward (Ed) Borasky]

I just discovered that ruby allocates memory in 8Mb chunks.
Forgive my ignorance.
Is there a way to change this behaviour? Since my app runs in an
environment with very strict memory constraints, a memory spike of 8Mb
can be lethal...
I would change it to a lower figure.
It doesn't matter if I have to patch my ruby source files in order to
achieve this goal.
Any clues?

Thank you in advance,
Emanuele Ricci.

That may not be Ruby's fault entirely. In many cases, the size of memory
chunks delivered by "malloc" and its friends is determined by the
compiler or operating system. I've worked on systems where you got the
next higher power of two, for example. If you asked for 5,000 you got
8192, if you asked for 65 you got 128, etc. And many modern systems
won't give you less than a physical page.

A physical page on most x86 operating systems is 4096 bytes, so 8 MB is
2048 pages. There is also an option to run larger pages in Linux. What
OS and architecture are you on?

P.S.: I'm cross-posting this to ruby-core -- that's where patches are
discussed.

 

[khaines]

I just discovered that ruby allocates memory in 8Mb chunks.
Forgive my ignorance.
Is there a way to change this behaviour? Since my app runs in an
environment with very strict memory constraints, a memory spike of 8Mb
can be lethal...
I would change it to a lower figure.

In the Ruby source, in gc.c:

#ifndef GC_MALLOC_LIMIT
#if defined(MSDOS) || defined(__human68k__)
#define GC_MALLOC_LIMIT 200000
#else
#define GC_MALLOC_LIMIT 8000000
#endif
#endif


Start there.


Kirk Haines

 

[Emanuele Ricci]

That may not be Ruby's fault entirely. In many cases, the size of memory
chunks delivered by "malloc" and its friends is determined by the
compiler or operating system. I've worked on systems where you got the
next higher power of two, for example. If you asked for 5,000 you got
8192, if you asked for 65 you got 128, etc. And many modern systems
won't give you less than a physical page.

A physical page on most x86 operating systems is 4096 bytes, so 8 MB is
2048 pages. There is also an option to run larger pages in Linux. What
OS and architecture are you on?

P.S.: I'm cross-posting this to ruby-core -- that's where patches are
discussed.

Thank you for answering.
My app runs in a linux gentoo environment on an arm processor embedded
device.
I cross-compile ruby with an arm-linux compiler in a gnu linux host.
The device has only 64Mb of ram and the last allocation of 8Mb happening
when the process is near 30-32Mb results in a NOEMEM error.

 

[M. Edward (Ed) Borasky]

Thank you for answering.
My app runs in a linux gentoo environment on an arm processor embedded
device.
I cross-compile ruby with an arm-linux compiler in a gnu linux host.
The device has only 64Mb of ram and the last allocation of 8Mb happening
when the process is near 30-32Mb results in a NOEMEM error.

Interesting ... not many people I know use Gentoo embedded. I run Gentoo
on my workstations and tried the cross-development chain on my Gumstix.
But it was a lot of work and the "native" Gumstix development tools were
a lot lighter weight.

Do you absolutely have to have Ruby 1.8? You might be able to use Ruby
1.6, which has fewer "features" and is a bit lighter weight as a result.
"miniruby" might also work. And there are other lighter-weight languages
-- Forth, of course, and Lua.

 

[Jano Svitok]

Do you absolutely have to have Ruby 1.8? You might be able to use Ruby 1.6,

I remember a thread where they used 1.6 and had problems with ruby crashes,
it was too late for them to upgrade and they offered a bounty for a fix
(though I don't remember what was the end of the story...)

Original thread:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/11368
this might be related:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/14085

1.6 might be unmaintained.

 

[Emanuele Ricci]

In the Ruby source, in gc.c:

#ifndef GC_MALLOC_LIMIT
#if defined(MSDOS) || defined(__human68k__)
#define GC_MALLOC_LIMIT 200000
#else
#define GC_MALLOC_LIMIT 8000000
#endif
#endif


Start there.


Kirk Haines

Thank you all for the very useful hints. I solved a problem I had for a
very long time. I thought it was a memory leak but I couldn't find any.
Now I know that it was simply ruby allocating memory. Due to the heavy
memory constraints I have in my enviroement (an arm-linux embedded
device) an allocation of 8Mb of memory could cause a crash of my app due
to not enough memomry.
To solve the problem I simply changed #define GC_MALLOC_LIMIT 8000000 to
#define GC_MALLOC_LIMIT 1000000 in gc.c. That is enough to solve my
problem and to not cause the garbage collector to be called too often.

Thank you again,
Emanuele Ricci.