How to reclaim memory without GC.start

[dtuttle1]

Hi - I have a memory and cpu-constrained embedded app. There's a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
stream_host_connection.write buffer
stream_host_connection.flush
# GC.start - too expensive
end

Thanks, Dave

 

[Ari Brown]

Hi - I have a memory and cpu-constrained embedded app. There's a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
stream_host_connection.write buffer
stream_host_connection.flush
# GC.start - too expensive
end

Unfortunately, I don't know how to do it. If I did, I would include
it in every single one of my programs. But you should have an extra
'=' after buffer. It should read...
...
while buffer == file.read(512)
...

Note the double '='.

G'luck, and PLEASE pop an email to the list if you figure out how to
accomplish it.
-------------------------------------------|
Nietzsche is my copilot

 

[Todd Benson]

Unfortunately, I don't know how to do it. If I did, I would include
it in every single one of my programs. But you should have an extra
'=' after buffer. It should read...
...
while buffer == file.read(512)

No, Ari. I think you misread the program. There is an assignment to
buffer, not a comparison. The while loop will exit when
file.read(512) returns nil. I know why you thought it was a
comparison, though.

Todd

 

[Travis D Warlick Jr]

Hi - I have a memory and cpu-constrained embedded app. There's a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
stream_host_connection.write buffer
stream_host_connection.flush
# GC.start - too expensive
end

Someone please correct me if I'm wrong, but...

You don't need to deallocate buffer every time because it is not getting
duplicated at the beginning of every iteration -- it is being
overwritten, so the memory usage for buffer will always stay at 512
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you're really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration. However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.

--
Travis Warlick

"Programming in Java is like dealing with your mom --
it's kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend -- it's cold, unforgiving, and doesn't tell
you what you've done wrong."

 

[Lionel Bouton]

Travis D Warlick Jr wrote the following on 17.07.2007 01:31 :

Someone please correct me if I'm wrong, but...

You don't need to deallocate buffer every time because it is not getting
duplicated at the beginning of every iteration -- it is being
overwritten, so the memory usage for buffer will always stay at 512
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you're really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration. However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.

Actually, as variable affectation is done by reference and the *content*
of buffer is originally duplicated by File#read memory usage is not
likely to change linearly with the value passed to File#read...

You can still play with different values passed to it and expect some
change of behaviour (ie: depending on the actual amount of memory
allocated for each read, you may or may not hit a sweat spot in the GC
heuristics).

My advise would be to do larger reads and GC.start less often:

read_amount = 65536
read_count = 0
loops_between_gcs = 100
while buffer = file.read(read_amount)
stream_host_connection.write buffer
stream_host_connection.flush
GC.start if ((read_count += 1) % loops_between_gcs) == 0
end


But note that if stream_host_connection.flush doesn't actually wait
until the buffer isn't referenced anymore anywhere in your code (or in
some other process sitting on the same system as a matter of fact),
calling GC.start probably won't have any effect.

Lionel

 

[dtuttle1]

Someone please correct me if I'm wrong, but...

You don't need to deallocate buffer every time because it is not getting
duplicated at the beginning of every iteration -- it is being
overwritten, so the memory usage for buffer will always stay at 512
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you're really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration. However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.

--
Travis Warlick

"Programming in Java is like dealing with your mom --
it's kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend -- it's cold, unforgiving, and doesn't tell
you what you've done wrong."

The object is being overwritten, but the old copies aren't being
garbage collected. I know this because the memory does stay constant
if I call GC.start on every iteration.
I think I'll go with a c extension for this small part of the code.

Thanks, Dave

 

[Travis D Warlick Jr]

Actually, as variable affectation is done by reference

I knew that... woops... **beats head on keyboard**

However, A chance to redeem myself: I remembered that IO#read returns a
string, so change the while to:

while buffer[0...512] = file.read(512)

That should overwrite the contents of the buffer

--
Travis Warlick

"Programming in Java is like dealing with your mom --
it's kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend -- it's cold, unforgiving, and doesn't tell
you what you've done wrong."

 

[Joel VanderWerf]

Actually, as variable affectation is done by reference

I knew that... woops... **beats head on keyboard**

However, A chance to redeem myself: I remembered that IO#read returns a
string, so change the while to:

while buffer[0...512] = file.read(512)

That should overwrite the contents of the buffer

Do you mean file.read(512, buffer) ?


---------------------------------------------------------------- IO#read
ios.read([length [, buffer]]) => string, buffer, or nil
------------------------------------------------------------------------
Reads at most length bytes from the I/O stream, or to the end of
file if length is omitted or is nil. length must be a non-negative
integer or nil. If the optional buffer argument is present, it
must reference a String, which will receive the data.

At end of file, it returns nil or "" depend on length. ios.read()
and ios.read(nil) returns "". ios.read(positive-integer) returns
nil.

f = File.new("testfile")
f.read(16) #=> "This is line one"

 

[Travis D Warlick Jr]

Lionel Bouton wrote:

while buffer[0...512] = file.read(512)

Do you mean file.read(512, buffer) ?

Ahh, yes. That would be an much prettier way to do it.

while file.read(512, buffer)

--
Travis Warlick

"Programming in Java is like dealing with your mom --
it's kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend -- it's cold, unforgiving, and doesn't tell
you what you've done wrong."

 

[Florian Gross]

file = File.new file_path
while buffer = file.read(512)
stream_host_connection.write buffer
stream_host_connection.flush
# GC.start - too expensive
end

Note that IO#read supports a buffer argument:

buffer = ""
while file.read(512, buffer)
...
end

In that case it won't create a new string object for each call. It
will just modify buffer instead.

 

[Ari Brown]

No, Ari. I think you misread the program. There is an assignment to
buffer, not a comparison. The while loop will exit when
file.read(512) returns nil. I know why you thought it was a
comparison, though.

Yups. That is correct. My mistake, and thanks for pointing it out! At
least now I know I can do assignment in my 'while' statements. Right?

~ Ari
English is like a pseudo-random number generator - there are a
bajillion rules to it, but nobody cares.

 

[Joel VanderWerf]

Lionel Bouton wrote:

while buffer[0...512] = file.read(512)

Do you mean file.read(512, buffer) ?

Ahh, yes. That would be an much prettier way to do it.

while file.read(512, buffer)

It's not just prettier. With an assignment like

buffer[0...512] = file.read(512)

there is an intermediate string that instantly becomes garbage.

 

[Robert Klemme]

Travis D Warlick Jr wrote:
Lionel Bouton wrote:

while buffer[0...512] = file.read(512)

Do you mean file.read(512, buffer) ?

Ahh, yes. That would be an much prettier way to do it.

while file.read(512, buffer)

It's not just prettier. With an assignment like

buffer[0...512] = file.read(512)

there is an intermediate string that instantly becomes garbage.

There is a better way to do it:

buffer = "." * 1024
while STDIN.read(1024, buffer)
print buffer
end

So the original code should be rewritten as

File.open file_path do |file|
buffer = "." * 512
while file.read(512, buffer)
stream_host_connection.write buffer
stream_host_connection.flush
end
end

The main point being here to reuse the buffer. No C extension needed.
See http://www.ruby-doc.org/core/classes/IO.html#M002295

Kind regards

robert

 

[dtuttle1]

2007/7/17, Joel VanderWerf <[email protected]>:

Joel VanderWerf wrote:
Travis D Warlick Jr wrote:
Lionel Bouton wrote:
while buffer[0...512] = file.read(512)
Do you mean file.read(512, buffer) ?
Ahh, yes. That would be an much prettier way to do it.
while file.read(512, buffer)

It's not just prettier. With an assignment like

buffer[0...512] = file.read(512)

there is an intermediate string that instantly becomes garbage.

There is a better way to do it:

buffer = "." * 1024
while STDIN.read(1024, buffer)
print buffer
end

So the original code should be rewritten as

File.open file_path do |file|
buffer = "." * 512
while file.read(512, buffer)
stream_host_connection.write buffer
stream_host_connection.flush
end
end

The main point being here to reuse the buffer. No C extension needed.
Seehttp://www.ruby-doc.org/core/classes/IO.html#M002295

Kind regards

robert

It works perfectly! Thanks everyone!

 

[Erik Veenstra]

buffer = ""

while file.read(512, buffer)
...
end

You could save one more line:

while file.read(512, buffer||="")
...
end

gegroet,
Erik V. - http://www.erikveen.dds.nl/

 

[Erik Veenstra]

buffer = "." * 1024

while STDIN.read(1024, buffer)
print buffer
end

What's the point of initializing the buffer with 1024 characters,
instead of just initializing it with an empty string? I must be
missing something...

Thanks.

gegroet,
Erik V.

 

[Ari Brown]

Finally! I've been waiting for someone to resurrect this thread.

Ok. I've decided that if a program consumes ALL of a computer's
available memory (free, inactive), then GC.start is run, it will be
able to free all that RAM.

I have done several scientific tests with this on World of Warcraft
(heh. work on WoW) and image rendering.

So what's the best way to consume all of a computer's RAM? I've been
toying with the idea of massive variable generation, but i think that
might be a bit difficult.

So what do you all think?
---------------------------------------------------------------|
~Ari
"I don't suffer from insanity. I enjoy every minute of it" --1337est
man alive

 

[Robert Klemme]

I think the idea was to do the allocation upfront instead of when witting to
the string. But since that will only happen once, (on the first read) it
seems unnecessary. But I could be wrong.

I believe you are right. I probably rather did it for documentary
reasons but in fact it's not necessary. There would only be a small
but likely unnoticeable advantage if for some reason #read would not
read up to limit chars right away. Then there might be multiple
reallocations (i.e. until the buffer has reached the limit). But I
guess this is not very likely, at least not with file IO. Situation
might be different for sockets.

Kind regards

robert

 

[Robert Klemme]

Finally! I've been waiting for someone to resurrect this thread.

Ok. I've decided that if a program consumes ALL of a computer's
available memory (free, inactive), then GC.start is run, it will be
able to free all that RAM.

I have done several scientific tests with this on World of Warcraft
(heh. work on WoW) and image rendering.

So what's the best way to consume all of a computer's RAM? I've been
toying with the idea of massive variable generation, but i think that
might be a bit difficult.

So what do you all think?

I'm not sure what you're after. Consuming memory is easy. For example:

a=[];a << " "*1024 while true

Or, a bit more involved

Chain = Struct.new :data, :next
c = nil
c = Chain.new("."*1024, c) while true

Kind regards

robert

 

[Todd Benson]

Finally! I've been waiting for someone to resurrect this thread.

Ok. I've decided that if a program consumes ALL of a computer's
available memory (free, inactive), then GC.start is run, it will be
able to free all that RAM.

I have done several scientific tests with this on World of Warcraft
(heh. work on WoW) and image rendering.

So what's the best way to consume all of a computer's RAM? I've been
toying with the idea of massive variable generation, but i think that
might be a bit difficult.

So what do you all think?

Why oh why are you asking for this, Ari?

Todd