simple garbage problem

[cchayden.nyt]

When I run the program:

STDIN.each_line { |line| line.split("\t") }

with a big input file, the process size grows rapidly, reaching 1G in
less than 1 minute.
If I let it go, it continues to grow at this rate until the whole
system fails.
This is ruby 1.8.6 on fedora 9.

Is this a known problem?
Is there some way to work around it?

 

[Andrew Timberlake]

[Note: parts of this message were removed to make it a legal post.]

When I run the program:

STDIN.each_line { |line| line.split("\t") }

with a big input file, the process size grows rapidly, reaching 1G in
less than 1 minute.
If I let it go, it continues to grow at this rate until the whole
system fails.
This is ruby 1.8.6 on fedora 9.

Is this a known problem?
Is there some way to work around it?

I haven't done much input processing in ruby but I've used this with no
problems on large files:

while line = STDIN.gets
...
end

Just check that your file line breaks and the line breaks being used by
gets/each_line are the same.
Also make sure you actually have line breaks (sometimes you just gotta
check)

Andrew
http://ramblingsonrails.com

 

[Roger Pack]

Is this a known problem?

Is there some way to work around it?

Which version of ruby are you using? Very old ones have some leaks.
Besides that you may be able to apply some recent patches to GC [1] and
see if they help.
-=r
[1] http://www.ruby-forum.com/topic/170608

 

[Charles Oliver Nutter]

When I run the program:

STDIN.each_line { |line| line.split("\t") }

with a big input file, the process size grows rapidly, reaching 1G in
less than 1 minute.
If I let it go, it continues to grow at this rate until the whole
system fails.
This is ruby 1.8.6 on fedora 9.

Is this a known problem?
Is there some way to work around it?

Might be worth trying on JRuby. We've had no such reports, and the
garbage collector is rock-solid.

- Charlie