newbie: unpacking binary data

[Mark Van de vyver]

Hi,
Thanks for the efforts put into making Ruby available!
While new to Ruby it seems worth investing some time in. Eventually I'd
like to unpack some data from a binary file and insert it into a MySQL
table.
At the moment I'm having trouble looping over the source data file, and
just viewing the data to make sure things are proceeding correctly.

FYI: eventually I need this to run quickly (many GB's to be processed),
but for the moment I'll settle for getting anything to work

The ruby details are:
$ ruby -v
ruby 1.8.5 (2006-08-25) [i386-cygwin]

-----------
Attempt 1 ( an error )
-----------
File.foreach("./INPUT/T200510A.BIN", "rb") do |f|
print f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
end
NoMethodError: [FATAL] failed to allocate memory

-----------
Attempt 2 ( an error )
-----------
File.open("./INPUT/T200510A.BIN", "rb").readlines do |f|
data = f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
print data
end
/usr/lib/ruby/1.8/irb.rb:298:in `inspect': failed to allocate memory
(NoMemoryEr
ror)
from /usr/lib/ruby/1.8/irb.rb:298:in `output_value'
from /usr/lib/ruby/1.8/irb.rb:151:in `eval_input'
from /usr/lib/ruby/1.8/irb.rb:259:in `signal_status'
from /usr/lib/ruby/1.8/irb.rb:147:in `eval_input'
from /usr/lib/ruby/1.8/irb/ruby-lex.rb:244:in
`each_top_level_statement'

from /usr/lib/ruby/1.8/irb/ruby-lex.rb:230:in `loop'
from /usr/lib/ruby/1.8/irb/ruby-lex.rb:230:in
`each_top_level_statement'

from /usr/lib/ruby/1.8/irb/ruby-lex.rb:229:in `catch'
from /usr/lib/ruby/1.8/irb/ruby-lex.rb:229:in
`each_top_level_statement'

from /usr/lib/ruby/1.8/irb.rb:146:in `eval_input'
from /usr/lib/ruby/1.8/irb.rb:70:in `start'
from /usr/lib/ruby/1.8/irb.rb:69:in `catch'
from /usr/lib/ruby/1.8/irb.rb:69:in `start'
from /usr/bin/irb:13

 

[Robert Klemme]

Hi,
Thanks for the efforts put into making Ruby available!
While new to Ruby it seems worth investing some time in. Eventually I'd
like to unpack some data from a binary file and insert it into a MySQL
table.
At the moment I'm having trouble looping over the source data file, and
just viewing the data to make sure things are proceeding correctly.

FYI: eventually I need this to run quickly (many GB's to be processed),
but for the moment I'll settle for getting anything to work

The ruby details are:
$ ruby -v
ruby 1.8.5 (2006-08-25) [i386-cygwin]

-----------
Attempt 1 ( an error )
-----------
File.foreach("./INPUT/T200510A.BIN", "rb") do |f|
print f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
end
NoMethodError: [FATAL] failed to allocate memory

This does not work because the second argument to foreach is the /line
separator/ - not the file mode.

-----------
Attempt 2 ( an error )
-----------
File.open("./INPUT/T200510A.BIN", "rb").readlines do |f|
data = f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
print data
end
/usr/lib/ruby/1.8/irb.rb:298:in `inspect': failed to allocate memory
(NoMemoryError)

This fails because you use readlines - this will try to read the whole
file into memory as /lines/ - and since there are no lines in a binary
file this chokes while attempting to store the whole file or a large
portion of it in a single line (i.e. String).

You should do this

File.open("./INPUT/T200510A.BIN", "rb") do |io|
while ( data = io.read(19) )
p data.unpack('I4I4I4I2I2a2a1')
end
end

Regards

robert

 

[mvyver]

On 14.12.2006 12:14, Mark Van de vyver wrote:

You should do this

File.open("./INPUT/T200510A.BIN", "rb") do |io|
while ( data = io.read(19) )
p data.unpack('I4I4I4I2I2a2a1')
end
end

Regards

robert

Thanks Robert. Much appreciated.