Errno::ENOMEM reading a device in Ruby, not in Java though

[Roger Pack]

Question on how to avoid an Errno::ENOMEM

Currently if I open a tape device from my machine [/dev/st2]
in ruby reading the second file from it results in:

read.rb:2:in `read': Cannot allocate memory - /dev/st2 (Errno::ENOMEM)
from read.rb:2

Java to do the same works, however.

the "cat" command [i.e. cat </dev/st2] also fails with "Not enough
memory"

Here is an strace of the two:

ruby:

open("/dev/st2", O_RDONLY) = 3
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff5721d900) = -1 EINVAL
(Invalid argument)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b5b53902000
read(3, 0x2b5b53902000, 4096) = -1 ENOMEM (Cannot allocate
memory)


java:

[pid 25476] open("/dev/nst2", O_RDONLY) = 4
[pid 25476] fstat(4, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 130),
...}) = 0
[pid 25476] mprotect(0x2aab3409b000, 65536, PROT_READ|PROT_WRITE) = 0
[pid 25476] read(4, "<familysearch-format>\n<type name"..., 65536) = 116

Anybody know if the difference between mmap and mprotect might be
causing this?
Thanks!
-=r

 

[Heesob Park]

Question on how to avoid an Errno::ENOMEM

Currently if I open a tape device from my machine [/dev/st2]
in ruby reading the second file from it results in:

read.rb:2:in `read': Cannot allocate memory - /dev/st2 (Errno::ENOMEM)
from read.rb:2

Java to do the same works, however.

the "cat" command [i.e. cat </dev/st2] also fails with "Not enough
memory"

Here is an strace of the two:

ruby:

open("/dev/st2", O_RDONLY) = 3
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff5721d900) = -1 EINVAL
(Invalid argument)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b5b53902000
read(3, 0x2b5b53902000, 4096) = -1 ENOMEM (Cannot allocate
memory)


java:

[pid 25476] open("/dev/nst2", O_RDONLY) = 4
[pid 25476] fstat(4, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 130),
...}) = 0
[pid 25476] mprotect(0x2aab3409b000, 65536, PROT_READ|PROT_WRITE) = 0
[pid 25476] read(4, "<familysearch-format>\n<type name"..., 65536) = 116

Anybody know if the difference between mmap and mprotect might be
causing this?

The another difference between ruby and java is read buffer size.
Ruby's buffer size is 4096 and java's buffer size is 65536.

The man page of st says:

RETURN VALUE

ENOMEM The byte count in read() is smaller than the next physi-
cal block on the tape. (Before 2.2.18 and 2.4.0-test6 the
extra bytes have been silently ignored.)

Refer to http://fts.ifac.cnr.it/cgi-bin/dwww?type=runman&location=st/4

Regards,

Park Heesob

 

[Roger Pack]

ge of st says:

|
|RETURN VALUE
|
| ENOMEM The byte count in read() is smaller than the next physi-
| cal block on the tape. (Before 2.2.18 and 2.4.0-test6 the
| extra bytes have been silently ignored.)

I didn't know that. In that case, you have to pre-allocate reading
buffer (string) and specify it to read method.

Interesting.
Perhaps somebody with brighter eyes can help me out with this?
[how to?]

a = File.open '/dev/nst2'Errno::ENOMEM: Cannot allocate memory - /dev/nst2

Also thinking out loud I wonder why java has 64kB blocks and ruby 4k? is
64 k any other benefit?

Thanks!
-=r

 

[Roger Pack]

b = " "*65000

a.read(b.size, b)

|>> a.read 65000
|Errno::ENOMEM: Cannot allocate memory - /dev/nst2

Hmm, since read with number pre-allocate specified sized buffer, it
should work. Perhaps you have to use sysread instead of mere read,
since read method retries until exact number of bytes specified read.

matz.

Interesting.
with 1.9, it will read all 64K at once (not with 1.8 though--it reads 4K
blocks until it reached 64K).

sysread did work with 1.8, though

for some reason I had to use exactly 64K block size.

file.sysread(64*1024)

guess something to remember if you have to read from a tape drive.

I did notice a few of these when using 1.9

ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfd172a8) = -1 ENOTTY
(Inappropriate ioctl for device)

dunno if that's cause for concern or not.
Cheers!
-=r

 

[Roger Pack]

Interesting.
with 1.9, it will read all 64K at once (not with 1.8 though--it reads 4K
blocks until it reached 64K).

sysread did work with 1.8, though

As a followup gotcha (in case anybody runs into this).

irb(main):001:0> a = File.open('/dev/nst2', 'rb')
=> #<File:/dev/nst2>
irb(main):002:0> a.eof?
irb(main):005:0> a.sysread 64*1024
IOError: sysread for buffered IO
from (irb):5:in `sysread'
from (irb):5

meant "when you call eof? it actually converts the file descriptor
internally into buffered mode, and read a byte from it to see if it
reads zero bytes (implying EOF), thus you cannot call sysread after a
buffered read call like eof?"

-r

 

[Roger Pack]

|irb(main):005:0> a.sysread 64*1024
|IOError: sysread for buffered IO ...
rewind it first to clear buffering.

Interesting. It seems to indeed allow for it.

I was unable to actually test it on a tape devicesince it appears that
eof? calls a read of size 8192:

read(3, 0x6c9ba90, 8192) = -1 ENOMEM (Cannot allocate
memory)

which would be smaller than what I would need [which is ok I can live
without eof].

Also as a note, it ends up that, with tape drives the trick is to read
from a device into a buffer that's at least as large as the largest
"block" of data written onto the tape, or else you'll get ENOMEM.

ex:
write 1000 bytes
File.open('tape_device like /dev/nst2', 'w') do |f| f.syswrite 'a*1000;
end
now read
File.open('tape_device', 'r') do |f| f.sysread 100; end # fails
File.open('tape_device', 'r') do |f| f.sysread 1000; end # succeeds

Nothing to do with ruby itself, but thought I'd mention it for
followers.
-r