O_DIRECT on stdin?

[Dan Stromberg]

Is there a way of setting O_DIRECT on a preexisting file like sys.stdin?

Does C allow this sort of thing?

Thanks!

 

[Gordon Burditt]

Is there a way of setting O_DIRECT on a preexisting file like sys.stdin?

Does C allow this sort of thing?

There is no O_DIRECT or fcntl() in Standard C.

fcntl() operates on an open file descriptor, and the file descriptor
for stdin is 0. Two calls to fcntl(), one with F_GETFL and one
with F_SETFL, would do what you want.

I'm not sure why you want to do that, though. It's not going to
get you character-at-a-time I/O, if that's what you want.

Gordon L. Burditt

 

[jepler]

I think this is fcntl(..., F_SETFL, ...), so something like
import os, fcntl, sys
flags = fcntl.fcntl(sys.stdin.fileno(), fcntl.F_GETFL)
flags |= os.O_DIRECT
fcntl.fcntl(sys.stdin.fileno(), fcntl.F_SETFL, flags)

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDb7CcJd01MZaTXX0RAi5+AJ0eeXXROgvPtbcg6br8Zf/KsGPAxgCeLFb0
xNIG26j8JMJIecGiNWmcU0E=
=2LQx
-----END PGP SIGNATURE-----

 

[Dan Stromberg]

[quoted text muted]

There is no O_DIRECT or fcntl() in Standard C.

fcntl() operates on an open file descriptor, and the file descriptor
for stdin is 0. Two calls to fcntl(), one with F_GETFL and one
with F_SETFL, would do what you want.

I'm not sure why you want to do that, though. It's not going to
get you character-at-a-time I/O, if that's what you want.

Gordon L. Burditt

I want to be able to read a HUGE file without having such a negative
impact on the system's buffer cache.

I'm trying:

if hasattr(os, 'O_DIRECT'):
try:
flags = fcntl.fcntl(sys.stdin.fileno(), fcntl.F_GETFL)
flags |= os.O_DIRECT
fcntl.fcntl(sys.stdin.fileno(), fcntl.F_SETFL, flags)
except:
sys.stderr.write('Setting O_DIRECT on stdin attempted but failed\n')
else:
sys.stderr.write('Setting O_DIRECT on stdin succeeded \n')

....but while this code doesn't error out, I get:

seki-root> reblock -e $[1024*1024*80] $[1024*1024] 300 < /dev/sda1 > /dev/null
stdin seems seekable, but file length is 0 - no exact percentages
Estimated filetransfer size is 85899345920 bytes
Estimated percentages will only be as accurate as your size estimate
Setting O_DIRECT on stdin succeeded
Traceback (most recent call last):
File "/Dcs/seki/strombrg/bin/reblock", line 276, in ?
main()
File "/Dcs/seki/strombrg/bin/reblock", line 222, in main
block = os.read(0,blocksize)
OSError: [Errno 22] Invalid argument
Mon Nov 07 12:25:53

....but if I comment out the fcntl/O_DIRECT code, then the same thing works
well.

Any other ideas folks?

Thanks!

 

[Gordon Burditt]

I want to be able to read a HUGE file without having such a negative

impact on the system's buffer cache.

I'm trying:

if hasattr(os, 'O_DIRECT'):
try:
flags = fcntl.fcntl(sys.stdin.fileno(), fcntl.F_GETFL)
flags |= os.O_DIRECT
fcntl.fcntl(sys.stdin.fileno(), fcntl.F_SETFL, flags)
except:
sys.stderr.write('Setting O_DIRECT on stdin attempted but failed\n')
else:
sys.stderr.write('Setting O_DIRECT on stdin succeeded \n')

...but while this code doesn't error out, I get:

seki-root> reblock -e $[1024*1024*80] $[1024*1024] 300 < /dev/sda1 > /dev/null
stdin seems seekable, but file length is 0 - no exact percentages
Estimated filetransfer size is 85899345920 bytes
Estimated percentages will only be as accurate as your size estimate
Setting O_DIRECT on stdin succeeded
Traceback (most recent call last):
File "/Dcs/seki/strombrg/bin/reblock", line 276, in ?
main()
File "/Dcs/seki/strombrg/bin/reblock", line 222, in main
block = os.read(0,blocksize)
OSError: [Errno 22] Invalid argument
Mon Nov 07 12:25:53

...but if I comment out the fcntl/O_DIRECT code, then the same thing works
well.

Any other ideas folks?

Does O_DIRECT perhaps invoke some of the restrictions of "raw"
device files, where the current offset and transfer size must be a
multiple of some block size? (I don't see any mention of that in
FreeBSD's documentation.) What is the value of blocksize at the
time of the traceback above? I suggest keeping it well under 2G.

Gordon L. Burditt

 

[Dan Stromberg]

[quoted text muted]

Does O_DIRECT perhaps invoke some of the restrictions of "raw"
device files, where the current offset and transfer size must be a
multiple of some block size? (I don't see any mention of that in
FreeBSD's documentation.) What is the value of blocksize at the
time of the traceback above? I suggest keeping it well under 2G.

Gordon L. Burditt

I'm not aware of such a restriction on O_DIRECT, but then I just learned
of O_DIRECT's existence earlier today.

The value of blocksize at the time of the error should be 1 megabyte.

 

[jepler]

Here's some text from my open(2) manpage:
Transfer sizes, and the alignment of user buffer and file offset must all
be multiples of the logical block size of the file system.
It's unlikely that in practice you can get Python's sys.stdin.read() or
os.read() to reliably use a buffer that fits the alignment restriction.

However, a small "C" extension could provide something like os.read() which
*does* meet the restriction. The meat of it would look something like
#define PAGESIZE 4096 // a guess which may be right for x86 linux
#define REQUESTSIZE 1048576

ssize_t result;
char *buf, *base;
PyObject *pyresult;

buf = malloc(bufsize, REQUESTSIZE + PAGESIZE - 1);
base = round_up(buf, PAGESIZE);

result = read(0, base, REQUESTSIZE);

if(result == -1) {
set python error from errno
pyresult = NULL;
goto DONE;
}

pyresult = PyString_FromStringAndSize(base, result);

DONE:
free(buf);
return pyresult;

Here's a clever but untested "C" macro that claims to implement round_up:
#define round_up(amount, align) ((((amount) - 1) | ((align) - 1)) + 1)

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDb8iGJd01MZaTXX0RAm/AAKCjJmBlQ/Lzu5kEnnA/GzqLeeE03QCffJWP
0T/vsxfB6dLlr4kvwaOtJjE=
=3Pyy
-----END PGP SIGNATURE-----

 

[Donn Cave]

Here's some text from my open(2) manpage:
Transfer sizes, and the alignment of user buffer and file offset must
all
be multiples of the logical block size of the file system.

Does that apply in the example he gave, < /dev/sda1 ?

It seems to me this would not go through any filesystem anyway.
That might account for the "invalid argument" error, but at any
rate it would be irrelevant.

Plus it doesn't seem to score very high on portability, according
to the Linux man page I'm looking at -- apparently not a POSIX
or any such standard, just borrowed from Irix in recent Linux
versions, and FreeBSD with slightly different behavior. Don't
see any trace of it in NetBSD, MacOS X.

It's unlikely that in practice you can get Python's sys.stdin.read() or
os.read() to reliably use a buffer that fits the alignment restriction.

Though of course os.read() would eliminate one layer of buffering
altogether. Might be worth a try.

Donn Cave, (e-mail address removed)

 

[Alex Fraser]

I want to be able to read a HUGE file without having such a negative
impact on the system's buffer cache.

[snip]
Does O_DIRECT perhaps invoke some of the restrictions of "raw"
device files, where the current offset and transfer size must be a
multiple of some block size?

Very likely. It is also likely that the same applies to the destination (ie
memory) address.

Alex