How to get number of bytes written to nonblocking FIFO when EAGAIN is raised?

[Aaron Staley]

Scenario. I have a fifo named 'fifo' on my computer (ubuntu linux)
operating in nonblocking mode for both read and write. Under normal
operation all is good:

Interpreter 1 (writer)
Interpreter 2 (reader):'k'

However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)IOError: [Errno 11] Resource temporarily unavailable

However interpreter 2 still receives data65536

It looks like interpreter 1 pushed data until the FIFO was full and
then raised the IOError. Interpreter 2 constantly received some, but
not all, of what interpreter 2 tried to send.
Unfortunately, the IOError seems to have no attribute indicating how
much data was successfully sent. I've looked through the docs and
can't seem to figure out how; can anyone land some advice?

Thanks,
Aaron Staley

 

[Roy Smith]

Scenario. I have a fifo named 'fifo' on my computer (ubuntu linux)
operating in nonblocking mode for both read and write. Under normal
operation all is good:

Interpreter 1 (writer)[...]
Unfortunately, the IOError seems to have no attribute indicating how
much data was successfully sent. I've looked through the docs and
can't seem to figure out how; can anyone land some advice?

I'm thinking you want to skip the os.fdopen() call and use the file
descriptor directly, with os.write(). I've never used this, but it
seems like what you probably want to try.

 

[Adam Skutt]

However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)>>>f.write('a'*70000)

IOError: [Errno 11] Resource temporarily unavailable

However interpreter 2 still receives data>> len(f.read())

65536

It looks like interpreter 1 pushed data until the FIFO was full and
then raised the IOError.  Interpreter 2 constantly received some, but
not all, of what interpreter 2 tried to send.
Unfortunately, the IOError seems to have no attribute indicating how
much data was successfully sent.  I've looked through the docs and
can't seem to figure out how; can anyone land some advice?

You need to do as Roy Smith suggested and use the actual OS I/O calls
os.read() and os.write(). os.write() returns the number of bytes
actually written to the underlying descriptor. Python file objects
are akin to FILE structures in C: they perform buffering and other
operations internally that makes them less than suitable for usage
with asynchronous UNIX I/O. In particular, they buffer I/O internally
before writing it to the descriptor, so there's no direct relationship
between calling file.write() and how much data is written to the
stream. In addition, file objects also simply raise the underlying OS
error when it occurs. The UNIX write(2) syscall assumes that you have
been keeping track of how many bytes you've successfully written to
the stream and does not track it for you.

Adam

 

[Aaron Staley]

However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)>>> f.write('a'*70000)

IOError: [Errno 11] Resource temporarily unavailable

However interpreter 2 still receives data>> len(f.read())

It looks like interpreter 1 pushed data until the FIFO was full and
then raised the IOError.  Interpreter 2 constantly received some, but
not all, of what interpreter 2 tried to send.
Unfortunately, the IOError seems to have no attribute indicating how
much data was successfully sent.  I've looked through the docs and
can't seem to figure out how; can anyone land some advice?

You need to do as Roy Smith suggested and use the actual OS I/O calls
os.read() and os.write().  os.write() returns the number of bytes
actually written to the underlying descriptor.  Python file objects
are akin to FILE structures in C: they perform buffering and other
operations internally that makes them less than suitable for usage
with asynchronous UNIX I/O. In particular, they buffer I/O internally
before writing it to the descriptor, so there's no direct relationship
between calling file.write() and how much data is written to the
stream. In addition, file objects also simply raise the underlying OS
error when it occurs.  The UNIX write(2) syscall assumes that you have
been keeping track of how many bytes you've successfully written to
the stream and does not track it for you.

Adam

That's for the info; lower level I/O solved the issue.
That said, is such behavior with the file objects intended? It seems
they don't work correctly with an underlying non-blocking file
descriptor, but the documentation doesn't state such. In fact, it
suggests you can use non-blocking with this comment for file.read:

Also note that when in non-blocking mode, less data than was requested
may be returned, even if no size parameter was given.