Buffering of sys.stdout and sys.stderr in python3 (and documentation)

[Geoff Bache]

Hi all,

Short version:

I'm a bit confused in general as to the changes between python2 and
python3 regarding how standard output and standard error do buffering.
A few things seem to have changed and I've failed to find any
documentation of how and why. Also, the meaning of "python -u" seems
to have changed and the docs don't seem to reflect the new behaviour
(and I can't find any docs about the change either)...

Long version:

From rude experiment it seems that:
1) In Python 2.x, standard error was always unbuffered while standard
output was buffered by default. In python3, both are buffered. In
both cases, "buffered" means line-buffered when writing to the console
and not line-buffered when redirected to files.
2) In Python 2.x, the "-u" flag meant everything was totally
unbuffered. In Python 3.x, it means that both stdout and stderr are
line-buffered also when redirected to files.

Are either of these changes documented anywhere? (1) seems important :
it can lead to not seeing exception printouts, if stderr is redirected
to a file and the program is subsequently terminated with SIGTERM. I
just wasted quite a bit of time due to this situation...

This is what the Python 3 docs have to say about the -u flag:

"Force the binary layer of the stdin, stdout and stderr streams (which
is available as their buffer attribute) to be unbuffered. The text I/O
layer will still be line-buffered."

The "still" seems misleading to me, as it is only relevant if writing
to the console. It would be useful to contrast the behaviour with and
without "-u" when writing to files I would say.

Regards,
Geoff Bache

 

[Terry Reedy]

Hi all,

Short version:

I'm a bit confused in general as to the changes between python2 and
python3 regarding how standard output and standard error do buffering.
A few things seem to have changed and I've failed to find any
documentation of how and why. Also, the meaning of "python -u" seems
to have changed and the docs don't seem to reflect the new behaviour
(and I can't find any docs about the change either)...

Long version:

1) In Python 2.x, standard error was always unbuffered while standard
output was buffered by default. In python3, both are buffered. In
both cases, "buffered" means line-buffered when writing to the console
and not line-buffered when redirected to files.
2) In Python 2.x, the "-u" flag meant everything was totally
unbuffered. In Python 3.x, it means that both stdout and stderr are
line-buffered also when redirected to files.

Are either of these changes documented anywhere? (1) seems important :
it can lead to not seeing exception printouts, if stderr is redirected
to a file and the program is subsequently terminated with SIGTERM. I
just wasted quite a bit of time due to this situation...

This is what the Python 3 docs have to say about the -u flag:

"Force the binary layer of the stdin, stdout and stderr streams (which
is available as their buffer attribute) to be unbuffered. The text I/O
layer will still be line-buffered."

The "still" seems misleading to me, as it is only relevant if writing
to the console. It would be useful to contrast the behaviour with and
without "-u" when writing to files I would say.

The difference from 2.x should be in What's New in 3.0, except that the
new i/o module is in 2.6, so it was not exactly new. You might be able
to find more in http://python.org/dev/peps/pep-3116/

You *should* be able to find sufficient info in the 3.x docs. If, after
you get other responses (or not), you think the docs need upgrading,
open an issue on the tracker at bugs.python.org with suggestions as
specific as possible, including changed or new lines of text based on
your experience and experiments.

 

[Geoff Bache]

Hi Terry,

The difference from 2.x should be in What's New in 3.0, except that the
new i/o module is in 2.6, so it was not exactly new.

The io module existed in 2.6, but it was not used by default for
standard output and standard error. The only mention of this in
"What's New in 3.0" is in the section marked for changes that were
already in 2.6 (which is wrong in this case), and it notes only that
io.TextIOWrapper is now used, but not what implications that has for
its behaviour and backward compatibility.

You might be able
to find more inhttp://python.org/dev/peps/pep-3116/

I skimmed through it but couldn't find anything relevant. It seems
more "advanced" and implementation-focussed.

You *should* be able to find sufficient info in the 3.x docs. If, after
you get other responses (or not), you think the docs need upgrading,
open an issue on the tracker at bugs.python.org with suggestions as
specific as possible, including changed or new lines of text based on
your experience and experiments.

OK, I'll do that if nobody points me at some existing docs here.

Regards,
Geoff Bache

 

[swatkins]

It's surprising and broken that stderr should be buffered in python3. python3 calls setvbuf(3) on stderr at startup to achieve this chuckle-headed behavior. It makes stderr line buffered if on a terminal, and fully bufferedif redirected to a log file. A fully buffered stderr is a very bad idea.

This change goes against the C89 and C99 standards, 40+ years of precedent,and the behavior of nearly every other current programming language.

Error messages will not show up in logs on time, and may be completely lostif the process is terminated before the buffer is flushed, e.g. with SIGTERM

How about fixing this?

Until then, we should write this in every script:

sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 0)

 

[swatkins]

sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 0)

which unfortunately doesn't work! I guess will resort to python3 -u, although I don't want stdout to be unbuffered.