CRLF Newlines and libc

[Eric]

I am working on the subprocess.Popen module for Google Summer of Code.
Right now, I am having difficulty trying to work out how to deal with
my '\n' newlines being converted to '\r\n' newlines when reading from
a pipe on windows; see this blog post (http://subdev.blogspot.com/
2009/07/stdout.html) and this chain on Python-Dev (http://
mail.python.org/pipermail/python-dev/2009-July/090720.html).

Right now, unless there is way to disable the newline conversions, I
am thinking I will just have to document the caveat and call it a day.
Is there a better way to handle this?

Eric

 

[Gabriel Genellina]

I am working on the subprocess.Popen module for Google Summer of Code.
Right now, I am having difficulty trying to work out how to deal with
my '\n' newlines being converted to '\r\n' newlines when reading from
a pipe on windows; see this blog post (http://subdev.blogspot.com/
2009/07/stdout.html) and this chain on Python-Dev (http://
mail.python.org/pipermail/python-dev/2009-July/090720.html).

Right now, unless there is way to disable the newline conversions, I
am thinking I will just have to document the caveat and call it a day.
Is there a better way to handle this?

It isn't clear to me what exactly your problem is. What are you reading,
using which functions, how did you open the file? On the writing side, how
did you open the file, which functions are used to write?

The Windows API functions like CreateFile, ReadFile, WriteFile... don't
perform any kind of '\n' conversion, neither reading nor writing. On the
other hand, the C library does differentiate between files opened in text
mode or binary mode (this comes from the ANSI C standard, it's not Windows
specific). The default is "text" mode (unless you add "b" to the fopen
call, or O_BINARY to the open call). In that mode, '\n' is translated to
'\r\n' on writing, and '\r\n' is translated to '\n' on reading.
Python files are implemented on top of the C library (on 2.x) so they
inherit that behaviour.
Console programs written in C and using printf/fwrite/write to stdout
share that behaviour too (standard handles are opened in text mode by
default).

Note that """'\n' newlines being converted to '\r\n' newlines when
reading""" cannot happen -- if you see '\r\n' when reading, it's because
a) you're reading the file in binary mode, and
b) the file was written in text mode.

b) is true for process output redirected to a pipe, so you'll have to fix
a). How to do that, depends on what exactly you're doing.