Open a command pipe for reading

[Rodrick Brown]

I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line.

I've tried using Popen to do this with no luck. I'm trying to simulate

/bin/foo myfile.dat

And as the records are being printed do some calculations.

pipe = Popen(exttool,shell=True,stdout=PIPE).stdout

for data in pipe.readlines():
print data,

This operation blocks forever I'm guessing it's trying to process the entire file at once.

Sent from my iPhone 4.

 

[Lawrence D'Oliveiro]

Sent from my iPhone 4.

Glad to hear you achieved it without losing the signal.

 

[Stefan Schwarzer]

Hi Rodrick,

I have a fairly large file 1-2GB in size that I need to
process line by line but I first need to convert the file
to text using a 3rd party tool that prints the records
also line by line.

I've tried using Popen to do this with no luck. I'm trying
to simulate

/bin/foo myfile.dat

Is foo the 3rd-party conversion tool you've mentioned?

It would be good to have a bit more context, e. g. an actual
snippet from your code.

And as the records are being printed do some calculations.

pipe = Popen(exttool,shell=True,stdout=PIPE).stdout

for data in pipe.readlines():
print data,

This operation blocks forever I'm guessing it's trying to
process the entire file at once.

If you use `readlines` on a file object it reads the whole
file at once and returns a list of the lines (including line
end characters, by the way). What you probably want is

for line in pipe:
print line,

which reads and prints the file contents line by line.

Stefan