Piping stdout to Python callable

[Edward Diener]

From within a function in my own module I need to take the output from a
Python module "A", which is sending data to stdout and which can not be
changed and which I need to invoke as a top-level module, and pipe it into
another function in my own module so that I can read it from stdin. Is there
an easy way to do this ? The only way I can presently think to do this is
through "system python A.py | python MyOwnModule.py", which seems a bit
laborious having to invoke python.exe itself twice. Any other solution would
be most welcome.

 

[Antoon Pardon]

From within a function in my own module I need to take the output from a
Python module "A", which is sending data to stdout and which can not be
changed and which I need to invoke as a top-level module, and pipe it into
another function in my own module so that I can read it from stdin. Is there
an easy way to do this ? The only way I can presently think to do this is
through "system python A.py | python MyOwnModule.py", which seems a bit
laborious having to invoke python.exe itself twice. Any other solution would
be most welcome.

What do you mean when you say you need to invoke it as a top-level
module? Do you mean you can't import it at all or that importing it will
startup the process of generating output immediatly?

What bothers you with twice invoking the interpreter? In these days
a program that is invoked multiples times will generally be only
loaded once in memory.

Are threads an acceptable alternative? Does your MyOwnModule.py needs
to write to stdout?

 

[Edward Diener]

What do you mean when you say you need to invoke it as a top-level
module? Do you mean you can't import it at all or that importing it
will startup the process of generating output immediatly?

I mean that it has a "if __name__ == '__main__' line and I need to trigger
it by calling 'python A.py'.

What bothers you with twice invoking the interpreter?

Nothing practically. Just seems inelegant.

In these days
a program that is invoked multiples times will generally be only
loaded once in memory.

Shared libraries may be loaded once in memory but python.exe itself gets
reloaded each time.

Are threads an acceptable alternative? Does your MyOwnModule.py needs
to write to stdout?

Threads are acceptable. MyOwnModule.py can do anything, depending on
parameters, but the idea is that a certain parameter tells it to read from
stdin on the other end of the pipe.

I have implemented this via "os.system("python A.py | python MyOwnModule.py
parameters") and it works fine. I thought there might be a better, more
elegant way but since the above works without any problem I will stick to it
unless you can suggest anything better. BTW "A.py" can be any number of
types of modules which must be invoked by python.exe and writes it's results
to stdout. I am telling you this lest you suggest that I somehow import and
parse A.py in order to call directly into it rather than having python.exe
invoke it. While that might be possible, it would be a real programming
PITA.

 

[Rich Krauter]

I mean that it has a "if __name__ == '__main__' line and I need to trigger
it by calling 'python A.py'.




Nothing practically. Just seems inelegant.




Shared libraries may be loaded once in memory but python.exe itself gets
reloaded each time.




Threads are acceptable. MyOwnModule.py can do anything, depending on
parameters, but the idea is that a certain parameter tells it to read from
stdin on the other end of the pipe.

I have implemented this via "os.system("python A.py | python MyOwnModule.py
parameters") and it works fine. I thought there might be a better, more
elegant way but since the above works without any problem I will stick to it
unless you can suggest anything better. BTW "A.py" can be any number of
types of modules which must be invoked by python.exe and writes it's results
to stdout. I am telling you this lest you suggest that I somehow import and
parse A.py in order to call directly into it rather than having python.exe
invoke it. While that might be possible, it would be a real programming
PITA.

You could do something like this:

$ cat A.py
def test(arg):
for i in range(1,5):
print arg*i

$ cat B.py
import A
import sys
import StringIO

sys.stdout = StringIO.StringIO()
A.test(5)
fileobj,sys.stdout = sys.stdout,sys.__stdout__

fileobj.seek(0)
for line in fileobj:
print line.strip()


Worth the effort to change your code? Probably not. Kinda neat, though.

Rich

 

[Rich Krauter]

[silly post snipped]

Oh, duh. You you said you need to invoke the code in the
if __name__ == '__main__': part of the code. Sorry, I saw your post
yesterday and didn't register that then. Obviously my suggestion doesn't
help you.

Well, unless A.py looks something like

def main():
...
...

if __name__ == '__main__:
main()


Rich

 

[Antoon Pardon]

I mean that it has a "if __name__ == '__main__' line and I need to trigger
it by calling 'python A.py'.

Nothing practically. Just seems inelegant.


Shared libraries may be loaded once in memory but python.exe itself gets
reloaded each time.

That depends on the O.S. It is possible the O.S. notices that an
invoked program is already loaded en uses the same code-segment
in memory for following invocations.

Threads are acceptable. MyOwnModule.py can do anything, depending on
parameters, but the idea is that a certain parameter tells it to read from
stdin on the other end of the pipe.

Then I fear there is no other solution. As far as I understand, you
can't redirect standard output on a per thread basis, (at least not
in python.) That means that if you redirect stdout of the A.py module
to go into a pipe, the stdout of MyOwnModule.py will go into the
same pipe. I don't think you want that.

 

[Edward Diener]

That depends on the O.S. It is possible the O.S. notices that an
invoked program is already loaded en uses the same code-segment
in memory for following invocations.


Then I fear there is no other solution. As far as I understand, you
can't redirect standard output on a per thread basis, (at least not
in python.) That means that if you redirect stdout of the A.py module
to go into a pipe, the stdout of MyOwnModule.py will go into the
same pipe. I don't think you want that.

Just wanted to write that someone else pointed out to me os.popen("python
A.py") and that did work well as an alternative to using os.system(etc.) .
Nonetheless thanks for your help. I mentioned this so that you would know
also know about it yourself if you hadn't encountered it.

 

[Antoon Pardon]

Just wanted to write that someone else pointed out to me os.popen("python
A.py") and that did work well as an alternative to using os.system(etc.) .
Nonetheless thanks for your help. I mentioned this so that you would know
also know about it yourself if you hadn't encountered it.

I know about popen, but that didn't seem to resolve your concern.
Working with popen will result in launching a second interpreter.
Since one of your mayor concerns about your own solution was
having two interpreters running, I didn't thought popen was a
viable solution.