Python Subprocess module

[Dave Sampson]

hey folks,

A simple question hopefully. despite all my searching I have not found a
satisfactory response.

The goal. Interact with a command line program. Simple enough, but the
key is INTERACT.

I tried the shell and comand approaches but that initiates, it does not
allow interaction with the programs.

So then I went with Popen and such... which then led to the subprocess
module.

I can create the object and read a few lines of output. but if I go too
far then the program hangs. the number of lines will differ depandening
on many function including the format of an input file so I can;t
hardcode how many lines to read.

I want to read all of STDOUT without failing because I went out of range.

next I want to read the final line of the output because it tells me
what is required for the next line of input.

I am supposed to be able to entre 'y' and return for the program to
continue if I agree with what I see in the stdout.

A problem exists though that I have tried
'y'
'y\n'
'y\r'

and nothing seems to get the program going again for I still cant; read
past the same point in the standard output. then I have to kill and
start over.

So the next approach included looking at Pexpect, which got realy
confusing realy fast and despite running fedora core and python 2.4.4 I
would like my application to be cross platform and there is no Pexpect
for Windows That I can see.

I have checked out many mailing lists and posts and tutorials but they
all admit to not be samples of complex interactivity. I guess I am
having that special case.

ASPN Python cookbok provided some code for a new Popen Class that allows
for interaction but I don't think I need to go that root....

I am essentialy trying to build and Python wrapper for a coomand line
program that i want to build a GUI around.

Any links or suggestions would be great. Unfortunaetly I don't
understand that abstract concepts in the manual reference. I am fresh in
the python world and find examples better than technical abstracts.

Cheers

 

[Dan Stromberg - Datallegro]

You could start up a pty and do your own Pexpect-like thing, using
read/readline/write.

However, this may not be portable to windows. Maybe if you get the python
that comes with cygwin, or build your own python under the cygwin
environment - that might be more likely to get you pty's under windows.

 

[Eric_Dexter]

hey folks,

A simple question hopefully. despite all my searching I have not found a
satisfactory response.

The goal. Interact with a command line program. Simple enough, but the
key is INTERACT.

I tried the shell and comand approaches but that initiates, it does not
allow interaction with the programs.

So then I went with Popen and such... which then led to the subprocess
module.

I can create the object and read a few lines of output. but if I go too
far then the program hangs. the number of lines will differ depandening
on many function including the format of an input file so I can;t
hardcode how many lines to read.

I want to read all of STDOUT without failing because I went out of range.

next I want to read the final line of the output because it tells me
what is required for the next line of input.

I am supposed to be able to entre 'y' and return for the program to
continue if I agree with what I see in the stdout.

A problem exists though that I have tried
'y'
'y\n'
'y\r'

and nothing seems to get the program going again for I still cant; read
past the same point in the standard output. then I have to kill and
start over.

So the next approach included looking at Pexpect, which got realy
confusing realy fast and despite running fedora core and python 2.4.4 I
would like my application to be cross platform and there is no Pexpect
for Windows That I can see.

I have checked out many mailing lists and posts and tutorials but they
all admit to not be samples of complex interactivity. I guess I am
having that special case.

ASPN Python cookbok provided some code for a new Popen Class that allows
for interaction but I don't think I need to go that root....

I am essentialy trying to build and Python wrapper for a coomand line
program that i want to build a GUI around.

Any links or suggestions would be great. Unfortunaetly I don't
understand that abstract concepts in the manual reference. I am fresh in
the python world and find examples better than technical abstracts.

Cheers

There is something called http://www.rutherfurd.net/python/sendkeys/
send keys for the p.c. you can use a try: except: concept and on the
windows platform there is also a way to use autoit in python using
pywin. At one time (2001) there was a module that allowed you to use
gvim but it has dissapeared unless it is listed somewhere else (gvim
doesn't seem to work with the sendkey method and it is a shame it has
alot of cool text functions including command line support)

http://sourceforge.net/projects/dex-tracker/

 

[Noah]

hey folks,

A simple question hopefully. despite all my searching I have not found a
satisfactory response.

The goal. Interact with a command line program. Simple enough, but the
key is INTERACT.

You need Pexpect.

So then I went with Popen and such... which then led to the subprocess
module. I can create the object and read a few lines of output. but if I go too
far then the program hangs. the number of lines will differ depandening
on many function including the format of an input file so I can;t
hardcode how many lines to read.

There is nothing you can do about this when using a pipe.
This is because stdio will change the type of buffering when sending
stdout and stderr to a pipe. From the client side there is no way you
can change the buffer mode (you CAN change the mode, but it only
changes
it on YOUR side of the pipe -- not your child application's side).

and nothing seems to get the program going again for I still cant; read
past the same point in the standard output. then I have to kill and
start over.

Your child application is waiting for you, but you can't see what it
wrote
becuase it has not filled or flushed the output buffer. When a program
write to stdout and it is connected to a TTY it will automatically
flush
the output buffer when it writes a newline (or, rather, the clib will
do this).
But when writing to a pipe it will automatically use block buffering
and
will only flush the buffer when it is full.

It's not very complicated, but it can be very confusing at first.
It is also surprising that this behavior cannot be changed by the
parent application.

So the next approach included looking atPexpect, which got realy
confusing realy fast and despite running fedora core and python 2.4.4 I
would like my application to be cross platform and there is noPexpect
for Windows That I can see.

You are going to need something like Pexpect that uses pseudo-ttys
(pty).
Pexpect works under Cygwin on Windows, but not under the native
Windows Python.

Email me if you have questions about Pexpect and I'll try to help you.

Yours,
Noah

 

[Josiah Carlson]

I am supposed to be able to entre 'y' and return for the program to
continue if I agree with what I see in the stdout.

A problem exists though that I have tried
'y'
'y\n'
'y\r'

Depending on the platform, you may need to send '\r\n'.

ASPN Python cookbok provided some code for a new Popen Class that allows
for interaction but I don't think I need to go that root....

You probably mean:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Why is that not sufficient? It will allow interaction with a called
subprocess, in a cross-platform manner, etc.


- Josiah

 

[Dave Sampson]

You could start up a pty and do your own Pexpect-like thing, using
read/readline/write.

However, this may not be portable to windows. Maybe if you get the python
that comes with cygwin, or build your own python under the cygwin

thanks for the suggestion. I will follow up on Noah's pexpect
sugestion bellow.

thanks for the input, cross platform is key with low overhead.

 

[Dave Sampson]

send keys for the p.c. you can use a try: except: concept and on the

windows platform there is also a way to use autoit in python using
pywin.

thanks for the uggestion about sendkeys. I might be able to build that
into the wrapper.

So you're sugesting:

NIX = Pexpect

WINDOZE = sendkeys

Correct?

 

[Dave Sampson]

Thanks for your response. It;s good to see you're promtoing Pexpect. I
was actualy wokring on a project at work that could have used Pexpect
to automate the creation of maps by tieing togethr a bunch of
processes through SSH and Pexpect looked like the money shot.

however it was enough of a battle to bring Python into the game since
it is government that bringing CYGWIN was a battle not worth
fighting. Which is unfortunate because I think Pexpect was exactly
what we needed in that case

You need Pexpect.

There is nothing you can do about this when using a pipe.
This is because stdio will change the type of buffering when sending
stdout and stderr to a pipe. From the client side there is no way you
can change the buffer mode (you CAN change the mode, but it only
changes
it on YOUR side of the pipe -- not your child application's side).

I obviously lack skills and understanding in many aspects of
programming. After all I'm a geographer / recreologist not a comp-sci
professional.

I understand the CONCEPT of pipes but don't know how to use or
implement.. the bigest things I ever do is something like

dmesg | grep wlan0

Where | is my pipe... but that is the extent of my knowledge and
skill.

Pexpect may very well be the option I need, and having a project that
supports the module is great. I am curous what is stopping Pexpect
from becoming cross platform.

Above, a user suggested implementing send keys in windows. Why not
include that as an try/except in Pexpect instead of my wrapper? That
way it can benifit many more people?

Cheers

 

[Dave Sampson]

You probably mean:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Why is that not sufficient? It will allow interaction with a called
subprocess, in a cross-platform manner, etc.

- Josiah

Sorry maybe 'not sufficient' was the wrong excuse. Maybe the right
thing to say is that when I look at it I don't know if I'm supposed to
copy paste in my own module, or paste into a seprate module, or use it
in a pre-existing module. Does the liscence for the content through
ASPN even allow reuse of code. or Am I supposed to modify it?

If I use, how is it to be used?. Wher eare the exmpales and
documentation.

By sufficient I mean I don't have sufficient info from the site to
implement it. And I don't have sufficient know how to implement it
base don the comments in the source code.

IN SHORT... IT'S SCARY....

So if you can answer any of those questions I am open.