Python, Tkinter and popen problem

[norseman]

This was sent 5/19/09 and as yet has received no comments.
I'm resending just in case a new reader might have an answer.
==================================



I have come across a problem that, as I understand all documentation I
have found, should not exist.

Python version is 2.5.2, Tinker in that package.
Linux Slackware 10.2

I went to test os.popen and got mixed answers.

1) IF os.popen opens a command line or command window type
THEN the read and writes work as expected
2) IF os.popen opens a Tkinter contained python program
THEN the read is not consistant and write to it fails.

OK - #1 runs as expected on shell scripts and other programs that were
written for the keyboard and redirectors only. (non GUI types)

#2 popen sometimes connects to stdout OF the child and sometimes
not and has never shown signs of connection to stdin OF child

The Tkinter usage has three (3) print statements. Two (2) never show up.
One does. The one that does is the exit notice.

In the 'child' program; print, sys.stdout.write(string), and
os.write(stdout,string) all work for the one that does write back, but
none work for the two that do not. Don't get picky on the syntax, the
purpose is to get the message across. (Actual syntax was adjusted per
documentation in each effort.)


The print statements are:
print "Notice of action"
print VariableHavingTextOfAction

print "Closing down\n"

Yes I tried adding the '\n' to the others - no change.

Same loop reads/prints FROM the child's stdout for all three prints.

All three work fine when Tkinter included program is run 'solo'.
Print statements show up in the text (command, whatever) window.
The entire Tkinter included program runs normally except for the I/O
linkage problem when using os.popen(..... or Popen2.popen2 or ... then
only third print shows up in text window. Acts as if first two prints
were captured but never forwarded to screen. If it was a case of not
sending until a system flush() took place then all three lines would
appear at the closing. Yes? That is NOT happening. Only closing notice
appears. With master having a print ":",returnVal to verify where the
text screen notices come from - the final print FROM the child is
verified as captured and printed from the master.

The child has a pause expecting a keystroke from the master immediately
following the print VariableH... which apparently is never received.
It is at this point the child hangs.


Again, it's a Python program running a Python program that uses Tkinter.


So what is going on and what can I do to get things running correctly?
I want the master to echo the child's print statements and the child get
the 'continue' character so it will.

Today is: 20090527
Versions noted above.

Steve

 

[Peter Otten]

This was sent 5/19/09 and as yet has received no comments.

I'm resending just in case a new reader might have an answer.

If you had posted two tiny scripts demonstrating your problem instead of the
longwinded explanation I might have tinkered.

Peter

 

[norseman]

If you had posted two tiny scripts demonstrating your problem instead of the
longwinded explanation I might have tinkered.

Peter

--------------------------

Since it got lost:
Python 2.5.2 with Tkinter in the download package
Linux Slackware 10.2

That's one of the problems - with Tkinter you don't get tiny files.

I have kept working on this and now have it to the point that sending TO
the child w/Tkinter works. Getting messages from the child now shows
that the flush() is not taking place in timely fashion. ALL print
statements FROM child do now flush and are shown correctly, but only
when the child terminates.

Question now is how to force a flush of stdout while the program is
running with tkinter in use.

tkinter section is print "string"
or sys.stdout.write("string")
sys.stdout.flush() fails totally
I don't care how it's flushed as long as it works on whatever machine
python runs on.


NOTE: program runs perfectly on both Linux and Windows XP Pro when run
from the keyboard. But not from another python program that wants the
phone line connected. (stdin/stdout) Gee... There is never a County
Lineman when needed is there Glen?


Steve

 

[MRAB]

--------------------------

Since it got lost:
Python 2.5.2 with Tkinter in the download package
Linux Slackware 10.2

That's one of the problems - with Tkinter you don't get tiny files.

[snip]

You're more likely to get help if you provide the minimum code that
demonstrates the problem.

Anyway, your original post said that you're running the child with
os.popen. The documentation says:

"""Open a pipe to or from command. The return value is an open file
object connected to the pipe, which can be read or written depending on
whether mode is 'r' (default) or 'w'."""

In other words, it's one-way.

Are you really using os.popen and not, say, os.popen2?

(Providing actual code would make this clear.)

 

[Peter Otten]

That's one of the problems - with Tkinter you don't get tiny files.

Nonsense. The minimal Tkinter program is

from Tkinter import Tk
root = Tk()
root.mainloop()

The idea is that you remove everything but the parts relevant to your
problem (or build a toy example from scratch). This makes it easier for a
non-guru to verify a potential answer. That non-guru might even be yourself,
by the way.

NOTE: program runs perfectly on both Linux and Windows XP Pro when run
from the keyboard. But not from another python program that wants the
phone line connected. (stdin/stdout) Gee... There is never a County
Lineman when needed is there Glen?

It's not clear why you need to start the GUI at all as you seem to have
control over boths scripts.

Peter

 

[MRAB]

norseman wrote:
[snip]

I have tried both and Popen2.popen2().
os.popen runs both way, contrary to docs.

# master.py
import os
# both lines work same
#xx= os.popen("/mnt/mass/py/z6.py").readlines()
xx= os.popen("/mnt/mass/py/z6.py",'r',1).readlines()

readlines() returns only when all the lines have been read, which is
when the child quits and the pipe closes. Try reading only one line at a
time. When the pipe closes the readline() will return ''.

# I had hoped small buffer would force a sync (flush())
# No such luck.
for i in xx:
print "\t"+i[:-1]

#"""
# end of file

The "\t" is to prove where the screen output came from.

========================================================
========================================================

From Peter (e-mail address removed)

Nonsense. The minimal Tkinter program is

from Tkinter import Tk
root = Tk()
root.mainloop()

Just to be clear, that's as much a minimal program as
[snip]
# child.py
import os
import sys
import array
from array import *
import Tkinter
from Tkinter import *
from Tkconstants import *
#
def AttPanel():

def PlaceIt():
sys.stdout.write( "Switching to ESRI for placement\n")
sys.stdout.flush()
sys.stdout.flush()

No need to flush twice in a row.

#print "Switching to ESRI for placement"
##zatt= bl_x+bl_y+acrs+c1+c2[2:]+c3[2:]+'\n'
zatt='123456\n'
#print zatt
sys.stdout.write(zatt)
sys.stdout.flush()

#set system variable to zatt
root.withdraw()
#root.iconify()
while raw_input() != ' ':
pass
root.deiconify()

def CRASH():
print "\nCRASH Initiated\n"
exit(1)
#
root = Tk()

LU = Frame(root)
LU.pack(fill="both", expand=1)

f1 = Frame(LU, relief = GROOVE, bd = 2)
f1.grid(row = 0, column = 0)
Button(f1, width= 45, state= DISABLED).grid(row= 0, column= 0)
Button(f1, text= "Place Attribute", fg= "black", bg= "green",
anchor= N, command = PlaceIt).grid(row = 0, column = 1)
Button(f1, width= 45, state= DISABLED).grid(row= 0, column= 2)
Button(f1, text= "Cancel Attributing", fg= "white", bg= "red",
anchor= E, command= CRASH).grid(row = 0, column = 3)
f1.pack()
#
root.mainloop()
#---------------#

if __name__ == "__main__":
while TRUE:
AttPanel()

 

[Peter Otten]

The direct question comes back to:
How does one force a sync or flush() to take effect in Python with
Tkinter in use? Or just in Python period. The keyword being force.

Here's some truly minimal code which shows the same buffering behaviour:

$ cat master.py
#!/usr/bin/env python
import os
for line in os.popen("./child.py"):
print "-->", line.strip()

$ cat child.py
#!/usr/bin/env python
import time

for i in range(5):
print i
time.sleep(.2)

On Linux you can work around it with pexpect:

$ cat master2.py
#!/usr/bin/env python
import pexpect
for line in pexpect.spawn("./child.py"):
print "-->", line.strip()

On Windows, I don't know.

Peter

 

[Piet van Oostrum]

n> I have tried both and Popen2.popen2().
n> os.popen runs both way, contrary to docs.

What do you mean `os.popen runs both way'?

n> # master.py
n> import os
n> # both lines work same

Of course, because 'r' is the default, and the bufsize for reading the
pipe only influences the performance.

n> #xx= os.popen("/mnt/mass/py/z6.py").readlines()
n> xx= os.popen("/mnt/mass/py/z6.py",'r',1).readlines()
n> # I had hoped small buffer would force a sync (flush())
n> # No such luck.

First, there is a difference between sync and flush. flush is writing
out any output that is buffered in the application program for that
file. Sync is writing out buffers (cache) in the operating system (to the
disc, network connection or similar). We are talking here about
flushing, not syncing. As we deal with a pipe, sync is not applicable.

Furthermore, whatever you do in the master program, such as setting the
buffer size has nothing to do with what happens in the child, so it will
not influence the flushing behaviour of the child. The only way to
influence that would be to use a pty instead of a pipe, because most
code uses a different buffering strategy for ttys and ptys compared to
files and pipes.

The next error in your program is that you use readlines()
This reads ALL input from your pipe before it proceeds. So essentially
it waits until the child process is finished and therefore gives you the
impression that it doesn't flush(). But it is not the child that is the
problem it is your master.

What you should do is take the pipe and do a readline()/print loop.

#xx= os.popen("/mnt/mass/py/z6.py")

# popen is deprecated. Replace with subprocess:
from subprocess import Popen, PIPE
xx = Popen(["/mnt/mass/py/z6.py"], stdout=PIPE).stdout

while True:
line = xx.readline()
if not line: break
print "\t" + line,

The obvious:

for l in xx:
print "\t" + l,

doesn't work as the iterator for a file, including pipes, does a
read ahead (see the doc on file.next()) and therefore is not suitable for
interactive use.

n> The direct question comes back to:

n> How does one force a sync or flush() to take effect in Python with
n> Tkinter in use? Or just in Python period. The keyword being force.

So it should be clear by now that the answer is to use flush().
And make sure you read AND process the flushed output in a proper way.

By the way, you have a nasty habit of using very unclear language to
pose your problems in this newsgroup/mailing list, instead of writing a
concise description and giving small examples of what code causes the
problem. This makes it extremely difficult to help you. I had to read
your messages at least 5 times to get an idea of what you were doing and
what the problem was that you wanted to solve.

Maybe you are dyslectic or something similar, in which case I apologize.
But anyway, I think you should learn to express yourself more clearly.
There are also courses for that.

 

[Piet van Oostrum]

n> I have tried both and Popen2.popen2().
n> os.popen runs both way, contrary to docs.

n> It reads from child while console writes directly to child - thus
n> eliminating the problem of coding a pass through from master.

Yes, but that is not `both way': popen connects the parent to the child
through a pipe. The pipe works one way: from the child to the parent
with 'r' (default), from the parent to the child with 'w'. You can't
communicate the other way through the pipe. So the communication from
the parent process to the child through the popen is ONE WAY. If you
want TWO WAY communication you can use popen2, or better use
subprocess.Popen.

A child process always inherits stdin, stdout and stderr from the parent
unless you change that (e.g. by redirecting to a pipe, like popen does
for one of them). It doesn't matter whether you use os.popen,
subprocess.Popen, os.system, or os.fork to create the child process. So
in your case if the parent inputs from the console, so does the child.
But note: this is not communication from the parent process to the
child, but from YOU to the child. So the parent-child communication is
ONE WAY.

n> "...
n> ..."

n> If I understand you the above can be stated as:

n> The above does not work as an iterator for any file type, including
n> pipes, but it does do read aheads .... and therefore is not suitable for
n> interactive use.

For files in general it is no problem because the contents of the file
is not interactively generated. Read ahead on a file works as long as
you do'nt use readline() on the file in between the iterator actions.
For a socket it could be the same problem if the other side generates
the output interactively.

n> If that is correct then read ahead is simply buffered file reads (grab a
n> chunk, parcel it out on demand) - yes?

I don't know, I have looked into the source code but it isn't clear to
me. I noticed that iteration didn't work and then looked up the
documentation. It talks about read ahead for efficiency.

n> As for "... not suitable for interactive ..." Really? Except for
n> special purpose use the current interactive components are all buffered
n> for read ahead use. Check the actual code for your keyboard, your mouse
n> and so forth. It's the read ahead that allows faster time to completion.
n> It's why C-code has the putch function.

Yes, but they only read what is available. The iterator apparently tries
to read more and then has to wait.

n> Yes - Sync IS the bigger hammer! If that is what is needed - so be it.
n> All character readers (byte at a time) should obey a flush(). Depending
n> on type, code for the reader controls whether or not it flushes
n> incomplete "lines" in the in-buffer(s). Proper implementation limits lost
n> data on system crash.

I don't understand what you say here. As I told before it has nothing to
do with sync(). Also for reading there is no flush(); the flush() is
done on the other side of the pipe. Yes, and sync() has to do with
system crashes but that is not what we are talking about in this thread.

n> In trying to use flush at the master side I keep getting messages
n> indicating strings (completed or not) are not flushable. Strange practice.

If you print to the console in the master the flush is done
automatically.

n> ---
n> from subprocess import Popen, PIPE
n> xx = Popen(["z6.py"], stdout=PIPE).stdout

n> while True:
n> line = xx.readline()
n> if not line: break
n> print "\t" + line,
n> ---
n> DOES WORK on Python 2.5.2 on Slackware 10.2 - THANK YOU VERY MUCH!!!
n> Isn't working on Windows. error message comes as one of two forms.
n> 1- %1 not found #as shown above
n> 2- file not found #as ...["python z6.py"]...
n> same #as #2 even with full paths given

That should be Popen(["python", "z6.py"], stdout=PIPE).stdout
And then with both python and z6.py given as full paths.

n> I get the impression subprocess ignores system things on Windows.
n> The routines it purposes to replace do use them. At any rate, subprocess
n> is NOT consistent across platforms.

subprocess is, but Windows isn't. On Unix-like systems, the python
command is usually in your PATH, so just giving "python" works. In
Windows PATH is underused, and commands like python often are not in
PATH, unless you as a user has adapted the path, or maybe there is an
installation option to adapt the PATH. The reason probably is that in
Windows almost nobody uses the command line but only clicks and the PATH
is not relevant.

n> Some questions:
n> 1) "...], stdout=PIPE).stdout
n> ^ ^ why the double use?

It is not a double use. Popen(["z6.py"], stdout=PIPE) gives you a Popen
object, not a file object. If you add .stdout you get the stdout
attribute of the Popen object of which you just before stated that it
should be a pipe. So the stdout=PIPE parameter makes it create a pipe,
and the .stdout returns you that pipe.

n> 2) "if not line: break" what tells what to look for EOL
n> or is the use of 'line' missleading?
n> is it a byte at a time output?

No, line is a line, as indicated by the readline() call. The
"if not line" is a test for end of file. In this case the end of the
child process (pipe closed).

n> how much CPU usage does the loop use?

Not much, It is mainly waiting for input but that doesn't consume CPU.

n> is there something in that loop that
n> uses the system pipe triggers to
n> reduce excessive CPU waste or does it
n> constantly poll? If so where does
n> what code get put to signal (or set)
n> the courtesy semiphores so it does
n> not hog the system?

It doesn't poll or use semaphores. The beauty of a pipe is that it
automatically synchronises the reader and writer processes. If you do a
read on a pipe and it is empty the OS just blocks you. When the other
end writes something in the pipe, the OS will unblock you.

 

[MRAB]

n> Some questions:
n> 1) "...], stdout=PIPE).stdout
n> ^ ^ why the double use?

It is not a double use. Popen(["z6.py"], stdout=PIPE) gives you a Popen
object, not a file object. If you add .stdout you get the stdout
attribute of the Popen object of which you just before stated that it
should be a pipe. So the stdout=PIPE parameter makes it create a pipe,
and the .stdout returns you that pipe.

I rather thought it might be something like the military:
Company - Dress Right - Dress
Get the attention, state what is to be done, order it done.

Python goes to great lengths to "be helpful" but drops the ball on the
obvious. stdout=PIPE means the user wants stdout piped back so it
should be helpful and do the obvious rather than have the user be
redundant.

What if the user requests both stdin and stdout? and what about all the
other useful bits which the returned object provides?

 

[norseman]

n> Some questions:
n> 1) "...], stdout=PIPE).stdout
n> ^ ^ why the double use?

It is not a double use. Popen(["z6.py"], stdout=PIPE) gives you a Popen
object, not a file object. If you add .stdout you get the stdout
attribute of the Popen object of which you just before stated that it
should be a pipe. So the stdout=PIPE parameter makes it create a pipe,
and the .stdout returns you that pipe.

I rather thought it might be something like the military:
Company - Dress Right - Dress
Get the attention, state what is to be done, order it done.

Python goes to great lengths to "be helpful" but drops the ball on the
obvious. stdout=PIPE means the user wants stdout piped back so it
should be helpful and do the obvious rather than have the user be
redundant.

What if the user requests both stdin and stdout? and what about all the
other useful bits which the returned object provides?

---------------------------
"... stdin and stdout?..."
I tried that - results were even worse. Have to kill the window to
get out. (both redirects are to master who is not running, until child
dies. meanwhile keyboard is effectively locked out)

"... other useful bits..."
Beauty is in the eye of the beholder. Or so the saying goes.
"Useful" to one may not be so to another. It is kinda like this:
I have been in some beautifully painted cars. Really great workmanship.
They had lots of 'extras' too. Soft leather seats, built in two way
radios and built in mobile phones and built in ice makers and the list
goes on. But in the USofA Western States that have lots of dirt and
gravel roads they are a complete waste of time, effort and money. Two
weeks and they look like any other beat up jalopy. The dust gets into
everything. Ever wanted to put ice cubes full of frozen dust and grit
into your favorite drink? The ice maker had been installed into a
friend's old pickup a week or so earlier. A month or so later I noticed
it had been removed. "Useful" is a vague measurement at best.


By the way - bits and pieces as told by yourself and Peter and Piet have
accomplished painting a useful picture for me.

My thanks to all three of you.


Steve