Pep 342 (val = yield MyGenerator(foo)), synchronous os.system() thatdoesn't block gui event loops

[Ville Vainio]

Has anyone implementing something like what the subject line
indicates?

The idea:

To run functions that execute a series of system commands without
blocking the ui, *and* without adding state machine logic.

The syntax would be something like:

def work():

showstatus("building")
r = yield runshell("make")
showstatus("installing")
r = yield runshell("make install")
showstatus("Success")

mygui.startwork(work)
# returns immediately, runs work() gradually in the background.


The catch is that showstatus() would need to be run in the mainloop,
so running the whole thing in a thread is a no-go.

I imagine runshell() would be implemented in terms of QProcess, or
subprocess.Popen/os.system and a worker thread.

Anyone done this already, or do I have to roll my own?

 

[Ville M. Vainio]

Has anyone implementing something like what the subject line

ImplentED.

I don't think this is that hard to do in the first place, but a
"generic" solution that can be easily tuned for different gui
mainloops would be nice.

 

[Ville M. Vainio]

I imagine runshell() would be implemented in terms of QProcess, or
subprocess.Popen/os.system and a worker thread.

Actually, the problem is that of general serialization of worker
thread operations. That is, it could be something akin to:

res = yield run_in_thread(lambda : os.system('make'))

run_in_thread would basically grab a worked thread, execute the
callable, and allow the generator to proceed after the thread is done.
Now, the only thing to do is the "generator dispatcher" in gui
mainloop that finds the correct generator and makes it proceed...

 

[John Nagle]

Has anyone implementing something like what the subject line
indicates?

The idea:

To run functions that execute a series of system commands without
blocking the ui, *and* without adding state machine logic.

At some level, there's going to be state machine logic.
You need interlocking so that the user can't initiate
simultaneous conflicting background operations from the GUI.

The usual solution is to run the background operations
in another thread, and have them put events on a queue
read by the GUI task.

You also have to work out how to cancel background operations,
if they're going to be as big as a "make".

Here's the very first application which did this sort of thing,
running commands in multiple threads without blocking the user
interface thread. This is from 1972.

http://www.fourmilab.ch/documents/univac/fang/

And yes, it did properly interlock everything it was
doing in the background.

John Nagle

 

[Terry Reedy]

Your subject line is so long that it is cut off even on my wide screen.
Better to repeat the question in the body.

 

[Ville M. Vainio]

You might want to look at twisted, in particular

 http://twistedmatrix.com/trac/wiki/DeferredGenerator

Indeed, DeferredGenerator looks pretty much what I'm interested in,
thanks for the pointer. Unfortunately, it has the big Twisted
dependency. That's probably not a showstopper, but Twisted requires a
degree of buy-in when something smaller would do.

 

[Ville M. Vainio]

Apologies for the long subject line, here it is again:

"Pep 342 (val = yield MyGenerator(foo)), synchronous os.system() that
doesn't block gui event loops"

    At some level, there's going to be state machine logic.
You need interlocking so that the user can't initiate
simultaneous conflicting background operations from the GUI.

This belongs to the "I don't care" category, as you'd expect to have
only one operation like this on the flight at one time. The idea is to
allow making simple scripts that you execute from the gui, that *may*
need to access the gui objects at one point or another (so execution
has to happen in the main thread most of the time - apart from
blocking, long operations).

    The usual solution is to run the background operations
in another thread, and have them put events on a queue
read by the GUI task.

Yeah, and the receiving end of the "queue" here would be the same
block of statements that launched the request - that is, I want to
avoid having to create handler functions, and keep the program from
linear.

    You also have to work out how to cancel background operations,
if they're going to be as big as a "make".

I think that is easy to accomplish with generators, provided that you
can let the long-lasting operation run to completion - just close()
the generator. If the long lasting computation was in a subprocess,
rather than a thread, it could be interrupted in a cleaner fashion
even.