Executing multiple subprocesses and waiting

[JB]

One of my python scripts that takes a bunch of inputs from a tKinter
gui, generates a set of command line stings, and then threads them off
to subprocess for calls to other programs like Nuke and our render
farm has recently started randomly crashing pythonw.exe.

I'm taking a look at my threading setup and attempting to clean it up.
I was wondering what a smart way of doing what I describe is? Take a
list of strings containing command line calls to other programs,
process them one at a time (waiting for the previous one to complete
before starting the next) and then finishing elegantly. Currently, the
gui passes them all to a "workerThread" which loops through each
string, sending it to a "processThread" which makes a call to
subprocess to execute it.

This has worked fine for over a year so the recent crashing is
mystifying me. I'm wondering if it's increased network stress (we've
grown) or something similar?

Any thoughts and suggestions on waiting for threads to complete are
appreciated.

 

[Jean-Michel Pichavant]

One of my python scripts that takes a bunch of inputs from a tKinter
gui, generates a set of command line stings, and then threads them off
to subprocess for calls to other programs like Nuke and our render
farm has recently started randomly crashing pythonw.exe.

I'm taking a look at my threading setup and attempting to clean it up.
I was wondering what a smart way of doing what I describe is? Take a
list of strings containing command line calls to other programs,
process them one at a time (waiting for the previous one to complete
before starting the next) and then finishing elegantly. Currently, the
gui passes them all to a "workerThread" which loops through each
string, sending it to a "processThread" which makes a call to
subprocess to execute it.

This has worked fine for over a year so the recent crashing is
mystifying me. I'm wondering if it's increased network stress (we've
grown) or something similar?

Any thoughts and suggestions on waiting for threads to complete are
appreciated.

Google 'Python execnet'.

The main purpose of this module is to execute jobs on a bunch of
different machines. However by simply declaring your local as unique
available gateway, execnet will take care of queueing the jobs on your
local. Look for examples with 'Popen' type gateways.

I love this module, (and any other that are doing the network stuff for
you).

JM