calling python scripts as a sub-process

[Catherine Moroney]

I have one script (Match1) that calls a Fortran executable as a
sub-process, and I want to write another script (Match4) that
spawns off several instances of Match1 in parallel and then waits
until they all finish running. The only way I can think of doing this
is to call it as a sub-process, rather than directly.

I'm able to get Match1 working correctly in isolation, using the
subprocess.Popen command, but calling an instance of Match1 as a
subprocess spawned from Match4 isn't working.

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]

and I'm calling it as:

sub1 = subprocess.Popen(command)

I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Thanks for any help,

Catherine

 

[Philip Semanchuk]

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ', '--
file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf
', '--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--
chmetric='M2' ", "--use_textid='true '"]

[snip]


I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in
_execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ', '--
file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ', '--
file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ', '--
block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]

 

[Mike Driscoll]

I have one script (Match1) that calls a Fortran executable as a
sub-process, and I want to write another script (Match4) that
spawns off several instances of Match1 in parallel and then waits
until they all finish running.  The only way I can think of doing this
is to call it as a sub-process, rather than directly.

I'm able to get Match1 working correctly in isolation, using the
subprocess.Popen command, but calling an instance of Match1 as a
subprocess spawned from Match4 isn't working.

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]

and I'm calling it as:

sub1 = subprocess.Popen(command)

I get the error below.  Does anybody know what this error refers
to and what I'm doing wrong?  Is it even allowable to call another
script as a sub-process rather than calling it directly?

  File "../src_python/Match4.py", line 24, in RunMatch4
     sub1 = subprocess.Popen(command1)
   File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
     errread, errwrite)
   File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
     raise child_exception
OSError: [Errno 2] No such file or directory

Thanks for any help,

Catherine

Try giving an absolute path to the python file rather than a relative
path. I don't think the subprocess module "knows" where to look
otherwise.

Mike

 

[Dan Upton]

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
"--use_textid='true '"]

[snip]


I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
"--use_textid='true '"]

I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)

 

[Catherine Moroney]

I just tried that, and I get the same error.

Interestingly enough, a shorter (and incorrect) version
of the command works well enough so that it gets into the
Match1 code and does the argument check there.

The following code gets into Match1:

>>> command = ['python', '../src_python/Match1.py','--filex="xyz"']
>>> sub1 = subprocess.Popen(command)

whereas this doesn't even get to call Match1:

command =
['python','/data/svn_workspace/cmm/sieglind/USC/EE569/tpaper/test/../src_python/Match1.py
', '--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', '--chmetric=M2 ',
'--use_textid=true']

sub1 = subprocess.Popen(command)

Can anybody see a reason for why the abbreviated version works, and
the full-up one doesn't?

Catherine

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ',
"--chmetric='M2' ", "--use_textid='true '"]

[snip]


I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]

 

[Catherine Moroney]

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
"--use_textid='true '"]
[snip]

I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
"--use_textid='true '"]

I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)

I added the shell=True and this time it got into Match1 (hurrah!),
but it then opened up an interactive python session, and didn't
complete until I manually typed 'exit' in the interactive session.

Match1 looks like:

if __name__ == "__main__":
<<< parse arguments >>>

RunMatch1(file_ref, file_cmp, iblock_start, iblock_end, \
nlinep, nsmpp, mindispx, maxdispx, mindispl, \
maxdispl, istep, chmetric, use_textid)

exit()

where the routine RunMatch1 does all the actual processing.

How do I get Match1 to run and exit normally without opening up an
interactive session, when called as a subprocess from Match4?

Catherine

 

[Dan Upton]

On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
",
"--use_textid='true '"]

[snip]

I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
",
"--use_textid='true '"]

I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)

I added the shell=True and this time it got into Match1 (hurrah!),
but it then opened up an interactive python session, and didn't
complete until I manually typed 'exit' in the interactive session.

Match1 looks like:

if __name__ == "__main__":
<<< parse arguments >>>

RunMatch1(file_ref, file_cmp, iblock_start, iblock_end, \
nlinep, nsmpp, mindispx, maxdispx, mindispl, \
maxdispl, istep, chmetric, use_textid)

exit()

where the routine RunMatch1 does all the actual processing.

How do I get Match1 to run and exit normally without opening up an
interactive session, when called as a subprocess from Match4?

Alternately, rather than using a list of arguments, have you tried
just using a string? (Again, that's the way I do it and I haven't
been having any problems recently, although I'm running shell scripts
or binaries with arguments rather than trying to invoke python on a
script.)

command = "python ../src_python/Match1.py
--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf
--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf
--block_start=62 --block_end=62 --istep=16 --chmetric='M2'
--use_textid=true"

proc = subprocess.Popen(command, shell=True)

 

[Jeremy Sanders]

I have one script (Match1) that calls a Fortran executable as a
sub-process, and I want to write another script (Match4) that
spawns off several instances of Match1 in parallel and then waits
until they all finish running. The only way I can think of doing this
is to call it as a sub-process, rather than directly.

I'm able to get Match1 working correctly in isolation, using the
subprocess.Popen command, but calling an instance of Match1 as a
subprocess spawned from Match4 isn't working.

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]

If you want to avoid going by the shell, and you *should* for security
reasons, you need to have each of your arguments separately in the list
without the shell quoting and extra spaces, i.e.

['python', '../src_python/Match1.py',
'--file_ref=xxxx.hdf', '--file_cmp=yyyy.hdf',
'--block_start=xx', '--block_end=62', '--istep=16', '--chmetric=M2',
'--use_texid=true']

Jeremy

 

[Catherine Moroney]

On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
",
"--use_textid='true '"]

[snip]

I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
",
"--use_textid='true '"]
I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)

I added the shell=True and this time it got into Match1 (hurrah!),
but it then opened up an interactive python session, and didn't
complete until I manually typed 'exit' in the interactive session.

Match1 looks like:

if __name__ == "__main__":
<<< parse arguments >>>

RunMatch1(file_ref, file_cmp, iblock_start, iblock_end, \
nlinep, nsmpp, mindispx, maxdispx, mindispl, \
maxdispl, istep, chmetric, use_textid)

exit()

where the routine RunMatch1 does all the actual processing.

How do I get Match1 to run and exit normally without opening up an
interactive session, when called as a subprocess from Match4?

Alternately, rather than using a list of arguments, have you tried
just using a string? (Again, that's the way I do it and I haven't
been having any problems recently, although I'm running shell scripts
or binaries with arguments rather than trying to invoke python on a
script.)

command = "python ../src_python/Match1.py
--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf
--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf
--block_start=62 --block_end=62 --istep=16 --chmetric='M2'
--use_textid=true"

proc = subprocess.Popen(command, shell=True)

Thanks - that did the trick. I just passed in one long string
and everything actually works. Wow! I had no idea if this was
even do-able.

This is so cool, and saves me a lot of code duplication. I can
spawn off half a dozen jobs at once and then just wait for them
to finish. It's great that python can function both as a
scripting language and also a full-blown programming language
at the same time.

Catherine

 

[Jeremy Sanders]

I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)

That's really papering over the bug. You need to have the parameters
separately, including the name of the program, separately in the list. You
need to remove any shell quoting you may use on the unix/dos command line.

Jeremy