start function in new process

[wongjoekmeu]

Hello all,

I would like to run a python function completely in a new process. For
example I have a parent process. That parent process needs to start a
child process in which the function is called. After the child process
is finished however I want that the child process should stop and then
only the parent process remain. When I start the function I want to
pass arguments also which include a function reference. I tried
os.fork() as I need to write it in linux. However it seems like the
child process remain alive. I discovered this when I loop through the
function. I see that the child process continue with the loop and
starts again a new grand child untill the loop is finished. See code
snippet. f() is the function I want to run in a new process. It got as
one of the input a refence to a function c().
Function g() I called in a loop for 3 times. Now it seems that when
child process get started in the function g() it return to the loop
and it calls g() itself and starts itself a nwe child process. I do
not want thant. I want the child process to perform the function f()
and just finished. Does anyone know how I can achieve this ?


def c():
print "function c"

def f(a,b, d):
# function I want to run in a new child process
print a, b
d()

def g():
pidID = os.fork()
if pidID == 0:
# child do something here
f(2,3,c)
else:
# parent do something here
print "Parent

for i in range(3):
g()

 

[Martin P. Hellwig]

On 03/05/10 19:21, (e-mail address removed) wrote:
<cut os.fork problem>
Any specific reason why threading.Thread or multiprocessing is not
suitable to solve your problem?

 

[wongjoekmeu]

On 03/05/10 19:21, (e-mail address removed) wrote:
<cut os.fork problem>
Any specific reason why threading.Thread or multiprocessing is not
suitable to solve your problem?

Because I got a memory leak in my function f(). It uses scipy, numpy,
pylab, and I am not planning to solve the memory leak because its too
complicated. So I thought of just calling the function then when it is
finished the process is gone and all memory is released. With
threading I don't think I would solve this problem. I am not sure
though.

 

[Martin P. Hellwig]

Because I got a memory leak in my function f(). It uses scipy, numpy,
pylab, and I am not planning to solve the memory leak because its too
complicated. So I thought of just calling the function then when it is
finished the process is gone and all memory is released. With
threading I don't think I would solve this problem. I am not sure
though.

I would be surprised if you can't do the same with
subprocess/multiprocessing, since you seem to know how to identify the
memory leak it shouldn't be a problem scripting out a test to see if it
works this way. I would be interested though in your findings.

 

[wongjoekmeu]

I would be surprised if you can't do the same with
subprocess/multiprocessing, since you seem to know how to identify the
memory leak it shouldn't be a problem scripting out a test to see if it
works this way. I would be interested though in your findings.

I can't use multiprocessing module since it comes only with python 2.6
and I am bound to python2.4. But subprocess does exist in python2.4,
but the question now is, how do I achieve that ? Any example ?

 

[Jonathan Gardner]

You're really close. See inline comment below.

def c():
   print "function c"

def f(a,b, d):
   # function I want to run in a new child process
   print a, b
   d()

def g():
   pidID = os.fork()
   if pidID == 0:
       # child do something here
       f(2,3,c)

You'll need to exit here -- not return.
http://docs.python.org/library/os.html#os._exit

 

[Martin P. Hellwig]

I can't use multiprocessing module since it comes only with python 2.6
and I am bound to python2.4. But subprocess does exist in python2.4,
but the question now is, how do I achieve that ? Any example ?

Sure, for example if I want to check the openssl version (didn't specify
I need to provide a useful example

I would normally do on the command line:
[martin@aspire8930 /usr/home/martin/Desktop]$ /usr/bin/openssl version
OpenSSL 0.9.8k 25 Mar 2009

The python subprocess equivalent is:
[martin@aspire8930 /usr/home/martin/Desktop]$ python
Python 2.6.4 (r264:75706, Jan 31 2010, 20:52:16)
[GCC 4.2.1 20070719 [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.

>>> import subprocess as _sp
>>> ssl_version = _sp.Popen(['/usr/bin/openssl', 'version'], stdout=_sp.PIPE)
>>> print(ssl_version.stdout.readlines()) ['OpenSSL 0.9.8k 25 Mar 2009\n']
>>> quit()

[martin@aspire8930 /usr/home/martin/Desktop]$

If you get any error in the Popen part, you probably did not use the
full path or an illegal parameter.

If you get a long wait in the readlines part, this usually means that
there is either nothing written to stdout or something is still being
written (I am not sure about this part though, it has been a while).

hth

 

[Robert Kern]

I can't use multiprocessing module since it comes only with python 2.6
and I am bound to python2.4.

It is available as a third party package for Python 2.4:

http://pypi.python.org/pypi/multiprocessing

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[wongjoekmeu]

I can't use multiprocessing module since it comes only with python 2.6
and I am bound to python2.4. But subprocess does exist in python2.4,
but the question now is, how do I achieve that ? Any example ?

Sure, for example if I want to check the openssl version (didn't specify
I need to provide a useful example

I would normally do on the command line:
[martin@aspire8930 /usr/home/martin/Desktop]$ /usr/bin/openssl version
OpenSSL 0.9.8k 25 Mar 2009

The python subprocess equivalent is:
[martin@aspire8930 /usr/home/martin/Desktop]$ python
Python 2.6.4 (r264:75706, Jan 31 2010, 20:52:16)
[GCC 4.2.1 20070719  [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
 >>> import subprocess as _sp
 >>> ssl_version = _sp.Popen(['/usr/bin/openssl', 'version'],
stdout=_sp.PIPE)
 >>> print(ssl_version.stdout.readlines())
['OpenSSL 0.9.8k 25 Mar 2009\n']
 >>> quit()
[martin@aspire8930 /usr/home/martin/Desktop]$

If you get any error in the Popen part, you probably did not use the
full path or an illegal parameter.

If you get a long wait in the readlines part, this usually means that
there is either nothing written to stdout or something is still being
written (I am not sure about this part though, it has been a while).

hth

Yes, I saw this example also before. HOwever what I want is to call an
internal function which gets a reference of another internal function
as input and not calling an external program. Do you have any example
on that with subprocess module ?
Thanks.

 

[Martin P. Hellwig]

On 03/06/10 09:45, (e-mail address removed) wrote:

Yes, I saw this example also before. HOwever what I want is to call an
internal function which gets a reference of another internal function
as input and not calling an external program. Do you have any example
on that with subprocess module ?
Thanks.

Well yes that is possible but you would more or less be just reinventing
parts of the multiprocessing module, so I would use that one (at the
location Robert Kern suggested).