freebsd and multiprocessing

[Tim Arnold]

Hi,
I'm intending to use multiprocessing on a freebsd machine (6.3
release, quad core, 8cpus, amd64). I see in the doc that on this
platform I can't use synchronize:

ImportError: This platform lacks a functioning sem_open
implementation, therefore, the required synchronization primitives
needed will not function, see issue 3770.

As far as I can tell, I have no need to synchronize the processes--I
have several processes run separately and I need to know when they're
all finished; there's no communication between them and each owns its
own log file for output.

Is anyone using multiprocessing on FreeBSD and run into any other
gotchas?
thanks,
--Tim Arnold

 

[Philip Semanchuk]

Hi,
I'm intending to use multiprocessing on a freebsd machine (6.3
release, quad core, 8cpus, amd64). I see in the doc that on this
platform I can't use synchronize:

ImportError: This platform lacks a functioning sem_open
implementation, therefore, the required synchronization primitives
needed will not function, see issue 3770.

As far as I can tell, I have no need to synchronize the processes--I
have several processes run separately and I need to know when they're
all finished; there's no communication between them and each owns its
own log file for output.

Is anyone using multiprocessing on FreeBSD and run into any other
gotchas?

Hi Tim,
I don't use multiprocessing but I've written two low-level IPC
packages, one for SysV IPC and the other for POSIX IPC.

I think that multiprocessing prefers POSIX IPC (which is where
sem_open() comes from). I don't know what it uses if that's not
available, but SysV IPC seems a likely alternative. I must emphasize,
however, that that's a guess on my part.

FreeBSD didn't have POSIX IPC support until 7.0, and that was sort of
broken until 7.2. As it happens, I was testing my POSIX IPC code
against 7.2 last night and it works just fine.

SysV IPC works under FreeBSD 6 (and perhaps earlier versions; 6 is the
oldest I've tested). ISTR that by default each message queue is
limited to 2048 bytes in total size. 'sysctl kern.ipc' can probably
tell you that and may even let you change it. Other than that I can't
think of any SysV limitations that might bite you.

HTH
Philip

 

[Tim Arnold]

Hi Tim,
I don't use multiprocessing but I've written two low-level IPC  
packages, one for SysV IPC and the other for POSIX IPC.

I think that multiprocessing prefers POSIX IPC (which is where  
sem_open() comes from). I don't know what it uses if that's not  
available, but SysV IPC seems a likely alternative. I must emphasize,  
however, that that's a guess on my part.

FreeBSD didn't have POSIX IPC support until 7.0, and that was sort of  
broken until 7.2. As it happens, I was testing my POSIX IPC code  
against 7.2 last night and it works just fine.

SysV IPC works under FreeBSD 6 (and perhaps earlier versions; 6 is the  
oldest I've tested). ISTR that by default each message queue is  
limited to 2048 bytes in total size. 'sysctl kern.ipc' can probably  
tell you that and may even let you change it. Other than that I can't  
think of any SysV limitations that might bite you.

HTH
Philip

Hi Philip,
Thanks for that information. I wish I could upgrade the machine to
7.2! alas, out of my power. I get the following results from sysctl:
% sysctl kern.ipc | grep msg
kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384

I'll write some test programs using multiprocessing and see how they
go before committing to rewrite my current code. I've also been
looking at 'parallel python' although it may have the same issues.
http://www.parallelpython.com/

thanks again,
--Tim

 

[Tim Arnold]

Hi Philip,
Thanks for that information. I wish I could upgrade the machine to
7.2! alas, out of my power.  I get the following results from sysctl:
% sysctl kern.ipc | grep msg
kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384

I'll write some test programs using multiprocessing and see how they
go before committing to rewrite my current code. I've also been
looking at 'parallel python' although it may have the same issues.http://www.parallelpython.com/

thanks again,
--Tim

Well that didn't work out well. I can't import either Queue or Pool
from multiprocessing, so I'm back to the drawing board. I'll see now
how parallel python does on freebsd.

--Tim

 

[Philip Semanchuk]

Well that didn't work out well. I can't import either Queue or Pool
from multiprocessing, so I'm back to the drawing board. I'll see now
how parallel python does on freebsd.

Sorry to hear that didn't work for you. Should you need to get down to
the nuts & bolts level, my module for SysV IPC is here:
http://semanchuk.com/philip/sysv_ipc/


Good luck with Parallel Python,
Philip

 

[Pop User]

I'll write some test programs using multiprocessing and see how they
go before committing to rewrite my current code. I've also been
looking at 'parallel python' although it may have the same issues.
http://www.parallelpython.com/

parallelpython works for me on FreeBSD 6.2.