Win32::Process - Limiting the number of processes launched from within a script

[Patrick Paquet]

Hello everyone,

I've been working on a script that fetches information from all
workstations on our NT domain, the problem I've now run into at this
point is that it takes a very long time to execute completely. The
reason for this is probably simply because I'm checking each machine
in sequence (I check machine A, and only when I'm done do I move on to
B etc).

I've been looking into running these checks in parallel, via the
Win32:rocess module, but I haven't found a way to control the
number of simultaneous processes adequately. I can launch a certain
number of them, but I can't figure out how to check the status of a
process and to lower a counter when it's done, so that I can launch
another one to replace it. What I'm looking to do is lauch no more
then 10 (for example) instances of the script, and to stay as close to
that without going over as long as there are machines to check.
Making sense?

I have been looking thru this group and various ressources on the net
(thank you Dave Roth!) on and off for well over 2 weeks now, but
haven't made much headway (although as usual I've picked up a couple
of things along the way).

I am open to alternatives as to how I can achieve this as well, I'm
not dead set on Win32:rocess, so if you have ideas or suggestions,
I'm interested. My current method generates several text files with
listings of machines to check, and launches the script via the system
command attaching the file as an argument, but I'm sure there's a more
elegant way to do this, and I'm also pretty sure that it's not as
difficult as it seems, but for now the solution escapes me.

My current script is fully functionnal as far as the workstation
checks are concerned. It is also a bit lengthy (+750 lines) so I
won't post it here for brevity's sake. At this point it doesn't even
include the Win32:rocess module as I just haven't been able to do
much with it at all, besides launching or (way too) many processes.

What I'm really asking for (after all this...) is some pointers and
simple examples (if possible) as to how I can check the status of the
processes and launch a new process on exit from the previous one.

I'm running ActiveState Perl 5.005_02, and cannot upgrade it. Many
scripts are run centrally and I don't have authority to do the upgrade
on the script server.

Thanks in advance for any assistance, suggestions, pearls of wisdow
and all the fish!


Patrick PAQUET

 

[Bill]

Patrick Paquet wrote:
....

I've been looking into running these checks in parallel, via the
Win32:rocess module, but I haven't found a way to control the
number of simultaneous processes adequately. I can launch a certain
....
My current script is fully functionnal as far as the workstation
checks are concerned. It is also a bit lengthy (+750 lines) so I
won't post it here for brevity's sake. At this point it doesn't even
include the Win32:rocess module as I just haven't been able to do
much with it at all, besides launching or (way too) many processes.


I'm running ActiveState Perl 5.005_02, and cannot upgrade it. Many
scripts are run centrally and I don't have authority to do the upgrade
on the script server.

If I were you, I'd get 5.8 on the server and use threads and queues.

If you really cannt do this, I suggest a more primitive solution--create
a perl script (script1) that will stuff server names or IP's into a
queue. Create a second script (script2) that will read the queue and
check the server listed in the queue and output the results to a file
(use flock). The scripts should be run from a cmd file like this:
=== cmd or .bat file ===
perl script1.pl
exit
=============

With a third script,
set up a perl script like this:
===
system("start script1.cmd");
for(1..10) { system("start script2.cmd"; sleep 1 }

===

....and let the OS do the process management for you. Perl 5.005 process
management is too broken for Win32 to bother agonizing over IMO. (It
works okay in 5.8 though).

 

[Chris]

Hello everyone,

I've been working on a script that fetches information from all
workstations on our NT domain, the problem I've now run into at this
point is that it takes a very long time to execute completely. The
reason for this is probably simply because I'm checking each machine
in sequence (I check machine A, and only when I'm done do I move on to
B etc).

I've been looking into running these checks in parallel, via the
Win32:rocess module, but I haven't found a way to control the
number of simultaneous processes adequately. I can launch a certain
number of them, but I can't figure out how to check the status of a
process and to lower a counter when it's done, so that I can launch
another one to replace it. What I'm looking to do is lauch no more
then 10 (for example) instances of the script, and to stay as close to
that without going over as long as there are machines to check.
Making sense?

[snipage]
I'm running ActiveState Perl 5.005_02, and cannot upgrade it. Many
scripts are run centrally and I don't have authority to do the upgrade
on the script server.

Ewww. I tried using Perl 5.005_03 once (from Hip Communications - the
predecessor name to AS) and it was awful. I'd say as someone else said
that your options are limited with this version of Perl - very limited.
Not even David Roth can help you with that and he's king of Perl Win32...

If it were me, I might try (roughly) writing a listener that I could put
on every end point (node) that listened on say port 9010 (or some other
port). Have a central listener on a central machine on port 9011 (or
some other port) to handle replies to requests sent to the listener port
of the end nodes. Have the central box run a request out to all the (or
a batch of) listeners on port 9010 and wait for the replies to come back
in on port 9011. You can batch your requests based on the capacity of
the central listener (on port 9011) to handle replies. When the central
node receives all the replies (or gives up trying/waiting due to end
node down, etc), then you are done.

The simplest way to write a listener in Perl on Windows (with the
version you have) IMO is to combine your listener code with SRVANY.EXE
(available on Windows RK CD) and create Windows service daemons.

Chris

 

[Patrick Paquet]

(e-mail address removed) (Patrick Paquet) wrote in message <Original posting snipped>

Hello everyone,

I'm replying to my original post to clarify a bit the problem I was
having, and also to post the solution that I am using, in the hopes
that this will be helpful to someone else.

Apart from the replies I've gotten in the newsgroup, I've also gotten
2 suggestions directly to my e-mail. I've used one of those to fix my
problem, I will post it at the end.

First, some clarifications. The script is launched from a central
server, and polls all the workstations, so nothing is actually sent
from the PCs themselves, everything is fetched from the server. Most
of the traffic is netbios stuff, read from registry or from the HD,
nothing very fancy involved. Once the information is fetched, it's
pumped in an SQL database. After that, various tools are used to
generate batch jobs, or just reporting purposes.

The code below was provided by Brian Helterline, and works perfectly
for me once integrated and modified for my script. I launch 20 (still
scaling at this point though) processes with it, when one exits,
another is launched, it's exactly what I was looking for. I'm kinda
new at this Perl thing and do not have any formal programming
training, I just don't think of using some commands (never occured to
me to use 'while' like this, or 'shift' even), so this is all very
educationnal.

A big thank you to all who took the time to reply, it is very much
appreciated!

Hope everybody has a nice day!


Patrick


######

use strict;
use warnings;

use Win32:rocess;

my @machines = qw/ 1 2 3 4 5 6 7 8 9 10 11 12/; # list of machines to
test
my %running; # hash of processes running
my $done = 0;
while ( ! $done ) {
if ( keys %running < 10 && @machines ) {
my $obj;
Win32:rocess::Create($obj,'C:\winnt\system32\notepad.exe',
"notepad c:\\" . shift( @machines ) .
'.txt',
0,
NORMAL_PRIORITY_CLASS,
".") || die Win32::GetLastError();
$running{$obj} = $obj;
print "started object $obj\n";

}
foreach my $obj ( keys %running ) {
my $code;
my $rc = $running{$obj}->GetExitCode( $code );
delete $running{$obj} unless $code == 259;
print "removed object $obj\n" unless $code == 259;
}
$done = not %running;
}
print "all done.\n";