Forking for performance?

[ed]

Ok, first off I should preface this by saying I've never forked
before.
Guess you could call me a virgin

Anyways, I'm used to writing scripts (mainly PHP) that _don't_ execute
for a very
long time, and don't really need to do anything with process creation
or management.

So forgive me if this is a stupid question.


Here's the scenario:

I want to run some code in a loop that runs through sub directories
of a bunch of different users and does stuff to their
files/directories.

Now normally I would traverse each user's directory one at a time.

But could I get the job done quicker if I ran multiple processes and
had
several users' directories being traversed at the same time?

I'd have to write a little extra code to "schedule" the processes to
make
sure that all the users' directories were being traversed an equal
amount
of times(because the loop will run until it's sent a signal, so these
directories get
traversed over and over again). But that probably won't be a big
deal.

Could I get the job done quicker by running multiple processes?

--ed

 

[Greg Bacon]

: [...]
:
: Could I get the job done quicker by running multiple processes?

The answer depends in part on the communication overhead you need.
You're diddling the filesystem, so waiting on the s-l-o-w I/O may
eat up any gains.

Maybe, in other words. Code it and compare the versions' execution
times.

Greg

 

[Uri Guttman]

e> Anyways, I'm used to writing scripts (mainly PHP) that _don't_
e> execute for a very long time, and don't really need to do anything
e> with process creation or management.


e> I'd have to write a little extra code to "schedule" the processes
e> to make sure that all the users' directories were being traversed
e> an equal amount of times(because the loop will run until it's sent
e> a signal, so these directories get traversed over and over again).
e> But that probably won't be a big deal.

and your perl question is? forking can be done in almost any language so
this is not a perl problem.

e> Could I get the job done quicker by running multiple processes?

it depends.

uri

 

[ed]

: [...]
:
: Could I get the job done quicker by running multiple processes?

The answer depends in part on the communication overhead you need.
You're diddling the filesystem, so waiting on the s-l-o-w I/O may
eat up any gains.

Maybe, in other words. Code it and compare the versions' execution
times.

Greg

Thanks guys. I guess I'll just have to try it out and see.
I forgot to mention that I'd also be running queries on a relational
db(MySql for starters), so I suppose that could also have an impact.

--ed

 

[ed]

[snip]

forking can be done in almost any language so
this is not a perl problem. [snip]

uri

Sorry bout that. Didn't really consider that before I posted here.

--ed

 

[James Willmore]

Thanks guys. I guess I'll just have to try it out and see.
I forgot to mention that I'd also be running queries on a relational
db(MySql for starters), so I suppose that could also have an impact.

In you're OP, you mentioned using PHP, yes? And are new to Perl, yes?
Well ... you're in for a treat!

Perl has the DBI module, that, you can code once, access many data
sources. In other words, no more coding for a specific data source
such as MySQL in one script and PostgreSQL in another. There are DBD
modules that you can use to access various data sources (aka drivers),
but you only have to write the code once - just change the connection
string to access a different data source. At least, PHP was set up to
use different calls to access different data sources when I was using
it heavily - maybe it's changed.

And to start out, 'perldoc' is the command to get perl documentation.
Start with 'perldoc perl' to get a list of all the various pieces of
documentation available.

I just figured I'd throw that in to help you out. The cost is ...
$0.00
You're money back if you're not completely satisfied

HTH

Jim

 

[ed]

In you're OP, you mentioned using PHP, yes? And are new to Perl, yes?
Well ... you're in for a treat!

Uhh.. Not really new to it, but I've really only written about a
thousand lines. So in terms of real experience I guess you could say
I'm still on the beginner side. But Perl's pretty easy to get going,
and keep on going with.

Perl has the DBI module,

Yep, that's what I'm using.

At least, PHP was set up to
use different calls to access different data sources when I was using
it heavily - maybe it's changed.

Yeah DBI does a good job. It was pretty easy to get started with too.

There's a lot of DB abstraction layers out there you can use for PHP
too. ADODB is one of the more popular ones.
I've been using this one(Not just for DB abstraction):
http://www.students.cs.uu.nl/people/voostind/eclipse/
because it's lightweight and the code is simple(it's hard to make it
that simple) and easy to extend.

Anyways, better cut this short since it's a bit off topic ..

--ed