A threaded 'find' library?

[Daniel Berger]

Hi all,

The recent talk of threaded access to a file made me wonder if I could
do the same with a file searching lib (file-find in my case). This is
the basic logic:

paths.each{ |path|
Dir.foreach(path){ |file|
paths << file if file is a directory (and not already searched)
next unless file matches user-supplied pattern
}
}

I thought I setting up a thread queue and spawning a new thread for each
new directory would speed it up, but my (probably poor) attempts to
implement it did not result in any speed improvements, and usually
slowed it down a little bit.

Any suggestions?

Thanks,

Dan

 

[Michal Suchanek]

Hi all,

The recent talk of threaded access to a file made me wonder if I could
do the same with a file searching lib (file-find in my case). This is
the basic logic:

paths.each{ |path|
Dir.foreach(path){ |file|
paths << file if file is a directory (and not already searched)
next unless file matches user-supplied pattern
}
}

I thought I setting up a thread queue and spawning a new thread for each
new directory would speed it up, but my (probably poor) attempts to
implement it did not result in any speed improvements, and usually
slowed it down a little bit.

That's because ruby does not really support concurrent threads. Any
overhead you add by threading only slows things down. And after you
run the program for the first time you probably have most of the
directories cached so you cannot cover any waiting for io with threads
either.

To fake the delay you could put a sleep in front of calling
Dir.foreach. This is where the directory would be likely opened and
the filesystem might need to fetch a block or two.

You can also perform a test with a floppy, just unmount, eject, and
remount it each time to make sure the cache is invalidated

Thanks

Michal