options for running code in parallel

[Martin DeMello]

Our testing department wants a small tool that can make a number of
http calls simultaneously to the same server, the basic logic being

# -----------------------------------------------------------------
calls = ["call1"..."calln"].map {|x| "http://server/#{x}"}

calls.each do |x|
OnSignal(START) do
http_send call(x)
log_result
end
end

SendSignal(START)
# -----------------------------------------------------------------

What are the options for doing this cleanly and robustly? I haven't
been keeping up with the recent work people have been doing on
distributed/parallel code in ruby, but I get the general impression
there's been a lot of activity in this area.

martin

 

[Igal Koshevoy]

Our testing department wants a small tool that can make a number of
http calls simultaneously to the same server

Before writing your own, you may want to take a look at existing tools like:
* http://www.hpl.hp.com/research/linux/httperf/
* http://httpd.apache.org/docs/2.0/programs/ab.html

If you still want to build your own, please review the last couple days
of ruby-talk mailing list archives because there's been some spirited
debate about fork/thread that may be useful.

-igal

 

[ara.t.howard]

Our testing department wants a small tool that can make a number of
http calls simultaneously to the same server, the basic logic being

# -----------------------------------------------------------------
calls = ["call1"..."calln"].map {|x| "http://server/#{x}"}

calls.each do |x|
OnSignal(START) do
http_send call(x)
log_result
end
end

SendSignal(START)
# -----------------------------------------------------------------

What are the options for doing this cleanly and robustly? I haven't
been keeping up with the recent work people have been doing on
distributed/parallel code in ruby, but I get the general impression
there's been a lot of activity in this area.

martin

give this a whack:



cfp:~> cat a.rb
module Threadify
require 'thread'

@threads = 8
@abort_on_exception = true

class << self
attr_accessor :threads
attr_accessor :abort_on_exception
end
end

module Enumerable
def threadify opts = {}, &block
opts = {:threads => opts} if Numeric === opts
threads = Integer(opts[:threads] || opts['threads'] ||
Threadify.threads)
done = Object.new.freeze
jobs = Queue.new

each_with_index{|elem, i| jobs.push [elem, i]}
threads.times{ jobs.push done} # mark the end

consumers = Array.new threads

threads.times do |i|
consumers = Thread.new do
this = Thread.current
this.abort_on_exception = Threadify.abort_on_exception
loop{
job = jobs.pop
this.exit if job == done
args = job.first
jobs << (job << block.call(*args))
}
end
end

consumers.map{|t| t.join}
jobs.push done

ret = []
while((job = jobs.pop) != done)
elem, i, value = job
ret = value
end
ret
end
end

class Thread
def Thread.ify enumerable, *args, &block
enumerable.send :threadify, *args, &block
end
end


if __FILE__ == $0
require 'open-uri'
require 'yaml'

uris = %w( http://google.com http://yahoo.com http://rubyforge.org/ http://ruby-lang.org
)

Thread.ify uris, :threads => 3 do |uri|
body = open(uri){|pipe| pipe.read}
y uri => body.size
end
end



cfp:~> ruby a.rb
---
http://yahoo.com: 9562
---
http://google.com: 6290
---
http://rubyforge.org/: 22352
---
http://ruby-lang.org: 9984




soon to be released as a gem.


a @ http://codeforpeople.com/

 

[Martin DeMello]

give this a whack:

Thanks, will take it for a spin!

martin

 

[ara.t.howard]

Thanks, will take it for a spin!

just threw a gem up too - grab it from rubyforge by hand until it
propagates...

cheers.

a @ http://codeforpeople.com/