threads

[Brad Tilley]

I'm rewriting some old Python scripts in Ruby. Just to learn a bit more.
I want to thread this code to check websites concurrently. Could somone
offer advice? What are reasonable limits on threads in Ruby?

ips.each do |ip|
begin
Timeout::timeout(5) do
Net::HTTP.start(ip) do |site|
data = site.get( '/' ).body.downcase
if data.include?(minolta) and
data.include?(pagescope)
puts printer + "\t" + ip
elsif data.include?(xerox) and
data.include?(printer)
puts printer + "\t" + ip
else
puts "website\t" + ip
end
end
end
# Don't really care about exceptions just time outs mostly.
rescue Timeout::Error
puts "Timeout\t" + ip
rescue Exception
puts "General Exception\t" + ip
end
end

 

[Michael Fellinger]

I'm rewriting some old Python scripts in Ruby. Just to learn a bit more.
I want to thread this code to check websites concurrently. Could somone
offer advice? What are reasonable limits on threads in Ruby?

ips.each do |ip|
begin
Timeout::timeout(5) do
Net::HTTP.start(ip) do |site|
data = site.get( '/' ).body.downcase
if data.include?(minolta) and
data.include?(pagescope)
puts printer + "\t" + ip
elsif data.include?(xerox) and
data.include?(printer)
puts printer + "\t" + ip
else
puts "website\t" + ip
end
end
end
# Don't really care about exceptions just time outs mostly.
rescue Timeout::Error
puts "Timeout\t" + ip
rescue Exception
puts "General Exception\t" + ip
end
end

i made a small example here:
http://pastie.caboo.se/19495

^ manveru

 

[Brad Tilley]

i made a small example here:
http://pastie.caboo.se/19495

^ manveru

Thank you for the example!

 

[Brad Tilley]

What are reasonable limits on threads? Say I'm scanning a class b
network (roughly 65K hosts). How would you break up the threads? I seem
to get too many execution expired errors if I have more than 500 hosts
in one thread. It seems to work best with 256 groups of 256 hosts each
or 254 if you exclude the 0's and 255's

What are reasonable limits when working with threads in Ruby? Any tips?

 

[Michael Fellinger]

What are reasonable limits on threads? Say I'm scanning a class b
network (roughly 65K hosts). How would you break up the threads? I seem
to get too many execution expired errors if I have more than 500 hosts
in one thread. It seems to work best with 256 groups of 256 hosts each
or 254 if you exclude the 0's and 255's

What are reasonable limits when working with threads in Ruby? Any tips?

No tips there... it always depends on the task on hand - just use what works
for you

 

[Farrel Lifson]

No tips there... it always depends on the task on hand - just use what works
for you

You might try using a MapReduce/DRb based approach instead of threads.
Have a look at Starfish (http://rufy.com/starfish/doc/) for ideas.

Farrel

 

[ara.t.howard]

I'm rewriting some old Python scripts in Ruby. Just to learn a bit more.
I want to thread this code to check websites concurrently. Could somone
offer advice? What are reasonable limits on threads in Ruby?

Here's an EventMachine code sample that should do what you need. Notice,
this code is nonthreaded, but it still does all the HTTP GETs
simultaneously. Of course you'll want to do something more interesting in
the http.callback block.

#-------------------------------------

require 'rubygems'
require 'eventmachine'

$addrs = [
"www.apple.com",
"www.cisco.com",
"www.microsoft.com"
]

def scan_addr addr
http = EventMachine:rotocols::HttpClient.request(
:host => addr,
ort => 80,
:request => "/"
)

http.callback {|response|
puts response[:status]
puts response[:headers]
puts response[:content].length
}
end

EventMachine.run {
$addrs.each {|addr| scan_addr addr}
}

what would the preferrer way by to shared data from http.callback? does it
need protection? how about sharing with ruby green threads?

cheers.

-a

 

[ara.t.howard]

If you wanted for some reason to run this code simultaneously with unrelated
code on other threads, then of course you'd use the normal thread-safe
procedures to sync this data with your other threads.

k - that's the answer i was looking for.

cheers.

-a