Celluloid & performant HTTP requests - http

Question

I try to switch an existing crawler from EventMachine to Celluloid. To get in touch with Celluloid I've generated a bunch of static files with 150 kB per file on a linux box which are served via Nginx.
The code at the bottom should do its work, but there is a issues with the code which I don't understand: the code should spawn maximum 50 threads because of the thread pool size of 50 but it spawns 180 of them. If I increase the pool size to 100, 330 threads are spawned. What's going wrong there?
A simple copy & paste of this code should work on every box, so any hints are welcome :)
#!/usr/bin/env jruby
require 'celluloid'
require 'open-uri'
URLS = *(1..1000)
##requests = 0
##responses = 0
##total_size = 0
class Crawler
include Celluloid
def fetch(id)
uri = URI("http://data.asconix.com/#{id}")
puts "Request ##{##requests += 1} -> #{uri}"
begin
req = open(uri).read
rescue Exception => e
puts e
end
end
end
URLS.each_slice(50).map do |idset|
pool = Crawler.pool(size: 50)
crawlers = idset.to_a.map do |id|
begin
pool.future(:fetch, id)
rescue Celluloid::DeadActorError, Celluloid::MailboxError
end
end
crawlers.compact.each do |resp|
$stdout.print "Response ##{##responses += 1} -> "
if resp.value.size == 150000
$stdout.print "OK\n"
##total_size += resp.value.size
else
$stdout.print "ERROR\n"
end
end
pool.terminate
puts "Actors left: #{Celluloid::Actor.all.to_set.length} -- Alive: #{Celluloid::Actor.all.to_set.select(&:alive?).length}"
end
$stdout.print "Requests total: #{##requests}\n"
$stdout.print "Responses total: #{##responses}\n"
$stdout.print "Size total: #{##total_size} bytes\n"
By the way, the same issue occurs when I define the pool outside the each_slice loop:
....
#pool = Crawler.pool(size: 50)
URLS.each_slice(50).map do |idset|
crawlers = idset.to_a.map do |id|
begin
#pool.future(:fetch, id)
rescue Celluloid::DeadActorError, Celluloid::MailboxError
end
end
crawlers.compact.each do |resp|
$stdout.print "Response ##{##responses += 1} -> "
if resp.value.size == 150000
$stdout.print "OK\n"
##total_size += resp.value.size
else
$stdout.print "ERROR\n"
end
end
puts "Actors left: #{Celluloid::Actor.all.to_set.length} -- Alive: #{Celluloid::Actor.all.to_set.select(&:alive?).length}"
end

Answer 1

What ruby are you using? jRuby, Rubinius, etc? And what version of those?
Reason I ask is, threads are handled differently for each ruby. What you seem to be describing is added threads that come in for supervisors, and for tasks. Looking at the date of your post, it is likely that fibers are actually becoming native threads, which would make it seem like using jRuby perhaps. Also, using Futures often invokes the internal thread pool, which has nothing to do with your pool.
With both those reasons and others like them you could look for, that makes sense why you'd have a higher thread count than your pool calls for. This is a bit old though, so maybe you could follow up as to whether you still have this problem, and post output.