Bulk push of jobs

[danwa5]

I like to make a feature request for add support for pushing a large batch of jobs to the Faktory server, something similar to what Sidekiq offers with #push_bulk.

The network latency of enqueueing a job individually is starting to add up when dealing with a large data set.

Sample code

markers = ids.each_slice(10).to_a

markers.each do |m|
  SomeWorker.perform_async(m[0], m[-1])
end

Is there a workaround for something like this for the time being?

[mperham]

The only workaround today is to use a pool of connections and threads. Faktory does not offer a PUSH operation which takes multiple payloads. Here's sample code using 10 threads to push in parallel:

require 'faktory'
require 'thread'
require 'securerandom'

ids = (1..1000).to_a
q = Queue.new 
ids.each_slice(10).to_a.each {|m| q.push([m[0], m[-1]]) }
q.close

threads = []
10.times do |idx|
  job = { "jobtype": "SomeWorker", "queue": "default"  }
  threads << Thread.new do
    c = Faktory::Client.new(debug: true)
    while args = q.pop
      c.push(job.merge("args": q.pop, "jid": SecureRandom.hex(8)))
    end
    c.close
  end
end

threads.each(&:join)