Sidekiq and Background Jobs for Beginners

rails s
redis-server
sidekiq
mailcatcher
Redis isn’t doing anything

Convert a non-background-job to a background job

The essence of a background job is to do stuff in the background, without making the Rails app sit around doing all the work.

  1. create the job. (you can use rails generate job <job_name>, per the ActiveJob docs)
  2. Call the notify user job from the controller, instead of calling the user notifier directly.

Make the job

We’ll hand-roll this. Make app/jobs/send_user_gif_job.rb.

class SendUserGifJob < ActiveJob::Base
queue_as :default

def perform(*args)
# do da ting
end

end

Make a test

Working through the rubyonrails.org docs on testing jobs, I’ll set up the following:

# test/jobs/send_user_gif_job_test.rb

require 'test_helper'

class SendUserGifJobTest < ActiveJob::TestCase
test 'that email is sent' do
SendUserGifJob.perform_async("test@test.com", "hello")
# literally no idea what to assert here...
# assert
end

end

Messed up Sidekiq?

After a bit of playing in the rails console, I had a bunch of bad jobs that Sidekiq was trying to process. Every time I started Sidekiq, it broke with a stack trace for “uninitialized constant”, for a job/class/worker that didn’t exist.

Sidekiq::Queue.all.each(&:clear)
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
Sidekiq::DeadSet.new.clear

Reworked the test and worker

These are workers and not jobs. ActiveJob jobs live in /jobs. So, if you want a worker, don’t put it in the /jobs directory, put it in the /workers directory.

# app/workers/send_gif_to_user_worker.rb

class SendGifToUserWorker
include Sidekiq::Worker

def perform(*args)
# Do something
end
end
# test/workers/send_gif_to_user_worker_test.rb

require 'test_helper'

class SendGifToUserWorkerTest < ActiveJob::TestCase
test 'that email is sent' do
SendGifToUserWorker.perform_async("test@test.com", "hello")
# literally no idea what to assert here...
# assert
end

test 'that job is pushed to queue' do
assert_equal 0, SendGifToUserWorker.jobs.size
SendGifToUserWorker.perform_async("test@test.com", "hello")
assert_equal 1, SendGifToUserWorker.jobs.size
end
end
# test/workers/send_gif_to_user_worker_test.rb:5

def setup
Sidekiq::Worker.clear_all
end

Making Sidekiq do stuff via the Rails Console

Since the tests don’t push actual jobs to Sidekiq, I don’t see any indication that anything interesting is happening in Sidekiq web, or Redis, or the Sidekiq terminal window. :(

# app/workers/send_gif_to_user_worker.rb
class SendGifToUserWorker
include Sidekiq::Worker

def perform(email, thought)
UserNotifier.send_randomness_email(email, thought).deliver_now
end
end
# app/controllers/mailers_controller.rb
class MailersController < ApplicationController

def create
SendGifToUserWorker.perform_async(params[:mailers][:email], params[:mailers][:thought])
flash[:message] = "You did it! Email sent to #{params[:mailers][:email]}"
redirect_to "/sent"
end

def sent
end
end
main:0> SendGifToUserWorker.perform_async("test@test.com", "hello")
=> "08e6a309cf7c46dc0178c53f"
main:0> SendGifToUserWorker.perform_async("test@test.com", "hello")
=> "8b962d28217ae177564f0fd7"
2018-07-27T17:13:55.023Z 10221 TID-ovusw76r0 SendGifToUserWorker JID-08e6a309cf7c46dc0178c53f INFO: start
2018-07-27T17:13:55.023Z 10221 TID-ovusw76r0 SendGifToUserWorker JID-08e6a309cf7c46dc0178c53f INFO: done: 0.0 sec
2018-07-27T17:13:57.521Z 10221 TID-ovusw781o SendGifToUserWorker JID-8b962d28217ae177564f0fd7 INFO: start
2018-07-27T17:13:57.521Z 10221 TID-ovusw781o SendGifToUserWorker JID-8b962d28217ae177564f0fd7 INFO: done: 0.0 sec
Calling Sidekiq jobs from Rails Console

Restart Sidekiq when you make a change to a worker

It makes sense that the Sidekiq worker test might assert JUST that jobs get queued correctly.

Watching Redis

I want to make sure that this stuff is getting in and out of Redis as expected. Redis is a super fast key:value store, and we should see stuff getting written to, and read from Redis.

Redis-server not showing much
redis-cli showing LOTS
Redis-cli showing useful results

lpush

1532784329.095661 [0 127.0.0.1:53832] lpush queue:default 
{
class:SendGifToUserWorker,
args:[flip,flop],
retry:true,
queue:default,
jid:adfa15f29ed6c09cda7f6973,
created_at:1532784329.095427,
enqueued_at:1532784329.095466
}

hset

1532784332.778327 [0 127.0.0.1:53803] hset MacBook-Pro-6715.local:32134:cc8d1568c5c6:workers ow3kb5tjc 
{
queue:default,
payload:
{
class:SendGifToUserWorker,
args:[flip,flop],
retry:true,
queue:default,
jid:adfa15f29ed6c09cda7f6973,
created_at:1532784329.095427,
enqueued_at:1532784329.095466
},
run_at:1532784329
}

What does/does not occur

So, how can we be sure that our Sidekiq job is actually firing? Lets see what it looks like, using this worker with Sidekiq, and without.

SendGifToUserWorker.new.perform(params[:mailers][:email], params[:mailers][:thought])
SendGifToUserWorker.perform_async(params[:mailers][:email], params[:mailers][:thought])
On the left, you can see the POST request come in, redis (top right) and sidekiq (bottom right) responding to said POST request
Same Post request, but the request never gets to Sidekiq or Redis, because I used Worker.new.perform

Conclusion

One of the bugs I was working with, the root of the problem was the worker/job seemed to not be requeuing when it failed. I spent time touching up the worker, and logging around the specific error (it was a timeout on an internal API endpoint), but even after rescuing the timeout, the job wasn’t happening again.

Resources

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store