Web requests should be lazy, putting off any work they can. When a user updates a script on userscripts.org I send an email alert sent to its fans. But the uploader shouldn't have to wait while hundreds of people are emailed. There are many systems that allow for asynchronous tasks in rails. Here is what worked best for me.
No spam please
Since I don't want to be a spammer, I need to allow a method for users to opt-in to receiving updates of their favorite scripts. I've added a column to our user table to store whether a user wants emailed.
add_column :users, :email_favorite_activity, :boolean
UI changes are made to the settings page allowing them to subscribe to alerts. Next I make sure emails include messaging about why they were sent and how to stop getting them.
A small job
Next I create a directory called jobs in the userscripts project directory. I add a trival ruby script emails all the fans that opted into email of the passed in script. This code runs within the rails environment allowing the use of ActiveRecord, ActionMailer and all their friends.
script_id = ARGV[0]
script = Script.find(script_id)
users = script.fans.select { |u| u.email_favorite_activity }
puts "Emailing #{users.count} fans"
users.each do |fan|
Notification.deliver_new_script_version(script, fan)
end
The job can be ran manually using rails' built-in script/runner.
script/runner ./jobs/email_script_fans.rb 42
In development mode the emails won't be sent, but I can see I'm ready to integrate the task.
Why starling backgroundrb bj?
Starling is a queue system which speaks the memcache protocol. It was designed by Blaine Cook to allow Twitter to route with a massive of tweet. Starling requires a server daemon, and workers that are constantly asking for queued items. My email load is a fraction of a fraction of twitter's load, so the increased complexity isn't worth it.
BackgroundRB was created by Ezra Zygmuntowicz (of EngineYard). BackgroundRB requires a daemon as well, and if that daemon isn't running adding a task fails. After spending a few hours I could see the potential but the complexity was overkill for sending a few emails.
BackgroundJob (BJ) was a perfect fit. Jobs are stored in your database, so you don't have to worry about deploying and monitoring another daemon. I was initially concerned that tasks are not rails specific. Task are ran as regular processes, with the stdout/stderr being stored in columns after completion. To run rails code you use script/runner. Each job requires a new script/runner, so the overhead of loading rails for each task can become painful at volume.
BJ was received support from EngineYard/Ezra and the community has taken over BackgroundRB. There are other projects as well, but none of them seemed to:
- put the jobs in the database
- not require a daemon to be running
- not make the workers more complicated than the tasks
- really easy deploy
Installation
Start by install BackgroundJob as a plugin.
script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj script/bj setup
Unfortunately BackgroundJob has a couple issues with the timezone improvements in rails 2.1. The fix is to change all instances of Time.now to Time.now.utc in two of the files.
vendor/plugins/bj/lib/bj/runner.rb vendor/plugins/bj/lib/bj/table.rb
Then I trigger a job to be added when a new version of a script is uploaded.
class Version < ActiveRecord::Base
after_create :email_fans
def email_fans
Bj.submit "./script/runner ./jobs/email_script_fans.rb #{self.script_id}"
end
end
Server Setup
BackgroundJob upon receiving a new job will launch a worker if there isn't one running already, so after a cap deploy, emails will be sent.
Harder, Better, Faster, Stronger
Harder: You need to setup bounce detection. Sending an emails out without checking for bounces is a recipe for being blocked as your volume increases. If an email bounces twice in a month disable updates and ask them to update their email the next time they visit.
Better: Add the ability for users to select HTML or text based emails.
Faster: You can speed up the process by making the email body and subject is the same for all fans, then you should use BCC to send a single email to all recipients at once.
Stronger: Generate a daily email to yourself that tells you how many emails were sent. Have a cron entry generates this daily job.
