overstimulate

Your website is being hammered because Hammer told his 120,000 followers on twitter to check your site. The content is highly cachable and no one would notice if you sent the same data for up to a minute. To make matters worse the request takes 5 seconds to complete. Hammer is threatening to send his posse after you. What do you do?

To simulate this theoretical event we will use a small sintra app and apache bench.

require 'rubygems'
require 'sinatra'

get '/work' do
  sleep 5
  body "result: #{rand}"
end

Requests take 5 seconds to process, in addition to any time spent waiting in the request queue. Our single app server processes requests one at a time, so if requests are more often than every 5 seconds, requests will certainly be delayed.

We could add more app servers to improve throughput, but each request would still take 5 seconds, and our concurrency is limited by the number of app servers. Most requests are doing work that isn't needed since the results computed by the last request are good enough.

Getting Varnish

Varnish is a state-of-the-art, high-performance HTTP accelerator, meaning it is built for caching. While your framework might have internal support for memcache or file system based caching, it is worth exploring how Varnish could integrate with your systems. Varnish is a layer between your HTTP frontend server (nginx) and your app servers. Besides caching it constantly checks the health of backend services, and only sends requests to those which are live.

Varnish is currently 2.0.3 at the time of writing. To install on linux:

wget http://downloads.sourceforge.net/varnish/varnish-2.0.3.tar.gz?use_mirror=internap
tar -zxvf varnish-2.0.3.tar.gz
cd varnish-2.0.3
./configure
make
sudo make install

Connecting Varnish to Sinatra

Sinatra is listening on port 4567. We start our sinatra.vcl by specifying where to find sinatra.

backend sinatra {
  .host = "127.0.0.1";
  .port = "4567";
}

We can now start varnishd and verify that http://localhost:1234/work works.

sudo varnishd -f /path/to/sinatra.vcl -a 0.0.0.0:1234

The cache time to live is set in vcl_fetch. We add a ttl of 1 minute to our vcl and restart varnishd.

sub vcl_fetch {
  set obj.ttl = 1m;
}

At this point running ab -c 100 -n 1000 takes 5 seconds to complete. Note that Varnish only sent the first request to the app server, and queued the others. When the request finished the queued requests were served from cache.

Troubleshooting tip: Now comes the frustrating part. Nothing seems to be cached when you test this in your browser. Every request takes 5 seconds. Luckily varnishlog shows your browser is sending a cookie. You remember cookies are per domain (not per domain:port), so cookies set when working on localhost earlier are still being sent. By default Varnish doesn't serve cached results to sessions with cookies. This is useful to serve cached pages to anonymous visitors while logged in users get custom versions of the page. Varnish is quite customizable, but for now we will instruct varnish to ignore cookies in our vcl_recv.

sub vcl_recv {
  unset req.http.cookie;
}

Now we have a painful first request, then all further requests for a minute are speed (my $300 laptop gets over 6,000 req/s with 1000 concurrent requests). After the cache expires, we start over with requests being queued while a single request hits the backend. So every minute all requests are blocked for 5 seconds while the backend works.

Being graceful: release the queue

Varnish optionally allows serving the dirty page during a grace period. Varnish sends a single request is sent to the backend and serves all other requests from the dirty cache.

To make sure we never send pages older than 1 minute, we will reduce the TTL to 45 seconds and add a 15 second grace by updating vcl_fetch/vcl_recv.

backend sinatra {
  .host = "127.0.0.1";
  .port = "4567";
}

sub vcl_recv {
  unset req.http.cookie;
  set req.grace = 15s;
}

sub vcl_fetch {
  set obj.ttl = 45s;
  set obj.grace = 15s;
}

Now a single user will hit a 5 second request every minute but thousands of other requests will benefit from that user's generosity.

More caching

You should be caching the page on the client as well. Adding even a small Cache-Control for the browser will improve visitors experiences.

Wikia/Wikipedia, FunnyOrDie, and others have reported impressive results with Varnish. Varnish is very extensible and supports ESI (server side include). Varnish has great tools (varnishtop, varnishdist, varnishlog) to see how your systems are behaving. Varnish takes headers very seriously, specially regarding cache-control, cookies.