Deploying a Rails site with Passenger removed a lot of complexity. No more monit/god to monitor and restart mongrels. No more HAProxy/Pound/Nginx hacks to load balance between mongrels. But don't forget that while Apache will be setup to rotate its logs, your rails logs will consume your entire hard drive if you don't add a logrotate for them.
Add /etc/logrotate.d/passenger
A logrotate configuration file specifies the location of the logs with rules about how to modify them. Your capistrano deploy.rb will specify the base path that contains shared/log directory and current/tmp which our rotate script needs.
set :deploy_to, "/home/deploy/app"
Here is what I've found to work for me.
/home/deploy/app/shared/log/*.log {
daily
missingok
rotate 30
compress
delaycompress
sharedscripts
postrotate
touch /home/deploy/app/current/tmp/restart.txt
endscript
}
Interpretive 'man logrotate'
-
daily - Do this every day
On userscripts.org, my logs are rotated every morning at 6:25am. To find out when your system will run, cat /etc/crontab.
-
missingok - Don't worry if there are no logs
Your logrotate software is not how you want to learn that your site is down.
-
rotate 30 - Keep 30 days worth of these around
Rails logs compress well. 30 days compressed is less than a couple of days of uncompressed logs. If something goes wrong you can parse the logs.
-
compress - Compress the rotated logs
gzip the log files so they only take up a fraction of the space
-
delaycompress - Compress yesterday's rotation, not the current rotation
Logrotate will move production.log to production.log.1, but since passenger is still running it will still be writing to the file. Rather than than forcing all requests to be terminated so the file can be compressed, we'll just do it tomorrow. By then all your passenger processes will have been restarted so they will be writing to the new production.log.
-
postrotate/endscript - run some code after rotation which tells passenger to restart
After we are done moving production.log to production.log.1 we need to restart our application servers. Over the course of several minutes passenger will kill old app servers and start new ones. The old ones won't get any new requests so production.log.1 will stop growing after existing requests are finished.
-
sharedscripts - only run postrotate once, not once per log file in shared/log
You don't want to touch tmp/restart.txt for each log file (newrelic_agent.passenger.log, searchd.log, backgroundjob, and so on).
Testing and hourly rotation
Assuming you've been running your site for a while you will have some existing logs you want to test your new setup with. Rather than staying up all night, you can manually run logrotate using:
logrotate -f /etc/logrotate.d/passenger
Then you should have .1 files for each log in the log directory. Passenger should be restarting, and a new production.log should start growing with each new request.
Logrotate is smart enough to only run once daily even if you try to run it manually. There is no hourly option for logrotate, but if you remove daily and add size 1 you trick logrotate into rotating every time you call it. Cron will still only call it once daily, but you can add your own hourly call. Before newrelic I would rotate my logs hourly and analyze them with production log analyzer in postrotate.
Void where prohibited
I hope this helps make sure your server doesn't crash due to run away log files. Your setup might be different but logrotate is pretty easy to tweak if you know where to look.
