Exploring Amazon EC2

Thu, 24 Aug 2006 amazon ec2 rails comments

Amazon just released Elastic Compute Cloud (EC2). Amazon recently released scalable storage infrastructure (S3) and scalable (artificial) artificial intelligence (Mechanical Turk.) Now they have done the same thing for computation.

By creating (linux based) virtual machines, either from scratch or based on prebuilt ones (web, app and database servers), you can quickly deploy on a pretty powerful machine (1.7Ghz Xeon CPU, 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth.) You can remotely start and stop instances of your image using web service calls (as well as standard unix shutdown) and you are only charged for the time the server is running (since you have a dedicated machine.

You pay $0.10 an hour while the instance is running (and you can have multiple instances for more dimes.) Then you pay for bandwidth $0.20/GB. You can also send data to S3 for access to data when your EC2 is not running (normal storage fees for S3 apply, but you are not charged for the transfer between EC2 and S3.)

The possibilities this opens are large for the right kind of problems. For all those folks who are building YouTube clones, it should be much easier now! And at $876.00/year (not including bandwidth) it is very competitive with conventional hosting.

The product is in limited beta.

Getting Started

Step 0: have a linux box with java (or a mac)

Step 1: Create your account at http://aws.amazon.com

Step 2: Create a X.509 Certificate and download the private key and certificate to ~/.ec2

Step 3: Download the command line tools for working with EC2 (requres certificate from previous step)

Step 4: Setup environment varaibles (unzip and from within the folder it creates):

export EC2_HOME=`pwd`
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=`ls ~/.ec2/pk-*.pem`
export EC2_CERT=`ls ~/.ec2/cert-*.pem`

Step 5: List images: ec2-describe-images

This command should return a list available images. (on a mac you must set JAVA_HOME by doing: export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/ )

IMAGE   ami-68ae4b01    ec2-public-images/fedora-core4-base.manifest    206029621532    available       public
IMAGE   ami-69ae4b00    ec2-public-images/fedora-core4-apache-mysql.manifest    206029621532    available       public
IMAGE   ami-6bae4b02    ec2-public-images/getting-started.manifest      206029621532    available       public
IMAGE   ami-6dae4b04    ec2-public-images/fedora-core4-apache.manifest  206029621532    available       public
IMAGE   ami-6fae4b06    ec2-public-images/fedora-core4-mysql.manifest   206029621532    available       public

So we have Fedora Core with or without apache/mysql to choose from.

Step 6: Create a Keypair - ec2-add-keypair gsg-keypair

You will need to take the private key that it creates, save it to a file called id_rsa-gsg-keypair, then set it's permissions to be rw------- (chmod 600 id_rsa-gsg-keypair)

Step 7: Create an instance of your machine

ec2-run-instances ami-xxxxxxxx -k gsg-keypair (where ami-xxxxxxxxx is the id of the machine you used to use - found in step 5 with ec2-describe-images)

This step failed on me a few times, but eventually it reported that it created my instance.

RESERVATION     r-xxxxxxxx      ############    default
INSTANCE        i-xxxxxxxx      ami-xxxxxxxx            pending gsg-keypair

Step 8: Wait for it...

Using ec2-describe-instances you can check on the status of your machine, eventually it will display something like this:

RESERVATION     r-xxxxxxxx      ############    default
INSTANCE        i-xxxxxxxx      ami-xxxxxxxx    domU-##-##-##-##-##-##.usma1.compute.amazonaws.com      running gsg-keypair

Step 9: Authorize SSH & HTTP

ec2-authorize default -p 22 (ssh)
ec2-authorize default -p 80 (and http)

Step 10: Visit your page at the url given in the previous step

Now the fun begins!

You can ssh into the machine using the key you created in step 6
ssh -i id_rsa-gsg-keypair root@domU-##-##-##-##-##-##.usma1.compute.amazonaws.com and you will be greated with some ascii art

         __|  __|_  )
         _|  (     / 
        ___|\___|___|

 Welcome to an EC2 Public Image
                       :-)

Stats:

df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208    780852   9016068   8% /
none                    870472         0    870472   0% /dev/shm
/dev/sda2            153915428    192676 145904308   1% /mnt

cat /proc/cpuinfo

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 37
model name      : AMD Opteron(tm) Processor 250
stepping        : 1
cpu MHz         : 2405.452
cache size      : 1024 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm ts fid vid ttp
bogomips        : 4812.43

Lots more fun can be had, but I decided to try to setup a quick app

Yum included

I decided to test rails on the server, a quick setup based on Installing Ruby on Rails with Lighttpd and MySQL on Fedora Core 4. (install gcc, mongrel, and a few other friends)

You can check out the results here

Responses to "Exploring Amazon EC2"

Thu, 24 Aug 2006 Bernie Thompson says:
Thank for you this nice write-up. Hopefully, they open up that limited beta a little more soon. This kind of scalable computing resource is especially interesting to those creating small web services which may or may not take off. Many of those are getting done with Ruby on Rails (which attacks the problem of lowering initial development cost, while EC2 lowers initial hosting cost).
Fri, 25 Aug 2006 gmouse says:
How does one shutdown their machine once they get it started? ec2-terminate-instances
Sat, 26 Aug 2006 gmouse says:
Great article! Got my mac up and running with ec2 in about an hour. Lets say you setup and instance with some settings (ie install software, config users) how do you do a shutdown and still save have you made and just turn on your instance when you need it? thanks
Wed, 30 Aug 2006 Jeremy Dunck says:
BTW, tools have been fixed for Windows. They're now live at the normal download location The howto would obviously be different for Windows. :)
Wed, 30 Aug 2006 Jeremy Dunck says:
BTW, tools have been fixed for Windows. They are now live at the normal download location The howto would obviously be different for Windows. :)
Fri, 01 Sep 2006 Knox says:
Heads up re: Mac Java. Because Java 1.4 was the default on my Mac it gave an "Unsupported major.minor version 49.0" exception. I needed to install/update Java 2 SE 5 from http://www.apple.com/support/downloads/java2se50release3.html and then do export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.5/Home/
Fri, 08 Sep 2006 Jason says:
Great write-up, took me right through it. Any ideas on how to best install the kernel sources via yum to compile some stuff requiring the kernel sources? Not looking to change the kernel...
Fri, 08 Sep 2006 Brian says:
Thank you for the great guide. It helped me get an instance up and running very quickly.
Wed, 13 Sep 2006 Vishi says:
Hey may be you can put a neatly configured image with Rails installed on S3 and give access to others? Some capistrano scripts will make it so easy! I surely will pay for a neat rails setup that I can deploy my app on from my svn repo,
Tue, 19 Sep 2006 mathie says:
Thanks for the succinct instructions. I've taken the next step and described how to upgrade to FC5 and create your own AMI here: http://woss.name/2006/09/19/setting-up-an-amazon-ec2-server-with-fedora-core-5/ I'm aiming to get a reusable image for rails apps, but taking it one step at a time...
Sat, 15 Dec 2007 Martin Rehfeld says:
Now that Amazon has annouced SimpleDB, I think it's time to add that to the equation, some thoughts on http://inside.glnetworks.de/2007/12/15/amazon-simpledb-web-service-complementing-the-ec2-compute-cloud/ The Rails deployment stack just made a big leap ahead!
Fri, 28 Dec 2007 Rajakumar says:
How do I start a shutdown instances, Created one EC2 instances and shutdown with ec2-terminate-instances now I want to restart it Thanks
Mon, 31 Dec 2007 elliottcable says:
I hate to be that guy, but even looking at all these success comments... I can't get it to work. There are no instructions to 'install' the UNIX tools for EC2, so I'm assuming you just stick them in ~/.ec2 with everything else, so I've got my cert, my pk, bin/ and lib/ in ~/.ec2 Now, I try to run .ec2/bin/ec2-describe-images, and it just prints absolutely nothing - but it has a successful exit code. I can't get past that step.
Tue, 01 Jan 2008 elliottcable says:
Regarding my previous comment, it seems that something has changed so the command no longer prints Amazon's public images. For that, you need to run `ec2-describe-images -o self -o amazon`
Tue, 01 Jan 2008 Deepen Shah says:
elliot try `ec2-describe-images -a`
Fri, 07 Mar 2008 alex says:
Great stuff!! I found a few more here: http://www.newwebplatform.com/tips-and-tutorials/Amazon-EC2
Mon, 10 Mar 2008 afewtips says:
Has anyone figured out how much a "typical" setup would cost? Maybe compared to slicehost. It says that you pay .10 an hour - if noone visits your url, is your instance still running? Or is it running when an http call is made or processing? Would running Zope mean the instance is always running? Thanks
Tue, 11 Mar 2008 afewtips says:
I think I found the answer - it's based off of API calls. You start your instance and then close your instance if needed from an API call. It looks to me like this is the more expensive route if you are a small fish and don't have large scaling and speed to market needs. Thanks
Tue, 01 Apr 2008 Zak Kinion says:
Thanks, this has been helpful, even later on when some of the commands are different.
Thu, 08 May 2008 sd says:
To run a website on the cheapest EC2 configuration, you need to spend $72 per month (plus $0.20/GB extra for bandwidth). Is this correct?
Sun, 08 Jun 2008 tss says:
SearchBlox has released its content search software on EC2. Now pay as you go search available starting at 35 cents per hour. http://www.searchblox.com/gettingstarted_amazon_ec2.html