Mark Watson's opinions on Java, Ruby, Lisp, AI, and the Semantic Web

Awesome - in the past I have had to write my own code/scripts to manage attaching and EBS file volume to a new EC2 instance. This new feature will make it a lot easier to manage your own EC2 based services. If you would prefer a PDF with the complete documentation, then use this link.

This new feature will make using EC2 even easier for some applications - a welcome change.

That said, I have my own scheme for automatically mounting EBS volumes, assigning ElasticIP addresses, etc. and for some deployments I will continue to use temporary boot volumes and pull AMIs from S3.

Labels: Amazon, EC2

Please note that if want to more secure installation, SSL should also be installed following these instructions (I used these instructions and another web blog to create the following abbreviated instructions). For my purposes, basic HTTP authentication is good enough. I assume that you are used to using nginx and CouchDB and either installed them from source or using apt-get. I am using Ubuntu, so you might have to modify these instructions slightly. On my laptop, I created a simple crypt program because OS X does not include one:

#!/usr/bin/perl
print crypt($ARGV[0],$ARGV[0])."\n";

After giving this script execute permissions, I created an encrypted password:

crypt my12398pass61

You should save the output because on your EC2 instance you need to, as root or sudo, edit the file /etc/nginx/htpasswd adding a line:

couchclient:myEKNgP2ivVVo

where myEKNgP2ivVVo was the output from crypt for the plain text password my12398pass61. Then edit nginx.conf file adding something like:

    server {
        listen       9001;
        server_name  example123.com; # not a real domain name
        location / {
          auth_basic "Please login to use CouchDB";
          auth_basic_user_file /etc/nginx/htpasswd;
          proxy_pass http://localhost:5984;
          proxy_redirect off;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }

If you restart nginx, then you should be able to access

http://example123.com:9001/_utils

You will have to enter couchclient as the user name and my12398pass61 as the password. I allowed my browser to set an authentication cookie so I would not have to keep logging in. (Obviously, you should use a different user name and password with crypt and setting up your /etc/nginx/htpasswd file.)

For a ruby client, do a "gem install couchrest" and try this:

require 'rubygems'
require 'couchrest'

db = CouchRest.database!("http://couchclient:my12398pass61@example123.com:9001/testdb")
response = db.save_doc({:key => 'value', 'another key' => 'another value'})
doc = db.get(response['id'])
puts doc.inspect

You should be good to go writing Ruby applications that use your remote CouchDB service.

This installation is not very secure and should probably not be used on a production server containing sensitive data. I am not a security expert; if you are then I would appreciate your comments on this blog entry.

- - -

PS. an hour after writing this blog, I found a simpler solution of using a SSH tunnel. Check this out on the Disco Blog. You set a tunnel like:

ssh -i ~/.ssh/id_rsa-gsg-keypair  -L 5984:localhost:5984 root@ec2-31-111-149-100.compute-1.amazonaws.com

If you use an Elastic IP address so your server always has the same IP address, then this ssh command can be aliased, for fast temporary connections to CouchDB and other services that are confgured for only localhost client connections.

Labels: CouchDB, EC2, nginx, Ubuntu

The Lucandra project looks very interesting, but is incomplete at this time (see the "to be dones" at the bottom of the linked page).

Cassandra is a great project. I almost incorporated it into the design of a customer project recently, but we decided to host on Amazon so using their EC2, S3, SQS, and Electric Map Reduce services won out over rolling a custom stack.

I think that this must be a start up dilemma: long term, it is probably least expensive running one's own small server farm, but when you are just getting started a "pay as you go" cloud approach using very solid infrastructure tools like EC2, S3, SQS, SimpleDB, etc. makes sense.

I can't say this from personal experience, but my gut feeling is that if you can live within the constraints of Google's AppEngine, then it is probably less expensive using AppEngine than running your own server farm - even long term. BTW, if you have not read my DevX article on implementing search on the Java version of AppEngine, please check it out.

Labels: Amazon, EC2

I remain very enthusiastic about Google's AppEngine (and also I am very much enjoying my developer's Wave account). That said, Amazon's AWS services are having a much larger effect on my work for customers and my own work and research. AppEngine is great for some types of projects, but EC2 can be used for anything.

I have a text mining experiment that have been planning for a while, and today I have some free time to start setting it up. I have 3 old desktop computers (with a reasonable amount of memory and disk) that I usually haul out of my closet, run "headless," and set up for text mining and machine learning projects. Although I own these boxes, there is a drawback to leaving them running for several weeks in my home office: noise, heat generation, messing up my work environment, etc. I did a quick calculation and estimated that if I instead use one EC2 instance, a reasonably large ESB disk volume, and Elastic MapReduce when I need it to make Hadoop Map Reduce runs, the cost over a few week period is a small business expense. Anyway, I am setting up a work environment on an EC2 instance "as we speak."

My wife and I often evaluate what we really need in our home (we like to keep our small house tidy and elegant) and I would like to get rid of old computer hardware that I can do without - and, in the USA, our schools are poorly funded so contributing old hardware to our local high school with the latest Ubuntu installed could be a good thing.

Note: I would like to thank Amazon for providing a grant to me to cover my EC2, S3, and Electric MapReduce expenses for writing my last Ruby book's examples and permanently providing a MCI with all of the book's examples.

Labels: AI, EC2, Hadoop, textmining