Mark Watson's opinions on Java, Ruby, Lisp, AI, and the Semantic Web

Semantic Web: an alternative for RDFa

2010-02-15T16:25:00.003-07:00

A few years ago I thought that XHTML would eventually be widely used but when the W3C decided to standardize on HTML5 (which I love for non Semantic Web reasons), that may have been the beginning of the end for RDFa because RDFa is an XML application.

I believe that a better alternative in a HTML5 world is to keep RDF separate from web pages but have a clear set of rules for finding RDF data files that correspond to web pages (either static or generated). One rule might be to look for a file named index.rdf for top level domain URLs; for example, see if http://markwatson.com/index.rdf exists for http://markwatson.com. For a URL like http://markwatson.com/hobbies look for http://markwatson.com/hobbies.rdf or http://markwatson.com/hobbies/index.rdf.

Although CMS support (e.g., Drupal) for RDFa and helper libraries like the RDFa Rails plugin might make it fairly easy for some web sites to provide RDFa, I think that we need something simpler that might be adopted by more web sites.

I am writing an open source tool (that will be an example program in the Semantic Web book I am writing) that will generate RDF data from web pages. I'll post a link when the code is ready.

Click is now a top level Apache project

2010-02-12T15:56:00.003-07:00

Congrats to the Click team. My interest in doing Java web apps is low since I have mostly been using Rails for the last three years.

That said, Click hits a sweet spot with a good templating system and model super classes for pages, forms, tables, etc. Because I might be using Google AppEngine for more projects, I may need to use Java so I have just read through the documentation and code samples for the latest version that is compatible with AppEngine.

Saving money: using a 64bit Ubuntu VPS instead of a 64bit Large EC2 instance

2010-02-12T15:14:00.003-07:00

I need to frequently run a 64bit Linux web service (commercial app, not available as a 32bit app). Unfortunately, small EC2 instances are only available as 32bit. I ended up using a 64bit VPS instance (I use RimuHosting, but I think Slicehost also has low cost 64bit VPSs). Saving a lot of money over the long term :-)

A followup on using Windows 7 for Ruby and Java development

2010-01-24T12:00:00.004-07:00

I wrote 2 weeks ago about adapting to using my new Windows 7 laptop (inexpensive, well constructed, and 4 cores). This is a followup with some miscellaneous advice.

I was having some problems writing with Latex so I re-installed MikTex doing a complete installation (well over 1 gig). My Latex problems are solved with some brute force. Still, when I know that I will be mostly writing, I still boot my MacBook to use TexShop, etc.

With a much faster laptop (4 cores!), I am very much enjoying using the new IntelliJ 9 for Java, Scala, and Clojure development. I am currently writing code that wraps the Sesame RDF data store, my own geolocation code, and Lucene. (I am writing a book on the AllegroGraph product and I want all of the examples to also run using my wrapper for Sesame, my geolocation stuff, and Lucene.) I am also writing idiomatic wrappers (to my wrapper) in Clojure and Scala and it is great to have everything in one large IntelliJ project.

I continue to rely on E TextEditor (a Windows TextMate clone) for most Ruby development and miscellaneous text editing - highly recommended.

I find that once I get things set up on my Windows 7 laptop, then I have an enjoyable developers experience. However, setting it up has been a terrific pain in some cases, for example: installing PostgreSQL and PostGIS was a real nuisance.

I found the text in the Command Prompt windows where I run bash shells to be a little difficult to read so I increased the default window size, switched to a larger font size, switched the default font to Lucinda Console, and reduced the contrast by making the background a very light blue (almost white) and the text a medium blue.

I also find it useful to remap the CAPS LOC key to act as a third control key - this reduces strain on my left hand when hitting control characters on the left side of the keyboard while using Emacs and other software that makes good use of the control key. This saves wear and tear on the tendons because I don't have to twist my left hand or to use the control key on the right side of the keyboard.

On dual booting Ubuntu: as I mentioned a few weeks ago, I have both a bootable Ubuntu installation and a separate one that runs inside of Windows using VirtuaBox. I am just about ready to figure out how to remove grub and reclaim the separate partition. It is very convenient using VirtualBox so I think I am going to just use it when Linux is best for a work task.

One big advantage of using Windows is the utilities TortoiseSvn and TortoiseGit. Highly recommended to integrate svn and git support into the file explorer.

One huge disadvantage of Windows is that most new computers do not come with Windows install disks. This is awful - shameful behavior, really, on both Microsoft's and computer manufacturers' parts. This is simply lack of trust in their customers. I have burned a few "bootable repair/recover" disks and I am using the built in system backup software (backs up to an external disk), but I won't know if this works until I need it.

The bottom line is: when I am working in a bash shell, SSH'ing to remote servers, or using a heavy weight IDE like IntelliJ then I find Windows 7 to be equally pleasurable for development use as OS X or Linux. For some things, I still like my MacBook. I paid $800 for my new Toshiba laptop and a comparable MacBook Pro for my work flow would have been about $2000. I am not yet sure if the price savings has been worth it because of the time required to set up my new laptop. One advantage though is the flexibility of also having a Windows box handy for testing, etc.

Very bad decision by conservatives on the supreme court

2010-01-22T10:15:00.008-07:00

Horrible supreme court decision! A great deal for international corporations to get to powerfully affect our internal elections in the USA. I don't want to overstate my opinion, but I think that this is the beginning of the end for fair representative democracy in our country. I hate to say it, but I am so fed up with the greed and avarice of the "elites" that I find myself not caring much about politics anymore. It would be a good first step for all members of the supreme court to publicly state any family members' business affiliations that might affect their rulings on cases. The justices have a long road to regain our trust.

On a more positive note, I do feel more like a 'citizen of the world' and I believe that stronger business and personal relationships with people around the world help offset the terribly high level of corruption in the USA and other countries that don't give good service to tax payers.

This is a simple prediction to make: corrupt governments will over time become less competitive for promoting and encouraging productive business activities in their tax jurisdictions. I am all for strong corporations and strong businesses both large and small. What I am against is what I consider to be illegal activities (or activities that should be illegal). We need a small government that enforces a small number of fair laws and then stays out of the way of doing business. Sadly, that is far from what we have now.

The beauty of Latex: my AllegroGraph book becomes two books, one for JVM languages and one for Lisp

2010-01-21T10:49:00.009-07:00

I have been working on and off for 16 months on a book about Semantic Web (or Linked Data) application programming using the AllegroGraph product. I have decided to substantially increase the scope of this applications/tutorial style book to also include support for Sesame. The figure on the left shows the software architecture road map for the book using JVM languages.

I am splitting the book into two volumes, and using Latex makes this really easy to share small amounts of common material so both books stand on their own. Latex also makes it easy to combine both books into one all-inclusive book, eliminating the duplicated parts. The two volumes are:

Volume I: will cover the use of both AllegroGraph and Sesame using JVM languages: Java, Scala, JRuby, and Clojure. I am working on a common wrapper written in Java that supplies my own (rather simple) API to both AllegroGraph and Sesame. My wrapper implements Sesame support for geolocation and free text indexing and search so the wrapper is adequate to run all of the book examples using either AllegroGraph or Sesame "back ends."
Volume II: will cover only AllegroGraph using both the embedded and client Lisp APIs.

Both AllegroGraph and Sesame are great development tools, but fill different needs. On projects that can support a several thousand dollar a year per server license fee, I would choose Common Lisp + AllegroGraph for development. AllegroGraph is very scalable and the Lisp APIs are really nice to work with. For Java (or other JVM languages) applications, I would still choose AllegroGraph for the scalability and support if a project can support the license costs. The good thing is that for most small to medium size projects, the free version of AllegroGraph or the open source Sesame project both are good choices, so as a developer you have some real flexibility. There are also other good RDF data store platforms like Jena, Joseki, Kowari, Redland, 4store, Swi-Prolog Semantic Web library, Talis, Virtuoso, etc. but I have relatively little (or in some cases no) experience with these. I use AllegroGraph and Sesame so that is what I write about.

Looking towards a universal wrapper/proxy for knowlege and data stores

2010-01-13T08:42:00.004-07:00

I often start out by writing code specific for a single project and then refactor it to make working code more generally useful. I am working on an applications book for the AllegroGraph RDF data store services. Since most people (probably) use Java clients with AllegroGraph, the first step is to wrap Franz's APIs with my own interfaces so for my own work (that is, beyond the scope of writing this book), I can write implementations for other back end RDF data stores as I need them.

I also plan on writing "thin" Scala, JRuby, and Clojure friendly interfaces to my Java library. For the purposes of the book, I'll use this library to support example client applications written in Java, JRuby, Scala, and Clojure. I have a lot of material already written with Lisp examples but I think that I am going to set that all aside for a future writing project (that I may, quite honestly, never get back to). I also decided to not support Python in my book: Franz has a good Python interface library and examples and in any case, I am not a Python developer.

So far, I have a fairly clear road map of what I need for this specific book project. Long term, after this book is done, I am aiming to also wrap other knowledge sources like OpenCyc, Freebase, etc. While it is tempting to view most knowledge sources as graph data (RDF), it seems like a poor idea to give up the inferencing available in OpenCyc, all the features of the Freebase MQL query language, etc.

Since I often find myself reusing my own small code examples to access multiple knowledge sources, it may be time soon to step back and decide what can be placed behind common interfaces.

Using Windows 7 for Ruby and Java development

2010-01-10T13:53:00.004-07:00

As I mentioned in my last blog, I surprised my friends and family by buying a Windows 7 laptop. The combination of Windows 7 and Ubuntu is not quite as good as OS X and Ubuntu dual boot, but try buying a Mac laptop with 4 CPU cores.

Of course, the first thing to do to a Windows 7 system is to install cygwin. I installed just about everything available. To avoid confusion, I always run bash in command windows and I set up my .bashrc file for cygwin to mimic my .profile file for OS X and my .bashrc file for Ubuntu. With cygwin installed, life is good.

For basic Ruby and Java development, Rubymine and IntelliJ work identically under OS X, Windows 7, and Ubuntu Linux. I needed a plain text editor: I use TextMate on OS X and GEdit on Ubuntu. I tried, then bought a copy of E TextEditor that works with TextMate plugins. Recommended! I also installed XEmacs.

For writing I installed OpenOffice.org and Latex (I used the MikTex distribution). Both E TextEditor and XEmacs are fine for editing Latex "source code."

This was some trouble to set up, but my Toshiba laptop is very well constructed, has 4 CPU cores, 4GB of memory, and 500GB of disk. And, it was very inexpensive.

I have not totally made up my mind how I will use Ubuntu on my new laptop. Windows 7 has a nice utility to split the c: disk partition so I did this when I first started the laptop. I have Ubuntu installed twice: on a new partition and also on the Windows 7 file system using Sun's very good VirtualBox. Ubuntu is a little more responsive when I use grub to boot it directly, but it is very convenient running it under VirtualBox because of file system sharing, not having to install everything - just what Linux is best for. Since I use a remote git repository for virtually everything that I work on, it is easy dealing with two Ubuntu installations - I think that I will continue to use both for a few months, then decide which to keep and which to delete. Also: VirtualBox supports all 4 cores with 64 bit Ubuntu running as a guest operating system so the performance hit for using VirtualBox is small.

New laptop: Toshiba Satellite U505

2010-01-05T11:07:00.002-07:00

I very much like the MacBook that I bought almost three years ago, but it has it limitations (mainly not enough disk space).

I decided to buy a "Windows" laptop because for $800 I could get a laptop with 4 CPU cores (Intel CORE i3), 1/2 terabyte disk, and 4 gigs RAM. Ubuntu Linux uses all the cores :-)

Running OpenCyc 2.0 on OS X

2010-01-02T10:43:00.004-07:00

The latest release of OpenCyc uses a Java runtime so it is portable. I often keep OpenCyc running on one of my servers, but for convenience I also wanted to be able to run it on my MacBook. My MacBook only has 2GB of RAM but this seems to be adequate, especially because I don't run the JVM in server mode on my laptop. Start by downloading the Linux OpenCyc 2.0 distribution.

Make a copy of the file opencyc-2.0/server/cyc/run/bin/run-cyc.sh and replace the original file contents with:

#!/bin/bash                                                                                         

MIN_HEAP=1300m
MAX_HEAP=1300m
PERM_SIZE=98m

java  -Xms$MIN_HEAP -Xmx$MAX_HEAP -XX:MaxPermSize=$PERM_SIZE  -cp lib/cyc.jar:lib/subl.jar:lib/juni\
t.jar:resource:lib/ext:plugins com.cyc.tool.subl.jrtl.nativeCode.subLisp.SubLMain  -f "(progn (load\
 \"init/jrtl-release-init.lisp\")))" "$@"

With these settings, OpenCyc 2.0 starts up quickly and has fairly good runtime performance on my MacBook: fine for experimenting with OpenCyc. Run the system using the top level run script (which you do not need to modify):

cd opencyc-2.0
scripts/run-cyc.sh

OpenCyc 2.0 consists of a AI reasoning runtime, a "real world" ontology, hundreds of thousands of terms and millions of relationships.

My tech industry predictions for 2010

2009-12-31T11:50:00.008-07:00

As both an author and as a technical consultant, I am fairly opinionated in terms of what I expect for next year (tomorrow!). Here are my predictions:

There will be pressure to reduce IT expenditures. Increasing trend to favor outsourcing deployment platform (e.g., Google AppEngine and Heroku), infrastructure (e.g., Amazon AWS, RackSpace, both relational and NoSLQ data stores), and software as a service (e.g., CRM, etc.). Cloud computing gets more real.
Research will be concentrated on shorter term profits rather than long strategy. China might be an exception to this: a friend reports that he has seen willingness in China to fund very long term Artificial Intelligence research.
More people will spend more time using web based information and recreational resources using portable devices. There will be a shortage of wireless bandwidth in some areas.
Use of the Java platform will stay strong, but with more emphasis on alternative languages like JRuby, Scala, and Clojure.
Skills and education in data analysis, all types of modeling, text mining, and analytics will continue to be in increasing demand.
There will be well publicized sporadic failures by cloud service providers but customers will be tolerant because of cost savings.
The importance of the hugely increasing availability of all types of data will encourage development of new business areas like knowledge processing as a service.
Although the Semantic Web has seen slow adoption in the last decade, Linked Data, a new and more popular subset of the Semantic Web, will see wide adoption in the next few years.
The wide support for Linked Data will help make computer to computer interactions easier and more profitable.
Geo location will become increasing important. We are likely to see many interesting smaller scale systems using technology like PostGIS and Spatial Lucene that will be able to compete in local markets with Google, Microsoft, and Yahoo.
I think that the next decade will be a time of micro web businesses: a huge number of small scale web portals that offer very specific services to a relatively small number of customers who are willing to pay a use fee. Most of these will be deployed on very low cost cloud hosting. Many of these will be tightly integrated with the Facebook platform, the Wave platform, etc.
In the next 10 years: the USA and Europe will become much less important in technical inovation because their large debt liabilities will create business and research unfriendly environments.
In the next 20 years: educated, creative, and/or generally productive people will identify more with similar people around the world than with people in their own countries who may get bogged down in what looks to be a crushing world wide economic collapse (but not quite as bad in developing countries). There will be more of a feeling of world wide citizenship, the world will evolve first into a safer place (a trend over the the last 20 years has seen more safety, less civilian deaths - something that most people don't realize) and then eventually into greater prosperity.

Next year looks like an exciting time in our industry!

Best Wishes for a Happy New Year

$1.5 trillion a year for "defense" spending, little money left for local governments; living locally; Happy Holidays

2009-12-24T12:25:00.004-07:00

If you factor in the cost of the debt to pay for our military spending then I think that a reasonable estimate of our yearly defense spending is about $1.5 trillion.

The amount we spend on defense swamps other US government spending, including social programs. This is just my opinion, but I believe that we could keep our country relatively safe (compared to other countries) and spend far less money. The problem is basically banana republic style corruption: too much money is made by special interests for there to be any meaningful reform of military spending. The same comment is also true of other corporate interests like Wall Street, industrialized food production, pharmaceutical companies, insurance, etc. This is all enabled by total corporate control of the news media and shameful corruption in the lobbying industry and our federal government.

Fortunately, for most of us, life is still very good despite corruption of the world's "elite." Again, this is just my personal opinion: it is best to act in ways that help yourself, your family, and local community. Avoid doing business with large banks, preferring credit unions. Prefer buying food from local producers. Form or join local organizations for food distribution, computer clubs (promote open source solutions to help people use old PCs), and help with local school fund raising activities. Prefer to buy goods that are manufactured as close to where you live as possible.

Joy in life comes from our family and social interactions and in acting in service to others. A materialistic lifestyle is something that you have been conditioned to accept by lifetime exposure to media - try to resist the temptation to acquire material goods that you really don't need.

And, most of all:

Have a Merry Christmas and a Happy Hanukkah!

Building the EtherPad system and perusing the source code

2009-12-18T09:05:00.004-07:00

The EtherPad collaboration-enabled online system is now open source. Cool. Google bought the company, the developers are joining the Wave team, and their product is now released under the Apache 2 license.

Be sure to follow the instructions (failing to set the environment variable for the path to a MySQL client JAR file produces a strange "cp -f" error that has hung up a few people trying to build the system, as reported on Hacker News). It only took about 15 minutes to download the source and build the system - simple enough.

After running the system and trying it, I used IntelliJ 9 to set up a project (choose project from existing source, main directory etherpad/trunk) and I am spending some time perusing the Scala and JavaScript code. Really nice looking code base, and reading through the Scala code will be an education. My JavaScript skills are a little weak, but I might still take a careful look at the JavaScript code to understand how they used Comet.

Amazon Elastic Load Balancing (ELB) is pretty cool

2009-12-14T17:08:00.005-07:00

Using this service costs $0.025/hour so it may make sense to just run HAProxy yourself on a EC2 instance, but then you have to worry about fault tolerance/recovery if that instance fails. The ELB cost is small in comparison to running a cluster of EC2 instances and "outsourcing" as much of your system as possible to AWS (e.b., SimpleDB, Elastic Load Balancing, Relational Database Server, EBS, etc.) can certainly reduce both the complexity of your application architecture and also your implementation costs.

Here are my notes for a simple ELB setup for an AMI that contains a Rails web application:

export EC2_PRIVATE_KEY=pk-....pem  # different on your system
export EC2_CERT=/Users/markw/.ec2/cert-...pem  # different on your system

ec2run -k gsg-keypair  ami-e767ds71
ec2run -k gsg-keypair  ami-e767ds71

Note: specifying "gsg-keypair" matches later doing ssh -i ~/.ssh/id_rsa-gsg-keypair ...

elb-create-lb  MarkTest123 --headers --listener "lb-port=80,instance-port=4242,protocol=HTTP" --availability-zones us-east-1b

elb-configure-healthcheck  MarkTest123 --headers --target "HTTP:4242/ping" --interval 30 --timeout 3 --unhealthy-threshold 2 --healthy-threshold 2

elb-register-instances-with-lb  MarkTest123  --headers --instances i-X,i-Y

* substitute real EC2 IDs for i-X and i-Y, and this works great!

ssh -i ~/.ssh/id_rsa-gsg-keypair ubuntu@ec2-63-212-55-241.compute-1.amazonaws.com
create a file public/ping    - put a line of text in it
start Rails on port 4242

* note: repeat for other EC2 instance. In production, obviously you would want to start your Rails apps using /etc/init.d/...

Then, hitting this URL works: http://marktest123-1645979628.us-east-1.elb.amazonaws.com

Teardown:

elb-delete-lb MarkTest123
and delete EC2 instances

It took me a little while to get ELB working, so I hope that my notes save you some time.

The cost of commoditization of computing: infrastructure and software

2009-12-14T10:03:00.006-07:00

Discipline, a new view of system architecture, and rigorous automation procedures are required to take advantage of Amazon EC2, Google AppEngine, and other infrastructure as a service providers. Last week a customer commented on the rapid pace of Amazon's innovation. Yesterday they announced a new way to generate revenue from unused server instances by letting users bid on a spot market for unused EC2 instances.

Discipline When you own your own server farm, even if it is just a few back room servers, you can spread out applications over your servers in a haphazard way and usually get away with some sloppiness in your process architecture. When you are dealing with someone else's infrastructure a more disciplined approach is just about mandatory.

New view of system architecture Both Google and Amazon have published papers on dealing with very large scale geographically disperse systems comprised of many components, some of which are guaranteed to fail. These companies have also shared how for some applications they trade off immediate data consistency (between distributed data writers and readers) for scalability.

Rigorous automation procedures Large distributed fault tolerant systems involve two types of software development: core application development and writing automation code to manage managing servers (starting/stopping, DNS management) and partitioning an application over available servers.

Software The open source and free software movements have hugely contributed to the commoditization of computing. Even not counting operating system and networking infrastructure, almost all of the code in the systems that we create is written by other people and it is usually open source software. I would argue that in addition to classical skills like algorithms and raw coding skill, a modern developer must be well versed in available frameworks and libraries, their strengths and weaknesses, and knowledge of modifying and using them.

Commoditization comes with cost This is something that I am in the process of learning about. For most of my career I have had either dedicated hardware for development and deployment (starting in the 1980s, developing world wide distributed systems, I had expensive dedicated servers), or in the last 10 years, my "world" consisted of spending a lot of effort building systems on top of open source frameworks (J2EE, Rails, etc.) and deploying to a leased server, or a VPS for projects with few users. In this old scenario, deployment and admin would be a very small part of the effort of creating a new system. Now, designing, implementing, and deploying to Amazon AWS, Google AppEngine, etc. has changed that because of the effort it takes to properly "live" in a commodity computing world.

IntelliJ version 9.0

2009-12-13T11:48:00.009-07:00

This week JetBrains gave me an upgrade license for version 9 of IntelliJ. I don't do too much Java development anymore - mostly maintenance on some of my old projects and new AppEngine and Google Wave development.

Overall, version 9 is a nice upgrade: nicer source code repository integration, built in task management, the IDE seems faster, etc.

I use the JetBrains RubyMine product almost every day. Long term, my use of IntelliJ 9 will depend on how good the AppEngine support is. As a test, I generated a new AppEngine project, added some home page material, edited the appengine-web.xml file to specify a registered app name, set the version, then deployed to Google's servers with no problems. I did have a problem importing an AppEngine project from Eclipse but eventally realized that I needed to go to Module Settings -> Artifacts, and drag "Available elements" from the right window pane to the <output root> tree display in the left pane.

I tried writing a simple GWT 2.0 (new version) web app. When I created a new project, no template GWT client and server code was generated - a small disappointment. I deleted that project and used a command line tool to generate a project:

/Users/markw/Documents/WORK/gwt-2.0.0/webAppCreator -out mytest com.mytest

I then created a new IntelliJ project "from existing source", and had no problems using the generated template project, running the local development server, and using the web app in Firefox (after installing the GWT FireFox plugin).

I am really busy with client work right now but I hope to get some time next year to do an AppEngine based project of my own, and I am almost sure that I will use IntelliJ 9 for that because general Java development is a bit better, I think, than using Eclipse, and it looks like the AppEngine support is OK.

Balkanisation of Ruby?

2009-12-07T13:13:00.003-07:00

When I first started using Ruby, Matz's C-Ruby was mostly the only game in town.

I am also an enthusiastic user Ruby 1.9.x, JRuby, and MacRuby. Seeing the Ruby spec being developed in Japan under government funding, and that it is for Ruby 1.8.7, I get a feeling of déjà vu as a long time Lisp user.

The balkanisation of Lisp has been more than a small nuisance for me in the last 25 years. I would prefer that all Ruby implementations eventually implement a common specification, but I would rather it be for something that looks like 1.9.x.

Privacy and Security in the Internet Age

2009-12-06T18:24:00.002-07:00

Just some advice that I give friends and family:

Delete all cookies in your browser every week - it is easy enough to sign in again to web sites that require authentication. People who do not delete their cookies never see what sites are tracking them. It is easiest to do a 'delete all cookies' operation and not to try to save the 5 or 10 cookies out of thousands that are stored in your local browser data.
Keep a text file with all passwords in encrypted form - and, do not use the same password for different purposes.
Every time you use your super market's discount card (or possibly pay with a credit card), your purchases are permanently associated with you - do you care? maybe or maybe not.

I do use a lot of web services that track what I do (GMail, for example) but I make the decision to give up privacy vs. benefits on a service by service basis.

Coolness! good instructions for trying Rails 3.0pre

2009-12-06T16:27:00.004-07:00

This is worth sharing: easy to follow instructions for installing Rails 3.0pre.

Thanks to Oscar Del Ben!

Yesterday I spent 90 minutes trying a fresh install of Ruby 1.9.1 and Rails 3.0pre with no luck, so thanks to Oscar for his writeup. BTW, I Used Ruby 1.8.7 when following Oscar's directions.

New Amazon Web Services feature: boot from EBS

2009-12-03T11:32:00.004-07:00

Awesome - in the past I have had to write my own code/scripts to manage attaching and EBS file volume to a new EC2 instance. This new feature will make it a lot easier to manage your own EC2 based services. If you would prefer a PDF with the complete documentation, then use this link.

This new feature will make using EC2 even easier for some applications - a welcome change.

That said, I have my own scheme for automatically mounting EBS volumes, assigning ElasticIP addresses, etc. and for some deployments I will continue to use temporary boot volumes and pull AMIs from S3.

Playing with Chrome OS

2009-11-25T07:57:00.002-07:00

I have to say that even in its alpha (or beta?) version, Chrome OS looks good. I like the home "apps page" and except for not having an application that is a terminal emulator, it is fairly complete: my calendar, google docs, etc., are all available, and web browsing is fast.

Again, given a good terminal program for use cases where I need to quickly SSH to a server, I can see a light weight and low power netbook nicely augmenting my laptop + external monitor setup.

I am watching the live Chrome OS Webcast

2009-11-19T11:37:00.003-07:00

For end users, Chrome OS is a great idea - I would argue that most of my friends and relatives would be better off not running Windows, OS X, or (full) Linux.

What about software developers: still useful, but not a replacement for a laptop. In a pinch, assuming a terminal window to run remote bash shells, Emacs, etc., I could still get work done while travelling. Still, for me, a MacBook with Ubuntu and OS X with RubyMine, IntelliJ, Eclipse, OmniGraffle, etc. is just about perfect for my workflow.

That said, I will buy a Chrome OS netbook when they are available.

Hosted MongoDB and CouchDB

2009-11-17T10:49:00.002-07:00

After I finish up some client work this morning, I am planning on finishing a DevX article on using Heroku as a deployment platform. Since deploying to Heroku is so simple and so well documented, you might think that I would have a difficult time writing new material :-)

After a short tutorial on getting started, I am writing mostly about using both CouchDB and MongoDB as data store, either hosted yourself on EC2 (or another server external to Heroku, which is itself hosted on EC2) or commercial managed solutions like Cloudant for CouchDB and MongoHQ for a managed MongoDB service.

I like to manage my own and customer deployments on EC2 - frankly, it is fun :-)

That said, I think that there are sometimes business reasons for using hosted solutions like Heroku, Cloudant, and MongoHQ. It is a balance between development and admin costs and paying for managed platform as a service offerings.

nice: Rubymine 2.0 released

2009-11-17T10:24:00.003-07:00

I use Rubymine for most of my Ruby/Rails/Sinatra development on Ubuntu, and use it in conjunction with TextMate on OS X. I find it convenient enough to alternate between TextMate when I don't need IDE features, and Rubymine when I do.

One of the biggest improvements is that indexing now occurs in the background and auto-complete and other features become available that depend on knowledge of an application and the gems that it uses.

This is subjective, but once Rubymine 2.0 loads up and is done with any background indexing then the CPU use is minimal, and I think improved from earlier versions (nice to not have the fan kick in on my laptop when the CPU cores heat up). For the Rails application that I am coding on right now, Rubymine is using about 360MB of resident memory - this is OK with me.

MongoDB has good support for indexing and search, including prefix matching for AJAX completion lists

2009-11-11T10:58:00.003-07:00

I have been spoiled by great support for indexing and search in relational databases (e.g., Sphinx, native search in PostgreSQL and MySQL, etc.)

I was pleased to discover, after a little bit of hacking this morning, how easy it is to do indexing and search using the MongoDB document-centered database. I have two common use cases for search, and MongoDB seems to handle both of them fairly well:

Search for words inside of text fields
Efficient word prefix search to support AJAX "suggest" style lists

My approach does require combining search results for multiple search terms in application code, but that is OK. Assuming the use of MongoRecord, here is a code snippet:

class Recipe < MongoRecord::Base
  collection_name :recipes
  fields :name, :directions, :words
  def to_s
    "recipe: #{name} directions: #{directions[0..20]}..."
  end
  def Recipe.make collection, name, directions
    collection.insert({:_id => Mongo::ObjectID.new, :name => name,
                       :directions => directions,
                       :words => (name + ' ' + directions).split.uniq})
  end
end
 
host = 'localhost'
port = Mongo::Connection::DEFAULT_PORT
MongoRecord::Base.connection = Mongo::Connection.new(host,port).db('mongorecord-test')

db = MongoRecord::Base.connection

coll = db.collection('recipes')
coll.remove({})

coll.create_index(:words, Mongo::ASCENDING)

Recipe.make coll, 'Rice Soup', 'Cook the rice, then add extra water to thin it out.'
Recipe.make coll, 'Cheese and Rice Crackers', 'Slice the cheese and layer on top of crackers.'

puts "\nSimple find"
puts Recipe.find_by_name(:name => 'Rice Soup').to_s

puts "\nFind recipe by regular expression (ignoring case) in array of words /water/i"
Recipe.find(:all, :conditions => {:words => /^water/i}).each { |row| puts row.to_s }

According to the MongoDB documentation, a regular expression match like /^water/i will use an index just as a relational database match in the form like 'water%' does.

I am still in a learning mode with MongoDB, so I would appreciate any comments on improving this aproach.