Mark Watson's opinions on Java, Ruby, Lisp, AI, and the Semantic Web

While I love writing, I also like to read other people's efforts. I find that I learn a lot reading code that other people write. I started seriously reading other people's code in the 1970s - a habit I never outgrew. When I read what other people write, in addition to the content, I also pay attention to their writing technique: how they introduce a topic, make points, provide examples, the level of detail they use, etc. Check out the new PragMag - good reading.

Actually, this is specific to ActiveRecord classes: create a virtual RDF graph starting with an ActiveRecord class and all associated classes. I use a Java project D2R that provides a SPARQL endpoint service to a relational database. I had thought of wrapping D2R for use in JRuby, but I would rather have something that is pure Ruby and as light weight as possible. The implementation of D2R is fairly complex but a Ruby/ActiveRecord implementation should be much simpler (wishful thinking!)

I'll release it under an open source (free software) license when I have something worth sharing.

Labels: ActiveRecord, RDF, Ruby, Ruby Rails, semantic web

APress web page for my book.

And, an Amazon link..

There is a lot of challenging material in this book but I am hoping to save my readers some setup effort because they can use a pre-configured Amazon machine image (AMI) that I created with the book examples and required infrastructure software. Of course, you can also build the examples on your laptop or servers.

I am setting up a book support web page with errata information. I plan on making entries for most questions from my readers that I answer via email.

With all of the problems that my country is facing, I am still optimistic, if:

Parents do their job and turn off the TV after dinner. When I was in high school I did 2 to 3 hours a night of homework on week nights - that should, I think, be the norm for the new young generation.

Young people do their job and squeeze every bit of value from the educational opportunities that they have at their disposal.

Adults do their job and realize that education and job skills are something that they need to develop continually throughout their working lives. Be productive and prosper.

Congress and our president: suck it up, stop being bought off, and do what is right. Look out for your karma, try not not be total assholes.

Financial elite: realize that no matter how much money you accrue, your children and grandchildren need to live in this world so you should not ruin the world that they will need to live in. Suck it up and try doing the right thing for change. Look out for your karma, try not not be total assholes.

As a country, we need to work together and everyone do their part.

Labels: economy, politics

Here in the USA, we face a tough choice: to survive with any kind of lifestyle and robustness, we need to defund government spending on:

• Huge government subsidies to the Insurance companies, Beef Industry, etc. Subsidies may also take the form of not collecting a fair tax burden and for "under-regulating" corporations that strongly act against the public interest.
• Drastic curtailment of defense spending
• Elimination of the large amounts of money we give/loan to other countries to buy weapon systems from companies in the USA

Why are these tough choices? Well, because Congress (and the executive branch under Bush and Obama) mostly look out for corporate interests and not citizens' interests - that is just the way it is. Also a tough choice because almost everyone is simply too lazy to spend the effort to personally lobby their elected representatives.

Perhaps people get the government that they deserve.

Labels: economy, politics

I wrote an article for DevX on Java AppEngine that was just published. The example application implements simple document storage and search. I still very much like AppEngine, but most of my work right now involves Rails development. I am still waiting for a customer who needs AppEngine development :-)

My wife has been helping a local non-profit organization (Connections) that uses Equine (horse) therapy to help both children and adults with disabilities - good work. I put her latest video on Youtube if you are interested.

I continue to be very happy with Heroku - for a relatively small cost, my customer gets a good deployment platform that takes almost none of my time to use, thus saving them money. When it comes to deploying software, I do like control -- so, it seems strange that I am so happy with Heroku and AppEngine where you have to give up control in return for saving time and money.

Between 2003-2005 I often used Swi-Prolog for semantic web experiments before I more or less settled on using Sesame (and occasionally AllegroGraph and Redland).

I just saw a link to the ClioPatria semantic search web-server project. Assuming that you have a fairly recent copy of Swi-Prolog installed, trying ClioPatria only takes a few minutes:

git clone git://eculture.cs.vu.nl/home/git/eculture/ClioPatria.git
cd ClioPatria/project-iswc
./configure
/run.pl

Then, point your browser to http://localhost:3020/, load, for example, a OWL (XML format) data file, and try some queries. The default query language is SeRQL which I don't use, so I set the query language to SPARQL and all seems to work fine.

One good thing about Swi-Prolog and the bundled SemWeb library is that loading RDF data and performing queries seems very quick compared to Sesame which is what I usually use. As a result, the ClioPatria web application is also very quick.

Unite is an interesting idea, letting non-techies easily share materials with friends and relatives. Opera provides an intermediary service: they map your account to whatever temporary IP address you have, deals with NATs, acting as a proxy between the web server running on your laptop and friends' and family members' browsers.

While I have been getting very interested in cloud computing in the last 6 months (using Amazon's and Google's offerings, and I spent an hour with someone at Sun yesterday giving them usability feedback on their soon to be released cloud services web interface), I am also interested in peer to peer systems. With some inter mediation, Unite is sort-of peer to peer (except for the reliance on an intermediate proxy service, although in some circumstances UPnP is used and no proxy is required). It is fine that Opera provides the back end services for Unite inter mediation. I would also like to see open source implementations (could be as simple as a little proxy server written in a scripting language and a Firefox plugin - if there is not already something like this, I expect to see some projects soon). For example, I might want to run my own intermediary service for friends and family. Anyway, ignoring the hype, I think that Unite is a good and also interesting idea.

I just did a test deployment for a customer on Heroku.com this morning. Lots of non-standard things in the web app, but it still deployed nicely after a short remote SMTP setup. I have been reading about Heroku's architecture, implementation on EC2, etc. for a long while, so getting to use Heroku was fun!

For general development and flexibility I still like a semi-managed VPS (RimuHosting.com is the best I have found so far) because I can run a mixture of Java, Rails, Squeak+Seaside, etc. and have a home for master git and svn repositories. That said, for deployment, a custom deployment architecture on EC2, or an abstract scalable platform like AppEngine or Heroku really does make a lot of sense. I enjoyed talking to two principles of Engine Yard (Ezra Zygmuntowicz and Yehuda Katz) at Merb Camp last year but I have not yet had an opportunity to use their platform on a customer job.

My good friend Tom Munnecke (he took the great picture on my web site of my wife and I in front of the Taj Mahal) recorded an interesting interview last month with Peter Norvig who talked a lot about the Google's translation services and how they work (Tom has not posted his video interview yet - I will add a link when one is available).

Anyway, it is a real pleasure this morning to actually get to experiment with the Translator Toolkit - very impressive. I have fairly good reading knowledge of French so I am experimenting with translating English to French. My Spanish is very rusty (and my FORTRAN is rustier still :-) but I will try that also.

I noticed that Microsoft allows free use of their search and spelling correction APIs. I just played with the APIs for a few minutes. Here is a Ruby code snippet that I just wrote:

API_KEY = ENV['BING_API_KEY']

require 'rubygems' # needed for Ruby 1.8.x
require 'simple_http'
require 'json'

def search query
  uri = "http://api.search.live.net/json.aspx?AppId=#{API_KEY}&Market=en-US&Query=#{CGI.escape(query)}&Sources=web+spell&Web.Count=4"
  JSON.parse(SimpleHttp.get(uri))["SearchResponse"]["Web"]["Results"]
end

def correct_spelling text
  uri = "http://api.search.live.net/json.aspx?AppId=#{API_KEY}&Market=en-US&Query=#{CGI.escape(text)}&Sources=web+spell&Web.Count=1"
  JSON.parse(SimpleHttp.get(uri))["SearchResponse"]["Spell"]["Results"][0]["Value"]
end

You need a free Bing API key - notice that I set the key value in my environment. If you get a key, then try:

search "semantic web java ruby lisp"
correct_spelling "semaantic web jaava ruby lisp"

The first method does spelling correction before search.

Labels: Ruby, search

As a Linux user since the early 1990s (and a longtime OS X user), it was easy for me to get in the "./configure; make; sudo make install" habit, but I don't think that this is such a good idea for two reasons:

• Security: have you really read the source code to see what might be executed during "sudo make install"? I am constantly installing Ruby gems, infrastructure software, etc. and I often read code as an educational experience, but not for security. It is best to not run other peoples code as root.
• It is much easier for me to rebuild systems from backups when I "./configure --prefix=/home/mark/bin" (or wherever, but in my home directory).

I used to like to keep my home directory fairly small so backups take up less space but now costs of external disks, remote storage like S3, etc. are so small, that it makes more sense to have my home directory to be more self contained.

I also like to develop customer projects under a single master directory. It is nice to have everything in one place: my application code, nginx, PostgreSQL (with data), Ruby, gems, Java, Tomcat, Sesame, Erlang, CouchDB, etc. - whatever a project requires to run. A top level shell script can set up the environment for each different project. This also makes cloning a customer's system to one of their alternative servers just a quick rsync away...

Very nice - even the betas that were just released are very fast. Interesting to see how fast the final releases will be...

I had not tried Chrome before because of WAS (Windows Avoidance Syndrome). I like the minimalist Chrome UI - very nice. When I boot OS X, I also like to use the new Safari 4 beta.

I have been very busy this year - not too much free time to try new things. Now that I am done working on my new book, I hope to have time to experiment with some technologies that I have not tried before like writing a web application that optionally uses Google Gears for local storage.

I just made a small donation to Canonical (good shepard for Ubuntu Linux) while I was installing some security updates. A good investment.

As a few very large corporations continue to control resources and major infrastructure, I expect to see a trend towards small agile enterprises covering rapidly changing technology and business niches. I expect to see a three-way synergy between mega-size corporations, small agile businesses, and a mobile highly educated work force: all three sides win big. The losers in this new world order are the poor and the poorly educated workers who can not adapt to changing situations.

I think that open source software, other key infrastructure supported by the users of the infrastructure, and a general gift economy will continue to reduce down to a minimum the cost of doing business. Again, the winners are both people who are well educated and prepared on a global scale to move quickly to take advantage of new situations, or people who prepare themselves for work in high-value local jobs like health care, critical government services (fire control, police, etc.), and support of local physical infrastructure.

I believe that both my country (USA) and most of the world are going through a historic transformation created mostly by a new higher level of transparency. Throughout history, the ultra rich and powerful have worked behind the scenes to amass more power and wealth by starting wars, etc. The same things still happen, but now society better understands what is really happening: corporate ownership of governments, who benefits financially from planned wars, strife, and the manipulation of the world's financial infrastructure.

I believe that the "information genie" is out of the bottle, and is not going back in.

Predicting the future is tricky, and I will not try. Still, it will be interesting to see how the general quality of world-scale governance improves or degrades in a future that blends meritocracy (those who get great educations and are major producers will compete with conventional multi-generational dynasties of controllers), greater transparency mostly due to the Internet, and near absolute corporate control and manipulation of governments and news media.

Ideally, in the new world order governments become "infrastructure" in the sense that governments compete to provide:

• Highest quality of physical infrastructure services for the lowest tax base
• Highest quality of resources for education (which needs to cover people throughout their entire lives)
• Physical security for people living in and businesses in their jurisdictions, at the lowest cost
• etc.

We all live in interesting times :-)

Labels: economy, politics

I have been so busy this year that I have slacked off on fully getting up to speed on Scala (and Haskell, for that mater). A few people have been working on a Sinatra clone in Scala. (Sinatra is a very nice minimalist Ruby web framework that I like a lot.) I grabbed a version of the Step project off the web today and I just had some time to play with it.

You could not really write something like Step in Java without writing an interpreter. With Scala, you can add syntactic sugar similarly to Ruby. Step nicely emulates parts of Sinatra; here is a code block wrapped to handle an HTTP GET request mapped to the route "/form":

  get("/form") {
    Template.page("Step: Form Post Example",
    <form action='/post' method='POST'>
      Your name: <input name='name' type='text'/>
      <input type='submit'/>
    </form>
    <pre>Route: /form</pre>)
  }

With Scala's ability to define operators and deal with code blocks, and native XML support, Step was defined in less than 100 lines of Scala code. Very nice.

Yes, I should have gone to the Google I/O conference: I would have a sandbox Wave developers account right now. Oh well. It almost gives me a headache thinking about the server resources that will be required to support a world wide deployment and large scale adoption of applications built on Wave, end users of the basic Wave platform, etc. That said, I don't have to worry about how Google implements AppEngine, Wave, etc., and the details of how Amazon implements AWS. That is the point:

Cloud resources place an abstraction barrier between developers and deployment concerns.

As someone who actually enjoys dealing with server deployment issues, this is still a very good thing. Anything that lowers cost and makes development faster is a good thing, even when we have to leave doing some fun work behind us.

I like that there are already two other companies besides Google that are implementing Wave protocols and services (I want access to that Emacs Wave client :-) Open source implementations and multiple service vendors make the platform a safer technology choice. I liked the bit in the demo video on privacy and security (related: see Amazon's white paper on dealing with HIPAA privacy laws in EC2 hosted systems).

I'll obviously wait until I get a Wave developer's account and spend a 100 hours (or so) to kick the tires (some good documentation is already becoming available), but I might decide to write a short book on the Wave platform. I usually just write about things that I already use in my work, but since using resources like Amazon EC2, AppEngine, and Wave will be so important to my work and businesses in the future, time spent on Wave should be a good investment. BTW, I am working on a DevX article right now on the Google Java AppEngine. My APress book Scripting Intelligence: Web 3.0 Information, Gathering and Processing (should be in print in about 4 weeks) has some material on Amazon's EC2, S3, and Elastic MapReduce.

I am glad to see this because it will encourage more web developers to add semantic markup. The last appendix in my new book (APress: "Intelligent Scripting for Web 3.0) briefly covers RDFa. In principle I prefer publishing separate RDF feeds but with support in Drupal for RDFa (and, I hope, other CMS systems like Plone) RDFa may become commonly used - a good thing if it happens.

I continue to be happy with my decision to set up a "work only" Ubuntu Linux partition on my MacBook. I have been using Linux since I downloaded a mini-Slackware distro in 1992 over a 2400 baud modem. Desktop Linux has come a long way! I did a few things to make my working environment nicer:

• Set the CAPS LOCK key to another control key
• Installed Google Desktop and set the instant search popup hot keys to "Hyper space" (command space, same as popup search on OS X)
• Set up IntelliJ/RubyMine/Eclipse for Ruby, Java, Scala, and App Engine development

I did not even install the developer tools on my new OS X partition. I am just using OS X for video editing, photos, and video conferencing with my family.

I made three image backups of the hard disk on my MacBook (yeah, I am careful like that) and just used BootCamp to set up a small OS X partition and an Ubuntu 9.04 partition. This worked for me, but I make no guarantees for you (<grin>):

• Boot with OS X install DVD, wipe the disk, and installed to the entire disk
• Without doing anything else, I ran BootCamp and grabbed about half the disk for "Windows"
• I inserted a Ubuntu install CDR, and chose "advanced" partitioning. I deleted the fat partition on /dev/sd3 and made it a bootable ext3 partition with "/" as the mount point
• When I boot my Mac, I hold down the option key to switch between OS X and "Windows" (ha!).
• This is really non-optimal, but I had a false start: the first time I ran the Ubuntu installer, it recognized OS X and offered an express install. Anyway, I ended up with a small 2 gig partition /dev/sda4 that BootCamp boots the first Ubuntu install. I simply edited the /boot/grub/menu.lst file to set /dev/sd3 as the boot disk.
• Now I can "option key choose my OS
• Now that both operating systems work fine, I took the time to install all security updates for both operating systems

I would have liked to go through these steps again, leaving out the first false-step Ubuntu install, but I need to install my dev tools on Ubuntu and get back to work.

I bought version 1.0 when it was released this week and since then I have stopped using TextMate (or occasionally NetBeans) for Rails development. The things that won me over: very fast interface, auto-complete, command-B to jump to method declarations, command-F12 for a popup mehtod list for the current file, pretty much full refactoring support, jumping between view and controller action code, etc. I still think that TextMate is much better for browsing large projects, but if you are coding, I think that RubyMine is much more productive.

Although this article on Google and cloud computing sounds like a bit of an advertisement (it is!) it is also a good read. There is no way that individuals and all but a few companies can compete with Google on cost per computing unit (CPU, memory, data storage).

I am a huge fan of Amazon's EC2 services: easy to use and very flexible. I am not sure how well this will work, but I want to try using Amazon's Elastic MapReduce to produce data for a Java of App Engine application that I hope to have time to write in the next month or two (MyThingsOfInterest.com, that I mentioned in my last blog). Amazon charges for bandwidth reading from S3 and there will be a cost pushing data into Google's App Engine data store. It may well not make sense, cost wise, to split a system between two competing platforms.

I just sent in Chapter 15 for "Scripting Intelligence for Web 3.0" to Apress yesterday. Now I just have to review some final edits. This has been a very fun book to write, and I must admit some sadness over finishing this project and having to move on. I cover a wide range of topics: text processing (NLP), Semantic Web and linked data, strategies for deploying Semantic Web applications, several strategies for implementing search, "scaling up", use of Hadoop for distributed data crunching (with material on using Amazon's Elastic MapReduce), etc.

One of the examples in my book is a rewrite of something that I have been playing around with for years using Common Lisp with WebActions and Portable AllegroServe: a personal system for keeping track of things of interest. I took years of sporadic Lisp hacking, took some of the best ideas, and ended up with a concise Rails application. I am thinking of writing a third iteration of this and making it public: I have a placeholder web app MyThingsOfInterest.com running on the Java edition of App Engine. I need to spend a few weeks clearing a backlog of customer work and then I want to take a crack at version 3. In the last couple of years, I have been doing a lot of Rails web UI programming using the Rails AJAX helpers and some custom Javascript. I don't have very much experience with GWT so working on MyThingsOfInterest.com will be a good excuse to study GWT and compare it with using Rails. The App Engine is an interesting platform and I think it is a fun challenge working within a software stack that limits developer options in return for high efficiency and very low hosting costs. The first thing I need to work out is implementing local search on top of JDO and the non-relational data store. Using Lucene is not a possibility, but it should be fairly easy to support full word and prefix match search on top of JDO.

I was an early Mac enthusiast (I wrote a successful Mac application in 1984) and long before that I bought a very early Apple II (serial number 71) and I wrote the simple little chess program that Apple gave away on the demo cassette tape for the Apple II. Anyway, I am pretty much into Apple products. During the later "dark ages" before Apple released OS X, I did use Windows NT (and later Windows 2000) and Linux for work and play. During this time, I developed a great 'getting stuff done' strategy: I booted NT for a few customer projects that needed Windows and when I wanted to play - otherwise I booted into a very stripped Linux install that only had what I need installed for work spurts.

After I finish work on my new book for Apress (soon!, probably in the next 2 weeks :-) except for ongoing work for two customers, I want to concentrate on a new business venture that only requires a development setup for Ruby, Rails, and Java. I work almost exclusively on my Mac laptop, using my desktop Mac only video editing (huge amount of disk space and memory) and a local Linux box when I need to test networked applications. BTW, I am writing this on a new Ubuntu 9.04 installation - a nice 6 month upgrade from the last release.

This experiment may not last more than a few months, but I want to have a only small OS X partition on my laptop with fun stuff, and a larger Ubuntu partition with Java, Ruby, IntelliJ, RubyMine, and a minimal set of tools that I need. I tend to work in 2 to 3 hour spurts on both customer projects and my own stuff. I don't like to check email and I have my wife screen my telephone calls during the work spurts. You can quote me on this: multitasking is overrated!

My Mac laptop is a great do-everything system, but I think that having different "fun" and "work" environments helps get more productive work done in less time. As long as I am sharing some personal philosophy on work and life, I find that minimizing the time watching TV also helps make more time for friends and family, sports, enjoying nature, gardening, cooking, reading, etc. As a computer scientist, I am into performance analysis, and as a person, the same kind of performance analysis is good to evaluate the benefit of time spent on various activities.