Mark Watson's opinions on Java, Ruby, Lisp, AI, and the Semantic Web

This is a prediction made from a gut feeling and my own experiences: I think that in time, most jobs are going to involve building custom systems using mostly open source projects. In many cases off the shelf products are a poor fit to a business's work flow. On the other hand, building proprietary systems from the ground up is expensive and requires long development times.

Whenever a customer talks about a new system they need, the second step (after understanding the problem they need to solve) is always to identify quality open source projects to use. I believe that the effective developer of the future will:

• Need to know about and perhaps have experience with many open source projects
• Be skilled at reading code and understanding APIs
• Have strong architecture skills for creating new systems with custom code and existing projects
• Know when it makes business sense to turn back improvements to open source developers, even if the license does not require this
• Be a mentor to management on the trade offs of different forms of software licensing and different open source strategies

While there will always be jobs for computer scientists who break new ground and develop fundamentally new paradigms and technologies, most development work is in customized applications for organizations.

Labels: business, IT

Ten years ago I had a long email conversation with someone who was blind. He had bought one of my books and was using one of the example programs to write a better screen scraper for web sites than the one he was currently using. This made me realize how important it is to test our web sites so that everyone has access to them. I periodically use the text only web browser Lynx to test my sites. BTW, if you are in a hurry looking for real information on the web, you might try Lynx: sometimes plain text is better and faster, if sites are well designed: keyboard-only web browsing :-)

Spend a little time getting used to Lynx, and you may like it as a fast alternative for web browsing.

Ideally, XHTML or HTML is used for web content, with visual styles added with CSS. Javascript should be optional. It is very cool that using good web writing style also helps make sites more accessible for people with impaired vision. By the way, note that Blogger.com does a very good job of generating HTML that is easily viewed and navigated with a text only browser like Lynx. Try visiting this blog using Lynx.

Labels: CSS, IT

I had an "eye opening" experience last December: two sudden pulmonary embolisms that I could have died from. I am well on my way to a full recovery now, but this experience made me think about both my responsibilities to my family (e.g., making sure my wife knows where our financial records are, etc.) and to any customers who I sell commercial software products to.

Think about this: if you buy software from an individual or small company, what happens if that company or individual goes away? For small programs this might not be a problem but the situation is different if your IT infrastructure uses 3rd party software. Customers really should have source code, even for non-open source projects.

I am working on software licenses for my own projects that protect customers yet offer some revenue protection for me.

Labels: IT

$50/year per employee using the system seems like a great price point. Here in Arizona, the university system has transitioned to using GMail - at a great cost savings. Since Writely does a good job of importing Microsoft Word documents, transitioning most staff off of Microsoft Office seems possible for many organizations. Add in the benefit of shared calendars and GMail, and this seems like a slam dunk decision - at least it would be for me if I had employees in my business.

Often "less is more" when trying to maximize productivity. I often think that too many features in 'productivity' software actually slow people down when the core process involves thinking about a problem, finding solutions, and communicating with customers and co-workers in a simple and efficient way.

Labels: IT

I saw this in an Arstechnica article. I have used OpenOffice.org to write 3 of my last 4 books. OpenOffice.org does everything that I need for a variety of writing tasks (although I also like to use Latex) so I doubt that it lacks features that many businesses need. The integration of Microsoft Office with services like SharePoint may be compelling for some businesses, but even so, I would expect many businesses to look at the compelling cost savings with OpenOffice.org - free! Most people learn to use OpenOffice.org very quickly so re-training business costs should be low. Even businesses that want to stick with Microsoft Windows as their standard desktop should seriously look at saving some real money and switch from Microsoft Office.

I also believe that Microsoft's new Open XML format is flawed because of backward compatibility with many old Microsoft products and possibly with binary attachments that are difficult to process in 3rd party software. I have watched Microsoft's business model of customer lock in and standards non-compliance for too many years to not have natural suspicions about Open XML. The standards specification for Open XML is now 6,000 pages - a high level of complexity and a barrier for use by 3rd party developers until there are robust free libraries for processing Open XML.

Labels: IT, technical writing

Good news that

Minnesota and Texas may become the next states to adopt the OpenDocument Format as the required standard for their agencies, thanks to two state bills up for vote.

I expect Microsoft to apply aggressive lobbying to prevent this, but there are some polititions who try to act in the public interest so there is some hope that the proposed legislation will pass.

Both Sun and Microsoft are supplying OpenDocument translators for Microsoft Office - a good thing! Some people like Office and I believe in personal choice :-) BTW, If your document repository consists of Office documents, consider periodically using OpenOffice.org's batch option for converting a directory (and sub directories) of Office documents to ODF. I would strongly recommend doing this periodically and archiving the ODF documents with your Office documents: future-proof your access to our own work.

Labels: IT, technical writing

Several years ago, the Microsoft .Net product manager and his family were house guests at the same home where Carol and I spent a weekend. One of our longer topics of conversation (in addition to the Linux teeshirt that I quickly changed into :-) was virtualization and how it would change the deployment landscape.

Let me introduce my own prejudices: Virtualization is simpler when using open source operating systems and software - avoids license hassles. That said, regardless of which operating systems that you use, virtualization is a great tool for reducing the number of "physical" servers required to run older software. I used to lease very inexpensive dedicated Linux servers and configure and manage them myself. I now lease semi-managed Xen hypervisor based "virtual" servers - with operating system and infrastructure software choices selected for need. The time savings has been substantial for me and performance is both good and can be easily changed by upgrading/downgrading service plans. Service providers can inexpensively offer backup, automatic restore, and automated migration services.

Using Virtualization in your own data centers allows you to maintain old operating system configurations for legacy software systems, and migrate services between servers as needed - all at reduced cost.

Labels: IT

This is not a prediction for the 2007 New Year, but I believe that in the next 5 to 10 years Microsoft's influence on the computer/IT industry will be very much diminished. The transition to open source and open data formats has already started in Europe and developing countries looking to save money and is picking up some momentum in the US.

The new DRM features of Vista will turn off many users who are more interested in using their computers for information technology than for watching movies and listening to music. Any security advantages of DRM will probably be countered by more problems with interoperability with third party software and other operating systems (dual boot!).

Some pressure for the transition from Windows to open source will be the fear of some foreign governments of security issues with a closed operating system. Several years ago Microsoft had to provide Windows source code to the Chinese government.

Microsoft has been very effective in fighting back against open source using threats of patent infringement and "influencing" government officials. All large corporations "influence" government officials, so I am not particularly picking on Microsoft here. I also acknowledge that Microsoft has a responsibility to their stockholders to make a profit. BTW, I sold my Microsoft stock in 2005 because I was unhappy with their pushing proprietary document formats - not a financial decision.

I believe that in the next five years that more local and national governments will either use more open source software or completely switch over to open source. I expect this to be a slow and steady process. I also believe that organizations not using open source software will eventually find themselves with both economic and technical disadvantages.

Labels: IT, Linux, Microsoft

Since the Novell/Microsoft deal, more of us worry about Microsoft legal attacks on Linux using some of their (unnamed) patents. This is troublesome to me for at least two reasons: Linux and other open source software allow me to build systems for customers at a lower cost point (good for my business), and I enjoy developing on a Linux box for Linux deployments.

For Linux on the desktop, I believe (but I am not a lawyer) that as long as I (and other Linux users) typically buy a commodity PC with Windows and then install dual boot Linux, then how could I possibly be infringing on Microsoft patents: I have a paid up license for Windows on the same PC that I am running Linux on. The situation for server side Linux is less clear, but server side Windows is much less important to Microsoft's cash flow than the desktop.

Accurate predictions are difficult, but I would not be surprised if through corruption of politicians some countries put up legal roadblocks to using free (as in GPL) and other open source software. Long term, I believe that these countries will be at a competitive disadvantage over other countries where businesses of all sizes and individuals can continue to lower IT costs while benefiting from collective software development.

It is really simple: diversity of commercial and open source software, with fair competition is good for business. As a reluctant Microsoft customer, I would ask Microsoft to publicize which of their patents that they think Linux violates - provide real details. Of course they will not do this because I believe that their strategy is to threaten but to not take action. Computer professionals can push back against Microsoft by favoring companies like IBM that take a more fair and balanced approach to commercial and open source software. We can all help by making small contributions to organizations like FSF and the EFF.

Labels: IT, Linux, Microsoft

I have blogged several times on how much I enjoy Apple's iTunes store, always burning any songs I buy to a CDR as MP3 as backup. I just did get burned by Apple's DRM however. This was partially carelessness on my part and partially some bad luck with hardware. You can only "authorize" 5 computers at once to work with your iTunes store account. I had a disk go bad on a Mac and had to re-install OS X without first unregistering the computer with the iTunes store. I had a similar experience recently with a Windows laptop: re-installed Windows without remembering to de-authorize that computer with the iTunes music store. Now I can't watch several Battle Star Galactica videos I purchased on my laptop. Oh well, no great loss.

(Un)Trusted Computing is a bigger deal: if I buy a computer, I want control over:

• What software gets installed on my computer
• When I buy a license to run an operating system, like Windows, I want to be trusted as a customer, and not have my computer (partially) disabled if Microsoft makes an error

I am currently running Windows 2000 on my laptop, and I am happy enough: I still can get security updates, my system is stable, and I can run Windows-only VPN software that my customer provides me. My experience with Windows XP was less happy, but still reasonable. While Microsoft Vista's (Un)Trusted Computing might appeal to unsophisticated computer users, Vista does not appeal to me, and I am hoping to permanently avoid Vista.

Richard Stallman and Eben Moglen of the FSF.org have been warning about loss of freedoms for computer users for many years, and unfortunately some of their dire predictions seem to be coming true.

What I think is important is that people who want Microsoft's Vista should have the right to Use Windows. Same with Apple users. People who want to use these proprietary systems will certainly be able to get "locked in" all they want.

The problem I have is for the 1% to 2% of computer users (like me!) who appreciate the advantages of running Linux, even without having access to media like DVDs, etc. I want to be "left alone", and be free to set up my Linux computer as I like it. In regards to patent issues: I see great hope because large companies like IBM are sharing part of their protective patent "umbrella" with the Linux community. Good going IBM - I will certainly give them my business when I can. Also, I hope that the courts take a fair approach to enforcing patents: patent holders like Microsoft who might want to legally attack Linux users should be made to publish which patent violations Linux is supposedly guilty of, and I believe that the Linux community will quickly get rid of any legally offending code if any such code exists in Linux.

Labels: IT, media, open source

Backing up my user files under Linux is easy, if everything in /home/mark fits on a DVD-R. A Linux re-install is quick and then overwrite /home/mark. I usually install applications that I build from source in my ~/bin directory and Ubuntu's package manager makes it quick enough to restore other required apps (PostgreSQL, etc.)

The Mac is not so simple because installed applications store frameworks in the system library, etc. Still, it it easy to make a reliable backup, assuming that you have an external FireWire (or I suppose an USB) disk drive:

• Make sure that there is room on your external disk for your home directory and another 5 gigs
• Plug in your external backup disk, and restart with your install DVD
• Install OS X, choosing the external drive
• When the installer asks if you want to import previous user data, say yes and select your internal disk that contains your system and user account
• Reboot using your external disk and run software update
• Pack away your external disk as a backup of your working environment - preferably off site for extra protection.

If you run this procedure while you are reading or watching TV, this is a low overhead operation - cumulatively just takes about 10 minutes of time (about 1 hour wall clock time). An external disk can obviously also contain other backups of large video files, etc. in addition to your OS X and user setups.

Nothing is for sure in life, but external disks are likely to have longer shelf life for your digital assets than CDR-R and DVD-R media, but using both, and off site backup on remote servers, etc. is also a good idea.

Because of the (hopefully) very high reliability of external FireWire (or USB) disk drives, I also copy important Windows and Linux backup files to an external disk using either my LAN or DVD-R backups. Replicating really important files to multiple computers on your network using rsync is also a good idea.

Labels: IT

I like this :-) I added the underlined emphasis:

Now that the old days are long gone and word processors come preinstalled with every machine, why should we care about this remnant of history? The answer is that remarkably LaTeX is much better suited for composing and distributing most types of documents than any other modern word processor on the market that I am aware of. Just like programming languages tend to converge towards Lisp because it got things right the first time around, so do the Word Processors tend to converge towards LaTeX.

In the last few months almost all of my writing has been done using LaTex and most of my development in Common Lisp (using another old program: Emacs). I love to experiment with new technology, but I use whatever is best for a job to support my family.

I blogged a few years ago about this: in the distant future, I wonder if people will be using ancient software that has been thoroughly tested over the centuries, is bug free, and seems 'just right' feature-wise.

Labels: IT, Latex, Linux

The Subtext programming system has been getting some buzz, and I think that this is worth while if it makes us think about the disconnect between thinking about problems and writing software to solve these problems.

While state of the art IDEs like IntelliJ for Java and VisualStudio for .Net languages provide a comfortable working environment, I must say that both Java and the .Net languages are poor choices for many programming tasks.

Scripting languages like Ruby and Python help this thinking vs. programming disconnect in one important way: for small programming tasks, very short programs are sufficient and we can keep track of both problem task thinking and programming.

What about large projects? There are two good alternatives in programming languages: Common Lisp and Smalltalk:

Common Lisp lends itself really well to growing your own application specific language (using macros if you like, and functions). Once you build up an application specific language, a lot of the complexity of even complex programs goes away. Even more importantly, domain specific languages should help close the gap between thinking about problem solutions and programming these solutions.

The downside of Common Lisp is that while Emacs based IDEs are effective environments, even with add on code browsers, I find exploring large Common Lisp software projects to be tedious.

Smalltalk implementations generally have great code browsers because the simplicity and regularity of the language make it easier to automatically process the structure and semantics of code. Smalltalk blocks and closures, like in Ruby, allow many concise coding tricks - shorter programs are easier to understand and modify.

Labels: IT, knowledge management, Lisp, Smalltalk

I had to laugh a bit this morning while enjoying my coffee: I was reading responses on Slashdot to Tom Yager's optimistic article on Apple's market share growth. While I am a Mac fan (I am writing this blog on a Mac and I have been writing Common Lisp code on a Mac since 5:30 this morning), I think that so many people miss the point of operating system dominance in the future:

Step outside of pure technology for a minute, and think about the global economy and were the buzz is right now: the US is educating a very small fraction of engineers and scientists as places like China, India, Europe, etc. The largest growth will be in what are now poorer countries, but expect a more level playing field in the future. There is a good reason why high technology companies are increasingly setting up research labs outside of the US: less expense and a good supply of highly motivated and educated talent.

Using Linux in developing countries makes the most sense: computer science students get free access to source code and end users can run on much cheaper hardware using free software. As more creative work is done outside of the US, scientists and engineers will naturally use what they have used in school: Linux.

Long term, I expect to see an upswing in technology in the US: with the amount of perceived (fake) wealth in the US (from people pulling $400 billion a year in equity from their homes for a higher lifestyle, etc.) many young people simply lack the motivation to work at science and engineering in school when they see lucrative careers in real estate and other fields that require training but not too much education. Who knows when the crash in the US economy will come, but when it does occur it is likely to make the "comfortable" recession of 2001 look like a picnic. Usually success requires education and hard work, and in the future when we in the US are climbing out of what I think will be a very long economic downturn, I also believe that poorer economic conditions in our country will motivate both student and workers. During the upcoming economic slump in the US I would not be surprised if Linux becomes more popular with end users for the same reasons that Linux is gaining in popularity in developing countries right now: families trying to hold on to their homes and meeting other mandatory expenses might look favorably at $200 Linux PCs.

Labels: economy, IT

I really enjoyed listening to Tim Bray's talk on developing the ATOM specification on ITConversations. He made a lot of interesting points, but the one that resonated most was ATOM's requirement for a globally unique identifier for every feed and entry. With more syndication, we all see lots of duplicate material. Examples of duplication can readily be seen on rojo.com (used to be my customer, and I still enjoy their site a lot) and technorati.com: we end up with many URIs that refer to the same material.

It is possible to write software that detects duplicate feeds, but comparing two articles is not an inexpensive operation, and when comparing a very large number of feeds, the O(N^2) runtime is painful. I have experimented with much a less accurate algorithm: hash NGRAMs of articles and check for duplication using a hash lookup. I have found that this gives poor results - at least in my experiments. If you do partial matching of NGRAMs, you are back to O(N^2). (If anyone knows a good way to handle this, let me know :-)

Globally unique identifiers help solve many duplication problems, makes it easier to implement container relations, and in general ATOM just seems to be a better and more scalable platform than RSS 2.0 for complex new applications.

Labels: IT, knowledge management

The Sapir-Whorf hypothesis poses that our native language affects how we think. Computer programming languages also strongly affect how we think about, design, implement, and maintain code.

I was working on some tricky code this morning that builds on some Common Lisp CLOS class libraries. The new code is really orthogonal to the existing functionality and it seemed like a poor idea to merge the new in with the old, especially since the old codebase will probably be used as-is for a while. I decided to start a new module (as defined by physical file organization) that added the new functionality to the existing classes as generic methods. The new module stays small, and anyone needing to use the original codebase is not confused with extra code for functionality that they do not need.

In Ruby, I like to do the same sort of thing: have different modules (as defined by physical file organization) where new orthogonal functionality is added by defining new methods to existing classes in new file modules.

In Java (and other languages), you can always use Aspect-oriented programming (AOP) to add new orthogonal behavior to class libraries, but, to be honest, I dislike AOP - this is not the fault of AOP per se, but because I have only used AOP with Java.

Labels: AI, IT, Lisp

Am I the only consultant who has to spend a lot of time working out details for consulting contracts? I have a policy of never signing anything that I don't totally understand and agree with. I try to get customers to just use my standard NDA and consulting agreement but it is too simple for some tastes. The worse case scenario for me is that someone wants me to do a small job (perhaps 4 or 5 hours of work), we agree on the telephone what I am supposed to do, then they email me their "standard contract" that is many pages long, difficult for a non-lawyer to understand, and usually contains many terms that just don't seem required to me for a job where my customer and I are not sharing any real intellectual property except for the software that I am writing for them and that they will own when I am paid. Some of the worse terms, from my perspective, call for my divulging everything that I am working on (can't do this because I am NDAed with other customers) or say that I can not work on any similar projects for a specified period of time (OK if the work in question is a long term project but not OK for small one day jobs).

Sometimes long contracts are required. Long term engagements can require more complicated agreements because more IP is shared and a consultant might need access to a customer's software and/or servers, etc. Anyway, I think that agreements should be as simple as possible but still protect both side's interests.

PS. While I was writing this, I paused, and decided to offer customers who use my NDA and contract a 12.5% discount. Anyway, I just updated my web site so it will be interesting how many people take me up on this offer over the long term.

Labels: business, IT