Why don’t I like hadoop? I state for the record that I only have a passing familiarity with hadoop from reviewing the docs and some modest amount of administration. My opinions are mostly uninformed.

Hadoop includes a distributed filesystem… written in java… This sends up a big red flag in my head. Filesystems are an OS level concern in my experience. Writing a distributed filesystem in java with no OS hooks seems on its face to start with an inefficient model. It’s also not a general purpose filesystem, so the usual unix (or windows) tools for dealing with filesystems can’t be used on HDFS. Many current super computers use the Lustre filesystem. Lustre has been around for more then a decade and is a very mature product. I’m not suggesting that Lustre is right for hadoop, but the list of claimed features for both is pretty similar. Since Lustre uses a POSIX filesystem interface, I can use all of my unix tools on it.

Job scheduling is another well traveled path that Hadoop developers decided to walk again. The hadoop developers have arguments against MPI and PVM schemes which are used in most Beowulf clusters. Maybe the arguments are valid, but I don’t see a reason why one couldn’t build on such a mature technology to produce an adequate scheduler. In fact people have built built such a scheduler.

On the other hand, HBase and the actual MapReduce components are novel. My main problem with hadoop is that they decided to build components which are readily available elsewhere, are probably more efficiently written, are certainly more mature, and integrate well into the OS. One of the advantages of open source projects is that one can borrow from other good projects. I think the interesting parts of hadoop would have advanced farther if they had used the well studied portions of beowulf clusters and other technologies as a base instead of building them.

Once I got SpiderMonkey running, I plopped the js shell binary and the libjs.so library where the OS can reach them and started looking for a mod_js. I didn’t hold out much hope of finding one since I’ve looked for this in the past. Lo and behold though, someone had done some work in this area. It looks like active work stopped in mid 2007. There is a checkin which changed the license to a dual GPL/Apache license. I checked out the source and got it running after several more hours of work. I produced a patch to keep it from crashing on my Ubuntu system.

I ended up with a minimal mod_js which can handle files. I have a lot of work to make it really useful. Short term goals include setting up some system where tags set off the JavaScript from the content. PHP uses <? and ?> to delimit code. I’ll probably do something similar. There are the beginnings of SQLLite support in the mod_js code. I have no idea how complete the support is, but I’d like to get that running. I’d also like to add bindings for MySQL and PostgreSQL. If possible, I’ll try to build a set of modules which attach to the SpiderMonkey engine instead of attaching to mod_js. That way, the extended SpiderMonkey can be used for other projects easily.

As I move forward, I’d like to offer thanks to the folks at Mozilla for writting a nice C based JavaScript engine. I’d like to also thank the folks at modjs.org for writting something which functions and looks easy to extend. If either of the authors listed are around and happen to read this, I’d be happy to pass along any patches. If I don’t hear from them, I’ll build a repository and make it accessible. The folks at FreeSWITCH seem to have well written docs on the File object. I’ll probably look at their rendition of the SpiderMonkey engine and see if I can avoid some work.

In a future article, I’ll try to write up some docs for building SpiderMonkey and mod_js from a Ubuntu user’s perspective.

The next few articles will describe my adventures in trying to get a mod_perl like (though less ambitious) set of bindings for JavaScript. I’ll also be working on some missing bits of javascript functionality which will make it more useful on the server-side. I’m not the best programmer in the world, so I’ll use ugly hacks, borrowed code (attributed to its authors of course), and prayer to get it all working. I’ll be happy for any help coding or testing since I’m lazy.

If a company spends as much or more time fixing the problems caused by a release as they spent on the release, maybe some checks need to be put in place to cut down that wasted effort. The problem is finding a balance where the check costs less then this extra effort and pain.

Maybe unit tests help. People who do unit tests seem to have different opinions about how effective they are. Some say that it causes them to take twice as long to write the code. Others say it helps them write the code faster since they have a codified requirement.

Automated testing is another thing that can make the release process better. There is a large cost associated with this in many cases though. You need a highly qualified QA guy who knows how to program and knows the product to write the tests. If he’s that good, maybe his time is better spent helping build the product (if this is a small organization). Could the automated tests be written by the group who develops the software? Sure, but it will slow down “real” development work.

I think the real problem is that as complexity increases, the need for checks increases. Eventually, one gets to a point where progress is ponderously slow. What do you do about this? Well, all of the solutions that I know of are bad in some way. One has to make a choice between pain and searing pain.

• Keep it simple. Simple systems are easier to check and there is less risk in changing them since they’re easily understood.
• Keep it small. This is really a subset of simple. If you have 5 simple systems (like web servers), you’ll have an easier time managing them then if you have 100 simple systems.
• If you can’t keep it simple, only make it as complex as it has to be. This means managing customer expectations (which no one wants to do in my experience). Make sure the system isn’t more complex then your developers and operations staff can manage.
• Don’t let complexity creep up on you. Know that it’s coming and plan for it. Know that your costs will rise as the complexity rises. Find ways to make the cost rise as slowly as possible.
• If you have to have complexity, automate the heck out of everything you can find. This is easy to say, but if the system has grown so complex that all you do is fight fires, then you won’t have time to automate unless you are willing to accept a lower level of service for the time it takes to automate.

I am learning these things the hard way, so I don’t know of any elegant solutions to the problem of complexity which go beyond what I’ve already mentioned. Maybe after a few more startups I’ll have more useful advice then, “Watch out! You’re headed for a big bucket o’ misery!”

The Hawthorne Effect describes a situation where merely measuring workers improves their performance for a short period of time. I think something similar, but better is going on with WoW pickup groups and small companies. If you’re willing to measure yourself, you’re more willing to learn to perform better in fundamental and permanent ways. Can measurements be gamed? Maybe they can. In WoW, I will forgo other stats to bring my healing bonus up. This is probably not great for my long term raiding viability, but it is the one thing that other non-healers know how to measure. Fortunately, in WoW, there are ways to cover up the slack in some of my stats. I can eat buff foods, and drink elixirs to cover up my woefully low stamina for instance. For me, this leaves a guilty voice in my head that says I’m not measuring all of the stuff I need to, so yes I’m beginning to look for gear with more stamina and mp5 so I can stay alive and contribute longer.

In real life, I hate being measured and measuring myself. I’m a fat-body because I don’t like to get on the scale. I’m not a great programmer because (among other reasons) I’m not willing to profile my work and measure it against the good works out there, or against any performance goal. I just want to slam in the code and pray that it works. As a sysadmin, the idea of measuring my work by the number of warnings and alarms that nagios produces annoys me, but if I make good alarms, they are actually a passable measure of how stable the system is.

So what do I have to do? I have to think briefly about whether I’m measuring the right things about myself and then start measuring.

LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”" combined

and replace it with:

LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” %D” combined

The %D appends the response time to the end of the log line. After you have collected some hits with response times, you can use a perl one-liner to look at timestamps and response times like so:

tail -1000 access_log|perl -ne ‘chomp; split; $_[3] =~ s/^\[//; print "$_[3] $_[$#_]\n”;’

replace tail -1000 with whatever number of log lines you want to look at. I use some GNU date magic,  perl’s GD:Graph, and some shell programming to build a graph of average and high response times over the last hour. For now, I leave this as an exercise for the reader. GD::Graph is a great tool for visualizing all kinds of data where rrdtool may not suffice.

Let’s take a look at some stacks that I would choose:

• Linux, Apache, mod_perl, PostgreSQL.
• Linux, Apache, php, PostgreSQL (or mySQL).
• Linux, Lisp, and some database… or none.

Why Linux? Well, Linux is sufficient. Of the free unices, Linux is the most popular and widespread. Linux has good driver support.

Why Apache? Apache is the most featureful open source web server out there. Apache has a great API if you want to build components in C. Most popular languages in the open source world have an Apache API.

If I have to use a database, I prefer to use PostgreSQL. MySQL has come a long way in recent versions, but PostgreSQL still has some features that I think I need for web programming. One example is the network types. I can do somthing like this, select * from networks where “10.10.0.0/16″ >> ip_address; . The >> means “contains”. In mySQL, or Oracle, I have to build a table of networks and/or IP addresses, iterate through them, and perform the network math myself. PostgreSQL also has a mature procedural language, and if I don’t like pgpl, I can use perl… or tcl… or python as my procedural language.

For familiarity, I would probably choose perl as my language of choice. PHP is also a strong candidate, but for reasons that I will detail in a later article, I probably would shy away from PHP. Perl has very tight integration with Apache. You can perform amazing gymnastics with the life cycle of an http request in mod_perl. Perl also has the advantage of CPAN, a repository of perl modules that do almost everything under the sun. One has to be careful about using CPAN as the quality of the modules vary.

I’d love to use lisp someday in a web application, but only if I have all of the time I need to learn its advanced features. I can imagine a lot of uses for continuations for instance, but I don’t understand them well enough to make effective use of them. So, I’ll keep plugging away at my lisp books and some day, I’ll write a web app in lisp.

Obviously, other people will make different choices based on their experience. Are there technologies that I wouldn’t use if I could avoid them? Yes, there are.

• Windows: Everything you need to use with windows costs extra: office tools, dev tools, database, mail server… For reliability, I’ve never had good experiences with Windows. The company I work for has had production linux servers for 9 months and none of them have ever been rebooted. It’s rare that I find a windows admin that can say that… In fact, I’ve never heard a windows admin say that.
• Java/Jboss: From a sysadmin perspective, I’ve had bad experiences with Java/Jboss. It has a tendency to crash. Application startup is long to “holy crap, that’s long”.  The performance is variable due to garbage collection. Configuring a JBOSS instance requires a lot of specialized knowledge. Jboss programmers tell me that it has a lot of conveniences, but I’ve seen a lot of work go into fixing some of the problems caused by the Jboss infrastructure. As far as Java goes, I’m just not a big fan of the language. It feels like I have to type way too much to get anything done.
• Oracle: It’s a very featureful database, but it’s also very expensive to put into production, and it doesn’t perform significantly better for general database tasks then PostgreSQL or MySQL. Oracle also *requires* an Oracle DBA if you want a smooth running system in a fast changing environment like a startup. PostgreSQL and MySQL don’t require this kind of attention.

Well, that’s my navel gazing for the day. Next time, I’ll talk about my World of Warcraft adventures and the fact that my wife thinks (probably correctly) that I’m addicted.

Currently the interests that I am willing to rant about include; Computer Programming, System Administration, Computer Games, Politics (or meta politics), and some random philosophical navel gazing.