nextthing.org

Adding Default Descriptions to Trac Tickets

Andrew — Sat, 03 May 2008 04:37:37 +0000

I’ve been using Trac for the new projects I’m working on, and am liking it so far. I have, however, done a bit of customization to make it fit my needs a bit better. One thing that took awhile was figuring out how to change the default ticket description.

I like having some structure to my bug descriptions, including steps to reproduce, regression information, etc. Thankfully, Trac’s template system allows you to replace output with an override template (typically found in templates/site.html) that is applied before content is sent across the wire. It was a bit time-consuming to figure out how to do what I wanted to do, but I eventually ended up with a templates/site.html file that looks something like this:

"http://www.w3.org/1999/xhtml" xmlns:py="http://genshi.edgewall.org/" py:strip=""> "req.environ['PATH_INFO']=='/newticket' and not req.args.has_key('preview')"> "textarea[@id='field-description' and @class='wikitext']" py:attrs="select('@*')"> Your custom description goes here. 

I hope this is useful for someone.

On Regret Minimization, Leaving Apple, and Playing Pinball

Andrew — Thu, 01 May 2008 02:37:33 +0000

It’s been a month now since I left Apple to start a company. I’d been working on Time Machine (an interesting and challenging project) as a software engineer, with a fantastic group of people, and was in a situation that was really pretty great. So, why did I decide to leave and go do my own thing?

The answer is complicated and, in between bouts of programming, I’ve been trying to put it into words for a few weeks now without much success. However, there are a couple of quotes I picked up over the years that I think encapsulate my main reason for leaving and the main thing I want to do.

The first has to do with regret, and comes from Jeff Bezos, on leaving his stable, great job to start a startup:

So, it really was a decision that I had to make for myself, and the framework I found which made the decision incredibly easy was what I called — which only a nerd would call — a “regret minimization framework.” So, I wanted to project myself forward to age 80 and say, “Okay, now I’m looking back on my life. I want to have minimized the number of regrets I have.” I knew that when I was 80 I was not going to regret having tried this. I was not going to regret trying to participate in this thing called the Internet that I thought was going to be a really big deal. I knew that if I failed I wouldn’t regret that, but I knew the one thing I might regret is not ever having tried. I knew that that would haunt me every day, and so, when I thought about it that way it was an incredibly easy decision.

- Jeff Bezos Interview, Academy of Achievement

I’ve been thinking about starting a company since high school, and maybe even a bit before that. Certainly by the time I hopped on the Internet for the first time, it had become one of my goals. Over the intervening years, I’ve had a few opportunities to start a company, but somehow have never just gone and done it. I’d always just kept thinking it was something I was going to do “someday”, but that, for the moment, there was something interesting to do at the place I currently was. Apple in particular is a pretty difficult place to leave, in that there are a huge number of interesting things to work on, a huge number of great people to learn from and work with, and is just really a very special place to be, especially over the past few years (Apple’s Silver Age).

However, I was increasingly feeling like the “someday” was going to need to become less abstract for me, or it would never happen. If it was hard for me to leave Apple to start a company in my 20s, with no wife, kids, or mortgage, it was only going to get harder from there on out. That knowledge, coupled with knowing that if I never actually started a company I’d regret it my entire life, was a strong impetus for me to gear up and give it a go.

Given that I’ve been thinking about starting a company for a long time, I’ve given what I enjoy doing and what I want to do with my time a lot of thought over the years, and found that what I really enjoy is playing pinball:

They didn’t have to name the bigger game. Everyone who had been on the team for a while knew what it was called. It didn’t involve stock options. Rasala and Alsing and many of the team had long since decided that they would never see more than token rewards of a material sort. The bigger game was “pinball.” West had coined the term; all the old hands used it. “You win one game, you get to play another. You win with this machine, you get to build the next.” Pinball was what counted. It was the tacit promise that lay behind signing up, at least for some. Holberger felt that way. “I said, ‘I will do this, I want to do it. I recognize from the beginning it’s gonna be a tough job. I’ll have to work hard, and if we do a good job, we get to do it again.’”

- Soul of a New Machine, Tracy Kidder, p. 228

This quote comes from a book about the creation of a new computer by a small team at Data General. As the team eventually found out, Data General was a big company, and big companies don’t necessarily understand pinball. Success on a project likely means being kept on for continued maintenance of the project, not necessarily being drafted for the next big thing. Happily, however, some of them discovered later that the startup was a great way to play pinball.

Given that what I enjoy doing is building new things, I intend to do as much of that as possible. Essentially, my plan is to build a lot of small projects in succession, and see which ones take off. Certainly a big part of my inspiration in this approach comes from David Weekly, an acquaintance and Silicon Valley entrepreneur. David quit his job a few years ago, and dedicated his considerable talent and a huge amount of time to building a service that, ultimately, not many people used. Rather, the project that did take off for him was, literally, an overnight hack.

So, the lesson for me from all this is, as an individual entrepreneur, that my best bet for being able to play pinball is to build a lot of smaller, but still useful projects. Each project will either come at the right time and be the right idea, or it won’t be, but hopefully having more of them will ensure that the probability of eventual success will be higher. Or, at least, my probability of utter failure will be lower, and I’ll have a bunch of cool stuff to show for my time (and some “valuable operational experience”, as one of my friends describes failures).

So, that’s what I’m endeavoring to do, and so far I’m having a blast. I’ve eliminated a big potential regret from my life, and I’m getting to work on some new and interesting things. So, by those metrics, my little crash course in entrepreneurship is going very well. Anyway, enough rambling, I’ve got to get back to work!

Happy Leopard Day!

Andrew — Fri, 26 Oct 2007 23:43:23 +0000

In January, I started working as a software engineer on Leopard. I was lucky enough to get to work on two of the top ten features for this release of OS X:

For the past few months I worked seven days a week pretty much non-stop on making the back end of Time Machine faster and more reliable. It was a lot of work, but overall a very rewarding experience, and I’m happy that I got to work on such a kick-ass release of OS X. Enjoy!

Now, I’m off to a beer bash for some much needed relaxation.

UI Patterns: Wet Lazy Susan

Andrew — Mon, 04 Jun 2007 03:52:35 +0000

I ran across a couple of variations on the “wet floor” effect today which both introduce a rotating set of images that can be moused over for more information. For lack of a better term, I’ll call this the “Wet Lazy Susan” effect.

First up, Amazon.com has the following under the “Recommended Based on Your Browsing History” heading on their homepage:

I like the controls for scrolling through the circle of products. The tooltip-like popups on mouse over are nice too.

Then there’s the following, found on Nissan USA, shown here at half size:

Here the shadows that fade in when an item is highlighted are a nice touch, as is the text being in the same position for each item.

Has anyone else noticed any variations on this pattern floating around out there?

Update: Yeah, I probably should’ve remembered that Front Row has the same effect for its main menu. Doh.

Microsoft Web

Andrew — Sat, 31 Mar 2007 20:07:43 +0000

Here’s a snippet from Microsoft’s current corporate home page:

"-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> "en" dir="ltr"> "Content-Type" content="text/html; charset=utf-16"> Microsoft Corporation "Content-Type" content="text/html; charset=utf-8"> "SearchTitle" content="Microsoft.com"> "SearchDescription" content="Microsoft.com Homepage"> Download this code: /code/microsoft.txt

Can you spot the problem?

Update: James Booker is the first in with an answer: there are two Content-Type meta tags above, both of which specify different character sets.

Specifying the content-type in meta tags is a bit of a hack, as the browser has to seek through the first section of the document looking for a content-type declaration, then try reinterpreting the page with the character set the page specifies. Specifying a character set of “utf-16″ doesn’t make any sense in this scenario, as the browser is going to try the sniffing by interpreting the HTML as ASCII. If the page were actually UTF-16, this wouldn’t work, as the representation for the string “Content-type” in UTF-16 isn’t identical to its representation in UTF-8, as we can see in a Python shell:
>>> "Content-type".encode("utf-16") '\xfe\xff\x00C\x00o\x00n\x00t\x00e\x00n\x00t\x00-\x00t\x00y\x00p\x00e' >>> "Content-type".encode("utf-8") 'Content-type'

Thankfully, the HTML spec foresaw this problem:

The META declaration must only be used when the character encoding is organized such that ASCII-valued bytes stand for ASCII characters (at least until the META element is parsed). META declarations should appear as early as possible in the HEAD element.

So, there are actually three problems with the above HTML. There are two content-type declarations in meta tags, one of them is bogus, and the correct one isn’t as early as is possible in the head element. These problems, thankfully, are mitigated by the presence of an HTTP header that specifies the correct character set, and by the incredible amount of effort browser vendors have put into making their code accepting of mistakes such as these.

robots.txt Adventure

Andrew — Tue, 13 Mar 2007 06:58:19 +0000

Introduction.txt

Last October I got bored and set my spider loose on the robots.txt files of the world. Having had a good deal of positive feedback on my HTTP Headers survey, I had decided to poke around in robots.txt files and see what sorts of interesting things I could find.

Since then, I’ve taken 6 weeks of vacation and gotten to be very busy at work, so I’m just now getting around to analyzing all the data I gathered. These are some of the results of that analysis.

Robots?

To those of you completely unaware of what this post is about, here’s a brief primer. Google is a search engine. You probably use it. If not, odds are you use one of MSN Search (now called “Live Search”), Ask Jeeves (now Ask.com), or Yahoo! Search. How do those search engines grab web pages to search? Well, they use robots, also called spiders. Now, these aren’t the giant metal machines you see chasing tweaked out English factory workers through the streets of London, nor are they the giant eight-legged creatures you find lurking behind clocks. Rather, they’re pieces of software that surf around the web grabbing web pages. Since they’re software, they can surf the web much faster than humans, as well as find things most humans might overlook. As such, there arose a need for a standard for advising robots on what they should and shouldn’t look at.

The Un-Standard

The Robots Exclusion Protocol arose in June 1994 by consensus among a number of web spider developers. The original protocol description from 1994 describes the basic syntax of a robots.txt file to be placed at the root of a web site. So, for example, Google would place their robots.txt file at:

http://www.google.com/robots.txt

The basic format goes something like this. First, the file specifies a User-agent (the name of the robot) that is to follow the subsequent rules (until the next User-agent line):

User-agent: SuperHappyRobot

This line tells “SuperHappyRobot” that it needs to pay attention to the next few lines. Any other robot will ignore these rules. The next line might look something like:

Disallow: /tmp/

Which would mean SuperHappyRobot shouldn’t download any pages that start with the path “/tmp/” from this server. Variations on these lines are that * will match any robot name (in other words, “User-agent: *” should tell all the robots to pay attention), and blank Disallow statements mean anything goes. So, Apple’s robots.txt file of:

# robots.txt for http://www.apple.com/
User-agent: *
Disallow:

means, essentially, that any robot is free to grab any page it can get its hands on, at least for the “www.apple.com” website.

So, that was all well and good, but around 1996 there was a push to try to get robots.txt standardized, and an IETF draft (http://www.robotstxt.org/wc/norobots-rfc.html) was produced that clarified and added to the robots.txt syntax. The primary addition was a new “Allow” rule, which allowed a little more fine-grained control over which pages could be retrieved. For example, with the following set of rules:

User-agent: *
Disallow: /apache/
Allow: /apache/02/03/11/2228242.shtml

All documents except “/apache/02/03/11/2228242.shtml” in the “/apache/” path would be excluded from spidering. There was also a provision for “extensions” to the protocol, such that a rule line like “Crawl-delay: 10″ could be added. Spiders that didn’t support that extension would ignore it, while spiders that did might delay 10 seconds between page fetches.

Around the same time the IETF draft was being discussed, Sean “Captain Napalm” Conner proposed his own extension to the Robots Exclusion Protocol, which included Allow rules as well as regular expression syntax for rules, and new Robot-version, Visit-time, Request-rate, and Comment rules. Less than 100 of the sites I visited use rules unique to this spec.

Since none of these three documents have ever been ratified or adopted by a standards body, there has been a bit of persistent confusion over what constitutes a valid robots.txt document. The most definitive document is certainly the original 1994 document. Most commercial robots today, however, attempt to conform to the IETF draft document. And, given the large number of Allow rules around, it would be remiss of a robot not to try.

A Touch of Controversy

This de-facto standard has had its share of controversy over the years. Many webmasters object to having to opt-out of spiders crawling their site. Given that I found 47,738 sites that disallow spidering the root of their site with the wildcard (*) user-agent match, it appears that that viewpoint still has many adherents, and many just want to be left alone by the bulk of spiders. See the comments in this thread for some examples of this opinion from some relatively tech-savvy webmasters. Among them is the well-known IncrediBILL:

Lack of a robots.txt file should mean just that, they don’t know about robots so robots should STAY THE HELL OUT!

I’ll come back to this later.

Others have objected to the idea of putting up a roadmap to secret pages on their sites. Bertrand Meyer, the designer of Eiffel (the programming language, not the Tower) and a Very Smart Person even holds this viewpoint. To quote:

If you are just a bit absent-minded, isn’t it natural
to use this mechanism to exclude stuff from being indexed and hence believe
no one will find it? “Stupid”, maybe — but not unlikely.

Indeed, scanning through the robots.txt files I pulled down, I find disallow rules for 3,000+ “phpMyAdmin” paths, 40,000+ “stats” paths, 31,000+ “log” paths, 400+ “secret” paths, 100,000+ “admin” paths, and a host of other interesting looking entries. Even if the vast majority of these are properly secured with authentication, the chances of a few people being absent-minded, as Bertrand might say, are pretty good.

On the flip side of these opinions, there are those who have always viewed, and want to continue to view, robots.txt as a merely advisory standard. As courts and legislative bodies have begun to apply the force of law to this loose consensus protocol, some have spoken out in favor of information transparency and the essential openness of the Internet, including Marijn Koster, the creator of the protocol:

“I don’t think that’s in the spirit of free information exchange,” Koster says. Some robots may have legitimate reasons to ignore robot exclusion directives. For example, he says, a company might use robots to hunt for copyright infringing content.

Methodology

Having written a spider for my HTTP headers survey and run it against all of the domains in the Open Directory, I already had a large collection of web sites, and a decent spider. I further added to my list of domains by extracting links from the pages I’d downloaded for that project. Then, I ran my spider (written in Python, using PycURL) against this expanded list of domains, attempting to retrieve the robots.txt file at each. The HTTP headers and full body of the response were stored in a MySQL database. This database was then dumped via a custom “Big File” implementation, which amounted to a bit more than 12GB on disk. Then, I wrote an analyzer which could run through this logical file, processing the records, recording interesting statistics about the entries and reporting the results. This analyzer takes about half an hour to run on the dataset. In total, I received responses from about 4.6 million unique domains.

Status Codes

HTTP status codes (aka response codes) tell web browsers and robots both what kind of response they’re getting when they download a page. For example, “200″ means everything is okay and “404″ means the web server couldn’t find the file the browser requested. The IETF robots.txt spec says that a 404 response for robots.txt means the site is unrestricted for robots, and a 2XX response means the robot must respect the returned robots.txt content. Other status codes have recommended behaviors, but they’re not required.

Status codes are interesting primarily because they give a quick count of how many sites have a robots.txt file. I got responses from 4.6 million sites, so by tallying the response codes of different types, I can tell who has a robots.txt file and who doesn’t:

Status Code	Count
404	3,008,767
200	1,217,303
302	276,106
301	72,674
403	15,675
400	5,570
401	3,856
500	2,841
410	1,450
303	1,319
503	890
304	529
501	280
502	227
307	218
204	215
300	100
504	60
406	58
419	45
550	36
202	34
999	17
100	12
418	10
201	7
405	6
423	6
666	3
402	3
415	3
407	2
510	2
490	1
505	1
509	1
900	1
409	1
408	1
Total:	4,608,330

Broken down by class, we get:

Class	Count	% of Total
5xx	4,338	0.09
4xx	3,035,454	65.86
3xx	350,946	7.61
2xx	1,217,559	26.42
1xx	12	0.00
invalid	21	0.00

As we can see above, around 65% of sites return a 4XX status code, indicating they don’t have a robots.txt file. Another 7.6% redirect to a different URL, usually either the home page or an error page. This means, essentially, that about 26% of sites are attempting to serve up a valid robots.txt file. Of course, some sites may improperly return an error page with a 2xx status code, so this is only useful as a quick estimate.

MIME Types

MIME types (aka content types) are returned in the headers of HTTP responses by web servers to tell clients what the document’s type is. They consist of a type (text, image, etc), a subtype (like html or jpeg) and some other optional parameters (like the character encoding). So, for example, an HTML file usually has a MIME type like “text/html” and a text file a type like “text/plain”. An image file might have a MIME type like “image/gif” or “image/jpeg”. The IANA keeps an official list of registered MIME types at http://www.iana.org/assignments/media-types/.

The only MIME type that should be returned for a valid robots.txt file is text. True, the specs don’t specifically mention MIME types, but sites like Google follow the general HTTP rule of “if it’s not text/*, it’s not really plain text”. Of the robots.txt files I got back, 109,780 of them had MIME types other than text/plain. So, it should be no surprise that the big 3 search engines (Yahoo!, Google, and MSN) all will attempt to parse any text robots.txt file they get back from the server. For example, Digg.com serves up their robots.txt file as “text/html; charset=UTF-8″. Google, MSN, and Yahoo! all obey the rules in the file.

Besides for text/html and text/plain, some of the more common MIME types I got back were application/octet-stream, application/x-httpd-php, text/x-perl (mostly error pages), video/x-ms-asf, application/x-httpd-cgi, image/gif, and image/jpeg.

Even among files ostensibly marked as text, there were a wide variety of questionable MIME types:

Count	Content Type
2	application/txt
5	application/x-txt
2	file/txt
1	internal-gopher-text
30	plain/text
12	text
13	text/R*ch
2	text/aleph_save
2	text/ascii
6	text/asp
36	text/css
2	text/dhtml
73	text/enriched
1	text/htm
1	text/illegal
1	text/javascript
2	text/octet-stream
1	text/plane
4	text/rtf
1	text/ssi html
3	text/svg
3	text/text
9	text/txt
20	text/vnd.wap.wml
5	text/x-component
87	text/x-invalid
1	text/x-log
386	text/x-perl
2	text/x-python
40	text/x-server-parsed-html
23	text/xml
11	txt

No, Really, Robots Dot TEXT

An error similar to using the wrong content type is uploading a robots.txt file in a format other than plain text. Popular mistakes here include Word documents (examples: 1, 2, 3), RTF documents (examples: 1, 2, 3), and HTML. I even found LaTeX and KOffice documents.

One piece of server software (called Cougar, which looks, as near as I can tell, to be either Microsoft Small Business Server or IIS), even spits out ASF streaming video files when asked for a robots.txt file (examples: 1, 2). Fun.

Invalid Encodings

Character encodings specify what letters and other characters correspond to which specific bits. Sites specify what character set a response is in within the Content-type header. Some sites serve up robots.txt files in little-used encodings, such as UTF-16. UTF-16 is tricky for a number of reasons, not the least of which are the different endian encodings. Of the 463 UTF-16 files I found, approximately 10% were not valid UTF-16, even though they included a UTF16 BOM.

Otherwise, I saw close to 300 unique character sets claimed by servers, even discarding obviously incorrect ones and making them all lower case. These included some ones I hadn’t seen before, like “nf_z_62-010″, “ibm-939″, and “fi_fi.iso-8859-15@euro”.

Comments

robots.txt have one and only one proper way to comment, which is to put comments after a hash mark (#). However, I found HTML comments (), C++ style comments (//), and a variety of others, including simple in line comments.

Totally Confused

Some people seem rather befuddled as to what constitutes a robots.txt file. For example, the most common confusion I’ve found is people using the raw text dump of the Web Robots Database as their robots.txt file. I’m not just talking about a couple of sites, either. Approximately 1 in every 1000 websites I looked at do this. It’s really quite bizarre. This seems to be part of a more general mistake wherein people copy instructions on how to set up a robots.txt file into the contents of robots.txt files. For example, here are a few: www.cooljobscanada.com, www.numis.co.uk, www.volubilis2000.com, www.kickapoo-orchard.com, www.aplussupply.com.

Then there are just the random things you find. Religious texts and descriptions of churches. A catalog for MIDI tracks.

ASCII art, both pornography and otherwise.

A list of videogames. Several .htaccess files. Access logs. Lists of keywords and website descriptions, including an actual keyword stuffing example. Bash scripts, PHP pages, and everything in between.

I even found image files being served for robots.txt. Not to mention e-mail messages and newsgroup postings.

There’s even a description of a swimming pool. In German.

And, of course, plenty of human-readable instructions to robots which can’t read them: http://www.corsicamania.com/robots.txt.

info.txt

Apparently there’s another protocol, similar to robots.txt, for advertising the contact information for a site. A file called info.txt is supposed to be placed in the root of the site, which sites like Alexa will look for when trying to find out who owns the domain. I found a lot of these records in the robots.txt files.

Someday I’ll have to see how many of these there are in the wild.

Wildcards

There are no wildcards (also known as pattern matching) in the official robots.txt specs, but various search engines have added extensions to support this.

For example, Google, MSN Search, and Yahoo! allow an asterisk (*) to match any sequence of characters, and a dollar sign ($) to match the end of the URL. So, to block spiders from downloading any JPEG image files, one might use:

User-agent: *
Disallow: /*.jpg$

Indeed, blocking spidering of certain file types is the most popular use for wildcards. Most people who are using wildcards for anything else are doing so entirely unnecessarily. For example, a lot of sites have the following rule:

Disallow: /RealEstateTips/*

The use of the non-standard wildcard above is useless, as this rule is equivalent to:

Disallow: /RealEstateTips/

This is because rules are by default partial paths, and will match any path beginning with that string. It’s also worth noting that of all the sites which have the above rule with the wildcard, none of them have the rule without the wildcard. So, a spider which didn’t support pattern matching would be free to download urls that start with “/RealEstateTips/”, so long as they didn’t have an asterisk after the second slash.

Common Syntax Errors

So, besides for the above, what are some of the common errors? The spec says that records are separated by blank lines, and the most common errors center around that. First most is putting a blank line between a User-agent line and the rules that should apply to it, with 74,043 files doing this. Next up is the placement of a Disallow or Allow rule with no User-agent or Disallow/Allow rule immediately before it, with 64,921 files making this mistake. The next is placing a User-agent line immediately after a Disallow/Allow line, with no space in between. 32,656 files did this. Finally, lines which were neither comments, nor blank, nor rules showed up in 22,269 files.

Crawl-delay

The IETF robots.txt draft spec includes a provision for extensions to the robots.txt format. Basically, along with “Allow” and “Disallow” lines, spiders can optionally support extensions for enhanced control over the robot’s behavior. The most widely-deployed of these is the Crawl-delay extension.

MSN Search, Yahoo!, and Ask all support Crawl-delay, which is used to insert a delay between successive accesses of a web server. A typical Crawl-delay might look something like this:

User-agent: *
Crawl-delay: 5

Which spiders that support Crawl-delay would interpret as meaning they should wait 5 seconds between requests to the site. I found tens of thousands of these entries.

Typos!

I found a LOT of typos in these files. You wouldn’t think it would be very hard to spell the limited vocabulary of “User-agent” and “Disallow” correctly, but you’d be wrong. For example, I found 69 typos of Disallow. 69! That’s not even counting the ones I found with weird characters in the middle of the word.

Fingerprinting Using robots.txt

Sometimes, we can use robots.txt file contents for fingerprinting the sites that serve them up. For example, we can fingerprint the sites designed by Moriah.com by looking for robots.txt files with the contents:

this file placed here so you don't fill up my error log looking for it

Examples: 1, 2, 3, 4, 5.

Similarly, we can find the more than 7,000 real estate sites designed by Advanced Access by looking for the rule:

Disallow: /RealEstateTips/*

More usefully, we can identify one Korean domain squatter by looking for robots.txt files that contain only a meta tag like:

meta http-equiv=refresh content='0;url=http://www.hiplayer.com'

(brackets excluded because of a bug in WordPress).

At the time I spidered, we could identify another domain squatter by looking for a robots.txt file like:

User-agent: *
Disallow: /pixel/
Disallow: /library/
Disallow: /results_monitor.asp

They’ve since switched to a more generic, but still easily-identifiable robots.txt file.

Using similar methods, it’s easy to find a lot more domain squatters, mass-hosted websites, etc. A search engine could potentially maintain a list of such signatures and, based solely on the robots.txt file, not bother indexing the page. Or, more generally, it could increase or decrease the relevance and ranking of the site in its search results.

Conclusions

Okay, so what conclusions can we draw from this mess of data? The primary conclusion, I think, is that the Robots Exclusion Protocol is more complicated than it actually seems. As a spider, in order to properly parse the variety of robots.txt files you’ll find in the wild you’ll need to write an extremely lenient parser (following the Robustness Principle), mostly ignore content types, handle a variety of character encodings (and in many cases ignore those returned by the server), detect HTML and other content returned in the guise of robots.txt files, and potentially implement multiple extensions to the accepted standard.

How about the position, discussed above, that spiders shouldn’t spider or download content without the explicit permission of the webmaster? Belgium has certainly come down on the side of requiring explicit permission. However, the evidence shows that Google is in the right on this one:

“Given the vast size of the Internet, it is impossible for a search engine to contact personally each owner of a web page to determine whether the owner desires its web page to be searched, indexed or cached… If such advanced permission was required, the internet would promplty grind to a halt,” Google’s senior counsel and head of public policy Andrew McLaughlin told the Senate Legal and Constitutional Affairs Committee.

As seen in the status codes section, if this were to happen, nearly three quarters of domains on the web would go “dark” for search engines. If these sites went dark for search engines, they would essentially be offline for the majority of web users. Such an action would be in nobody’s best interest; not the site owner’s and certainly not in those of the web-using public at large.

On a less serious note, it’s always interesting to see just how vast the Internet really is. Few things drive that home for me as much as seeing how varied the content people generate on the web can be.

So, until next time, I leave you with a quote from one of the robots.txt files I came across:

are you searching something???

Yes. Yes I am. And so far, every time I look, I find it.

More resources:

Alexa robots.txt Search example
Google Webmaster Tools - Includes a robots.txt validator.
Sun’s amusing robots.txt file
Google Blog on the Robots Exclusion Protocol: First Post, Second.

New Zealand Trip 2006

Andrew — Fri, 09 Feb 2007 07:00:24 +0000

Introduction

In mid-2006, I decided that I wanted to use up some of my long-accrued vacation, and started planning a 5 week trip to New Zealand. I’d wanted to go to New Zealand for a couple of years, after having seen photos of the country from my friend Josh’s ’round the world trip.

Typically, for me, planning a trip involves booking transportation, getting a rough idea of what I want to do, and figuring things out when I get there, a tendency which annoys some of those close to me to no end. I didn’t want New Zealand to be any different, so I booked airfare and investigated my in-country transportation options pretty carefully. I finally settled on renting a campervan, so that I’d be able to set my own pace for exploring the country. That settled, I booked the only activity I really had to book in advance, a hut ticket for the Milford Track, which is reputed to be the “finest walk in the world.” That settled, I geared up for the trip, buying lots of photography and hiking gear, and prepared myself for my first vacation outside of North America.

Outward Bound

On November 15th, I boarded an Air New Zealand flight bound for New Zealand. It’s a long flight, so I’ll spare you the details, but after what seemed like an eternity (I don’t sleep very well on airplanes) I arrived in NZ, at 5am on the 17th. Losing a day threw me off a bit.

As I groggily made my way through customs, I noticed that New Zealand seems to care a great deal about “biosecurity”, aiming to preserve the local wildlife and native biology through preventing intrusive species from coming into the country. My hiking boots had to be declared, and were given a good looking at. I felt sorry for a forlorn looking cowboy in the X-ray line by himself, his saddle and tack up on the conveyor belt and boots off for inspection. He looked like he’d been there before and expected to be there for awhile.

After catching a taxi to the Auckland Hyatt, I hoped to do a super-really-early check-in, but had to wait until past 10am for my room to free up. I was glad I’d booked a few nights there, as I was totally exhausted, and, as is par for the course for me on flights, I’d come down with a cold. However, having five hours to kill, I wandered around downtown Auckland taking pictures, which was fun, although not quite as much fun as going to sleep would have been.

Santa in Auckland

Auckland is an easy city to navigate on foot, and being a pedestrian for a couple of days was a good way for me to get used to cars driving in the left-hand lane from a perspective other than that of behind a steering wheel. It’s also a tidy city, with people who speak in a strongly British-influenced accent, with a strong showing of Asians reminding me that I was on the Pacific Rim. In those ways, it reminded me of Vancouver. In others, the feel of the city is much like San Francisco, with quirky little shops, big company headquarters, a smattering of tech startups, and a wide selection of good places to eat. It’s also a city on the ocean, which helps with the San Francisco vibe, and so it didn’t take me long to start feeling comfortable.

After catching up on my sleep that night, I set out to explore a bit. First, I caught a bus to the Parnell area and walked over to the Auckland Museum. The Museum’s collection runs the gamut from natural history, to artworks and furniture, to Maori artifacts and WW2 warplanes. The most interesting thing, for me, was the collection of stuffed native birds, including reproductions and skeletons of Moa. Moa were giant flightless birds native to New Zealand that grew to up to 13 feet in height, and standing next to one made me appreciate how terrifying they must’ve been for the Maori who hunted them.

Another highlight of Auckland was Kelly Tarlton’s Antarctic Encounter and Underwater World. As the name would suggest, it’s a cross between an aquarium and an antarctic exhibit. This was the only place in NZ where I saw penguins, and was by far the best aquarium I went to, with the 110m long underwater tunnel and the giant squid on display being real highlights. Here I also learned that an adult octopi can fit through a hole the size of a dime. Apparently, Kelly Tarlton’s used to have a problem wherein crayfish (giant NZ lobsters) would be found eaten when they opened the aquarium in the mornings. It turns out that an octopus had been squeezing out a tiny hole at the top of its tank every night, crawling across the floor to the crayfish tank, eating a crayfish, and crawling back before morning. Neat.

Moa

Penguins at Kelly Tarlton’s

Also in Auckland, I picked up my transportation: a Toyota Hiace campervan from Ezy rentals. The Hiace is a popular, cheap, and incredibly wimpy van, which in my case had been professionally converted to have seats in the back that convert to a bed, plus a stove, sink, refrigerator, and microwave. Mine was also fitted with a giant fiberglass top, which served both to increase headroom and, as I discovered later, act as a huge sail in the wind, whipping me all over the road. Fun.

Freedom! And right hand drive!

Nevertheless, I quickly got up to speed on driving. As I’d given myself a few days as a pedestrian to observe traffic from the sidelines, it wasn’t that hard to adjust to driving on the left hand side of the road. It helps that Kiwis (as New Zealanders call themselves) are incredibly polite, even while driving, and the speed limits are pretty low compared to the US. I quickly grew to appreciate the consistent use of roundabouts, which make traffic move much faster than stop signs would allow.

From Auckland to Rotorua

On the 20th of November I set out from Auckland to Rotorua. The drive was most uneventful, taking me through lots of green grassy areas, and past plenty of sheep. After wandering around the town getting my bearings, I visited a fancy old spa built in the 18th century. There I learned that long ago they used radium-infused water to “treat” people. Aahhhh! Afterwards, I ate dinner in town and drove to my first campervan site for the night.

Campervan parks (also called holiday parks) are basically the equivalent of RV parks in the US, offering laundry and shower facilities, along with electric hookups for campervans. The site in Rotorua was pretty nice, set along side a small stream, with well-maintained facilities and even wireless Internet access (for a fee).

The next day, after some drama involving not being able to open the safe in my campervan to retrieve my camera gear, I headed out to see some of the sights. Rotorua is famous for its thermal formations, so I went to the nearest one, known as Hell’s Gate. George Bernard Shaw once said of this place, “I wish I had never seen the place, it reminds me too vividly of the fate theologians have promised me.” Indeed, George’s description is apt, and the Maori tribe which owns the site has a taken the theme and run with it, giving names to the various formations such as Devil’s Cauldron, Sodom and Gomorrah, Devil’s Bath, and Map of Australia (Kiwis have a friendly rivalry with their Western neighbors). While there, I also had my first chance to look at New Zealand tree ferns up close. These ferns grow to be quite large (20′ tall or more) and are quite sturdy. They also look like something dinosaurs would have eaten. After touring around the park for a bit, I exited past the spa, where you can bathe in the hot spring water or take a hot mud treatment. Personally, it smelled too bad for that to sound very enticing. And, as it was a hot day (I got my first sunburn of the trip here), the thermal bath seemed doubly unenticing.

Rotorua Bath House

Thermal Formations at Hell’s Gate

Tree Ferns at Hell’s Gate

Enjoying the Sun

Have I mentioned how bright the sun is in NZ? It’s bright. Extremely bright. Everyone seems to wear sunscreen, and I was warned constantly to do so by Kiwis. I saw several people in their 50’s and 60’s walking around who quite obviously had had a bout with skin cancer, so I took the sunscreen warnings to heart, applying sunscreen constantly.

After Hell’s Gate, I headed to Wai-o-tapu Thermal Wonderland. This place is incredible. The thermal pools are gigantic, and most are very colorful. My favorite by far is the Champagne Pool, which is a 65m diameter pool of bright blue water, with a brilliant orange ledge around the rim, just under the waterline. Sadly , it’s rather difficult to photograph, as there’s a tremendous amount of steam rising off the water at any given time, as the surface temperature is about 165 degrees farenheit. Nonetheless, I took a bunch of pictures, although they didn’t turn out as well as I’d like, mostly due to camera operator error (for the curious, I shot at ISO 1600 by accident all day). After leaving, I stopped by some mud pools just outside the park, having fun just staring at a giant boiling mud puddle, trying to guess when the next large air bubble would rise and cause mud to gush into the air.

The Champagne Pool

Hut at the Buried Village

The next day, I headed down to the Buried Village of Te Wairoa, which was a Maori and colonial village buried in a volcanic eruption in 1886. The guided tour was excellent, and conducted by the descendant of a famous Maori guide who lived in the village at the time of the eruption. Here I got to see the construction of Maori dwellings up close, including the usage of the tree fern trunks for walls and roofs, and the inclusion of secret escape doors (in case of attack from other Maori). Around the back of the excavated village was a trout stream, where the guide said there were “very large” trout, holding his arms out to show 2 feet or so of length. I thought he was exaggerrating, but if anything, he was understating how enormous the trout in the stream are. Also, there was a short hike past a beautiful waterfall, going down a steep set of steps to its base. The Buried Village was a definite highlight of my trip thus far.

Art Deco in Napier

At the village, I decided to check and see when I had to be in Fiordland for my one scheduled activity of the trip, hiking the Milford Track. It turned out that I had booked it for the 29th of November, which meant that I had only 7 days to get to the other end of New Zealand. Realizing this, I jumped in my van and started driving. First heading up to Rotorua to fill up with diesel, I then headed south. I considered stopping in Taupo, a major tourist town on the edge of a beautiful lake, but instead kept driving all the way to Napier.

Napier is a small city (50,000 people) on the Eastern seabord of New Zealand. Much of Napier was destroyed in 1931 by an earthquake (followed by a fire), and so much of the downtown area was rebuilt around that time in an Art Deco style. I spent some time walking around the city center (or centre, as they spell it in NZ) enjoying the cool looking buildings, many of which wouldn’t look out of place in California. Then, I had a seafood dinner at a cafe fronting the bay, and headed to another carpark. This carpark had a pleasant surprise in store for me: the fastest wireless Internet access I found in New Zealand. It was great. I even got to video chat over iChat with some of my friends back home, all while sitting in the back of my campervan. I also took this opportunity to upload a bunch of the photos I’d taken.

Thanksgiving

The next day, Thanksgiving, I booked passage on the Interislander Ferry from Wellington to Picton for the next morning and started driving to Wellington. The drive took me through some very beautiful country; country populated, mainly, by sheep. There were also a few vineyards and field crops thrown in for good measure, but still… Sheep. Lots of them. The other memorable part of this drive was the extreme wind. This had the unfortunate effect of making my campervan much harder to control than it should have been. At one point, I was considerably worried that my van was going to be toppled by a gust of wind, so I drove very, very slowly.

Fortunately, I made it to Wellington without any wind-related mishaps. As it was Thanksgiving, I decided to try and find something suitable to eat in Wellington proper. The best I could do was a steak dinner with guacamole and chips at a restaurant purporting to be Texan. Sadly, they didn’t know how to make chips, guacamole, or steak particularly well. All the same, I was thankful that I could have a meal at least resembling something American, and thankful that I was on such a great vacation. Being alone on Thanksgiving also made me that much more aware of how thankful I am for all of the people in my life, who I missed very much.

To the South Island

I got up early the next morning in anticipation of my ride on the Interislander. Heading into the ferry terminal an hour ahead of my departure, I noted that it was still windy in Wellington. In fact, as I waited in line behind the other campervans, I noticed them all rocking gently back and forth in the wind.

Once I had parked my car in the ferry, I headed upstairs to watch our departure from the viewing platform. It was a windy but mostly clear day, and I had a decent view of the Bay and of downtown Wellington as the ferry pulled out into the sea.

On the Road to Te Anau

After arriving in Picton, on the South Island, I immediately started to head south. The drive took me along the eastern coast of the South Island, through country that reminded me much of California. There were browning rolling hills that looked like the area around King City, as well as a few row crops and a lot of grape farming to round out the similarities. Of course, rather than cattle dotting the fields and hillsides, there were sheep.

Christchurch

The drive took me down the coast, through areas that reminded me of Big Sur, with a narrow highway winding a path between shrub covered rock on one side and the sea on the other. Around 7pm I finally arrived in Christchurch and checked into a carpark. After that, I wandered around town for a bit, saw that the city center is indeed dominated by a giant church, and got to see a bit of the soapbox rantings the city center is so famous for. After wandering up and down the streets looking for a place to eat, I settled on sushi at a restaurant called Sala Sala. I had a crayfish combo, which consisted of half a crayfish sashimi style (raw), and the other half grilled, with a salad, crayfish miso soup, some assorted snacks, and a small dessert. The crayfish was excellent! By the time I got back to the carpark, however, it was 11:30, and I realized that the extended daylight hours of the low southern hemisphere were really throwing my schedule off. All the same, I appreciated them, as I was able to do and see a lot, every day, all day, with mostly full sunlight until mid-evening.

The next day, rather than doing any more sightseeing in Christchurch, I spent some time booking transport to and from the Milford Track, something I had forgotten to do earlier. Luckily, the Department of Conservation and Real Journeys web sites together enabled me to get booked up easily, and I was back on my way. So, I set out for Dunedin, which is much further down the coast from Christchurch. It was a long, mostly uneventful drive. I will say, however, that this particular drive made me miss the US Interstate Highway System. The speed limit in NZ is about 62mph, and the roads go through every little town along the way, in each of which you have to slow down to 31mph or less. At home, the drive would’ve only taken 3.5 hours, but in NZ it took over 6. The tradeoff, however, is that in NZ there are loads of vibrant and healthy small businesses along the highways, with real communities rather than suburban sprawl, whereas in the US such towns are much less healthy.

Dunedin Train Station

Dunedin, it turns out, is an interesting town. It’s built in a bit of a hilly seashore area, and as such there are a lot of hillside houses with nice views of the ocean. There is also a lot of stone architecture in the town center, with the rail station being a prime example. It also seems like the churches all tried to outdo each other in their gothic stone grandeur.

The next day’s driving took me through a whole bunch of quirky small towns, including Gore, which bills itself both as the Brown Trout Fishing Capital of the World and the Country and Western Music Capital of New Zealand.

Gateway to Fiordland

On arriving in Te Anau (the “Gateway to Fiordland”), I got settled in yet another campervan park, and walked into Te Anau proper. The town is fairly small (population 1,900) and is mostly centered around tourism in the Fiordland region. There are tour operators for everything from fishing boats, to seaplanes, luxury tour boats, sightseeing buses, and everything in between, all jockeying for the attention of travelers seeking the spectacular outdoor scenery of the fiords. The town itself offers some impressive scenery, sitting on the edge of Lake Te Anau (the largest lake on the South Island), with snow-capped mountains lining the horizon.

On recommendation of the carpark staff, I headed over to the cinema to watch a 30 minute film about the Fiordlands. The film, Shadowland, was a kind of cinematic tour of the park, with no audio track except some mood setting music. It has a lot of really spectacular shots taken from a helicopter, and really conveys how spectacular the area is. I was so impressed with the scenery and camera work that I decided to purchase the DVD, and ever since have been showing it to whichever of my friends will watch.

The next day, I took a guided tour through the Te Anau glowworm caves. This involved a ferry trip across the lake, where we were led into a small limestone cave and back past waterfalls and a raging underground stream to board a small boat. This boat was pulled via a cable by our guide through a pitch black cavern in which we were able to view the glowworms all over the walls and ceiling. It was like staring at oddly close and vibrant green stars. It was a bit of a surreal experience, and one I’d recommend.

The next couple of days were spent doing some last-minute shopping for camping gear and food supplies, picking up tickets, and arranging for secure storage of my computer and campervan. The Department of Conservation staff were helpful, but also scared me with talk of water up to my hips on sections of the Milford Track, all-pervasive rain, and snow and wind to boot.

The Milford Track

Bus pickup from Te Anau on the 29th was at 9:45am at the Department of Conservation (DoC) center. After a short ride to Te Anau Downs, we took a ferry to Glade Wharf, the start of the track. The walk started in beech forest, with lots of mossy ground cover, and also lots of rain. In fact, it rained continuously all day. I was able to take my camera out for a bit, shielding it with my umbrella, and snap a few photos along the walk. The rain-obstructed views were of forested, snow-topped mountains rising straight up from fast-running streams and rivers, swollen with the day’s rain.

Clinton Hut

At one point, there was a 4 minute side walk to a “wetland” area; really a bit of a bog. It was interesting, however, as the track led to a small viewing platform in a boggy clearing, which afforded views of the surrounding mountains. Back on the main track, there were standing puddles several inches deep on the track. After a little over an hour we reached Clinton Hut, our first stop. I was surprised to find that Clinton Hut is both larger and more well-appointed than I had imagined. There are two bunk rooms, each with 20 bunks, plus a restroom outdoors (with sinks and flush toilets!), as well as another hut with gas cookers, benches, sinks, and a wood-fired stove. A ranger cabin sits nearby, with a friendly fellow staffing it full time. Out back is a walkway to a helicopter pad which also serves as a good viewing platform for some waterfalls (from which the Hut draws its water) and mountains.

As we arrived early in the afternoon, we built a fire and waited for the complement of our party to arrive. Later that day they started to trickle in, significantly wetter than ourselves. Apparently, the puddles had gotten quite a bit deeper since we’d arrived, and they’d had to walk through water that went past their boot tops. Around 7:30, the ranger dropped in to give a brief speech and collect our hut passes. He had walked up to the next hut and back to check trail conditions, and reported that the water was up to the middle of his thigh! So, I expected to get thoroughly wet the next day.

The second day’s walk was a gentle climb through a long valley. Most of the walk was through beech forest, while sections had only grass and low brush. The steep sides of the valley were dusted with snow part way up, giving them a rather striking appearance. These mountains rose straight up out of this narrow valley, and the rain and snow runoff gave rise to thousands of small and large waterfalls cascading off the sides and onto the valley floor. It was amazing, and by far the most awesome part of my trip thus far.

I stopped for lunch at a guided walk shelter. The guided walkers, who pay significantly more than us unguided (or “freedom”) walkers, have luxury lodgings (read showers), with meals prepared for them at stopping points along the walk. Thankfully, they let us use their huts for shelter when we ate lunch, so I was able to take my pack off and eat without getting soaked. Soon after I stopped, a kea showed up at the shelter. The kea is the world’s only alpine parrot. I’d heard about this “clown of the mountains” from my friend JD, but was not expecting them to be as large as this one was! it hopped around as I unpacked my food and ate lunch, peering at me in hopes of an opening in which to rummage through my pack for food and transportable objects. As I ate, other campers arrived, and rather than scaring the bird away, the increased attention only seemed to embolden it. Eventually, I started getting cold, so I packed everything back up and started back on the long walk to the Mintaro Hut. Along the way was a side walk to a small lake. There I was able to snap some shots of Paradise Ducks and of a rather nice waterfall falling directly from the cliffs into the lake on the valley floor.

Second Day of the Hike

Cheeky Kea

After what seemed like an interminable walk, I finally made it to Mintaro Hut. My hips had been hurting me for the last couple of miles of my walk, and I was glad to finally be able to dump my pack off. Outside the hut was another kea, even more aggresive than the first. It actually hopped up to one of the hikers and started nibbling on his jacket!

Mossy Trees

All that day, a helicopter had been making passes up and down the valley. During the hut talk that evening, the ranger (a woman in her 20’s) told us she’d been up repairing the trail after it had washed out during the rain. That night, after talking with my fellow trampers (as they call hikers in NZ) for a bit and eating a rather nondescript meal reconstituted from a freeze-dried packet, I went to bed early, feeling tired and sore.

On the third day of my hike, I got up and set about readjusting my pack so it would ride a little higher on my hips. That done, I set out late, having lost a lot of time reorganizing my pack. The initial walk was easy, but I quickly came to the ascent up to MacKinnon Pass. The ascent was difficult, but soon I was rewarded with awesome views back down the Clinton Valley, up which I had walked the previous two days. Eventually, I got to the pass, and saw some of the most incredible views I’d ever seen. At the top, there is a monument to Quinton MacKinnon, who discovered the pass in 1888. Resting at the monument were some guided Japanese tourists, who were having fun photographing the playful keas, and I had one of them take my picture next to the 12 Second Drop. Then, I hiked over to the rest hut, changing into warmer clothes along the way, as it was both very cold and very windy along the top of the pass.

Starting the Third Day

On the Ascent

At the shelter, I had lunch with some of my fellow hikers and went out to take a look from the “loo with a view”, a toilet perched at the edge of a cliff looking out over the valley. One of the hikers there said that a guide told her he’d once seen a kea grab someone’s jacket, fly over the edge of the 12 Second Drop, and drop the jacket in the valley. Apparently they also like to steal boots because they offer good protection from the rain.

Me by 12 Second Drop

View from the Pass Shelter

Sutherland Falls

After lunch, I started down the mountain. The climb down was much harder than the climb up, and took me much longer. The climate also changed quickly from very cold to very warm partway down, so I had to change clothes again. Eventually, I got down to the independent walker shelter, which was next to the hut the guided walkers would stay at that night. I set down my pack and started towards Sutherland Falls which was a short side hike away. I trekked to the falls with my camera and tripod, but wasn’t able to get many shots because the wind was blowing water everywhere around the base and I was immediately soaked in freezing water. On the way back, I tried to find some blue ducks in the river that a guide had directed me to, but was unsuccessful. After retrieving my pack from the shelter, I moved on, stopping further down the river to take some photos of the falls from a distance. A ranger stopped to talk to me, and he told me about some more spots to find blue ducks (I wasn’t able to find them there, either). Later, some people doing the Milford Run ran by, saying “beautiful view” as they passed. The Milford Run is a one day run of the Milford Track. Basically, people who don’t book hut passes can’t camp on the track, so these runners will run the entire track in one day, as a sort of endurance challenge, and presumably to make some sort of point.

Eventually, I got to the hut, just as the ranger’s talk for that night ended. As I ate dinner, I and some others chatted with the ranger, who was a really interesting guy. It turns out that he has ranger friends all over the world, and has a huge collection of park ranger hats and clothes from places he’s visited. After eating, I went to bed as soon as I could, knowing I’d have a long day ahead of me.

On the fourth day on the track, my hiking caught up to me. My feet hurt, badly, but I took some ibuprofin and after a quick breakfast I was on my way again. After passing the Boat House, an avalanche site, and a neat suspension bridge over a still river, I arrived at Bell Rock and Mackay Falls. Bell Rock is essentially a large rock shaped like a bell, having been hollowed out by running water and then flipped upside down. I knelt inside the rock, and took a few shots of the falls, but then had to move on. Later on that day, I stopped and ate lunch near Giant Gate falls with some of my fellow trampers.

Morning on the Fourth Day

Mackay Falls

Finally, as my body was about to give out, I reached Sandfly Point, our pickup point for the ferry to Milford Sound. After the brief ferry ride, during which I exchanged contact information with some of my fellow trampers, we reached Milford Sound. While the others went for a cruise around the fiord, myself and a couple of others went to the cafe for some Cokes, beer, and chocolate, three things I’d missed on the track. After that, we caught the TrackNet bus back to Te Anau, where I settled into a hotel room I’d booked, knowing I’d want to sleep in a real bed for a change.

Milford Trampers

On the Ferry

The next day, I drove back to Milford Sound and went on a cruise around the fiord. I also paid to stop off at the Milford Deep Underwater Observatory for a bit. The Deep Underwater Observatory is a viewing platform about 10 meters under the surface of the water. There’s a layer of dark fresh water covering the salt water from the ocean, which allows all sorts of ocean life that is usually only found at deep depths to thrive in the unusually dark water near the surface. My impression was that it was like being surrounded by a giant aquarium (that, or being in one). After getting back from my cruise, I walked along a path to a tidal flat and spent a good 40 minutes just taking photos of the awesome scenery.

Milford Sound

Falls Up Close

Fish at the Observatory

Path to the Tidal Flat

Queenstown

The next day, I drove to Queenstown, world renowned as the extreme sports capital of New Zealand. I wandered around town for a bit, familiarizing myself with the downtown area, and browsing through the hundreds of brochures for touristy activities in the area, deciding what I wanted to do over the next couple of days. My first activity, the next day, was to go on the Shotover Jet Boat. This is a jet boat that zips up and down a narrow river canyon, skirting mere feet past sheer rock walls, and going over sections with just a few inches of water. It was exhilarating, and by the time I got off the boat I was literally shaking from the adrenaline rush.

Queenstown From Above

Riverboat Jet

The next day, I went on the jet boat again (half price for the second ride), and in the afternoon I went on the Shotover Canyon Swing. This is billed as “the world’s highest swing”. After being strapped into a chest harness, I jumped off the edge of a platform extended out over a river canyon, free-falling 60m before the swing went taut and I swung through a 200m arc at 93mph. Wow. As people jumped off the edge, white water rafters in the river below would cheer at the people plummeting towards them and swinging over their heads.

Shotover Canyon Swing

Glacier Country

On the 7th of December, I drove from Queenstown to the West Coast. Along the way I stopped at Stuart Landsborough’s Puzzling World in Wanaka, where I wandered around a giant maze for awhile and gawked at an impressive collection of optical illusion rooms and puzzles. Then, I drove over the pass to Haast, stopping at the Blue Pools along the way. The Blue Pools are a bunch of deep pools along a river which are a bright shade of blue, and really quite striking. After a lengthy drive, and a bit of hiking in the hopes of seeing Fiordland penguins on a beach (hopes which went unfulfilled), I got to the town of Frans Joseph around 9:30, which was just after dark.

Roman Toilets

Blue Pools

The next day, I went climbing on Frans Joseph glacier for a half day hike. It was a terrific amount of fun! The guiding company I went with took me and probably 10 others up to the glacier, along with our guides. At the base of the glacier, we strapped crampons (spikes) onto our hiking boots, and climbed up the ice stairs the guides carved out with their mining picks. There were probably 5 guides working all day to maintain a very narrow trail of ice steps and bridges up the glacier. We hiked about a kilometer up the glacier, to a point where we stopped and took photos looking back down the valley from which we’d come. I got the distinct impression from the guides that the glacier was extremely dangerous, as they put a lot of effort into making sure the path was clear and the bridges tied down properly. The glacier moves quite a bit in a day, and the paths the guides carve out have to change on a daily basis due to the shifting and melting ice. I took a lot of photos, got extremely wet (it rained), and couldn’t wait to get warm again. All the same, it was an amazing experience.

View from a Glacier

Up the West Coast

The West Coast of New Zealand is a thin strip of very wet country between the Tasman Sea and the Southern Alps. The drive up this coast reminded me much of Monterey, California, with tall gangly trees, fog, and dense forest along with the requisite rugged shoreline. In the east rise the Southern Alps, snow capped peaks rising out of lush forest. It’s a spectacular drive, and one in which towns on the map are as likely to have 2 people as 200, there being only 30,000 people living on the several hundred mile stretch of the West Coast.

Along my drive up the coast, I stopped at Hokitika to buy some jade carvings. Jade (known as greenstone in NZ) was a treasured good among the Maori for its toughness and beauty, and NZ carvers do a healthy business carving traditional jade necklaces, statues, and other items, with Hokitika being the center of that activity. I also stopped by the Pancake Rocks and Blowholes at Punakaiki. These are rock formations of layered sandstone which look roughly like stacks of giant pancakes. When the tide is up, blowholes in the rocks send sea spraying up into the air in giant jets. I didn’t get to see the blowholes in their full glory (the tide was low), but the weather was bright and sunny, which I was told was quite unusual.

Pancake Rocks

Back to the North Island

On the way back to Picton to catch the Interislander, I passed through some country which looked a lot like the wine country in Sonoma or Monterey counties. There were golden hills, miles of vineyards, and even California poppies, which triggered a bout of homesickness. Back in Picton, I visited the Edwin Fox museum, where the last East Indiaman is on display. Then, I caught the Interislander to Wellington.

Looks Like Home

Skeleton of Edwin Fox

In Wellington, I spent a day just wandering around the city, taking photos and visiting Te Papa, the national museum. Te Papa is fairly large, and has some great exhibits on Maori culture, the cultural history of New Zealand, and New Zealand art. I also took the time to see the latest James Bond Movie, Casino Royale, at the Embassy Theatre. There I got to sit in the leather seats the cast of the Lord of the Rings movies sat in for the premiere, and to watch a movie on the largest screen in the southern hemisphere.

My next stop was New Plymouth, where I did a short day hike up the slope of Mount Egmont. Mount Egmont is a giant volcano which bears a striking resemblance to Mount Fuji, and which affords excellent views of the surrounding countryside. Afterwards, I drove across an extremely hilly area of New Zealand to get to Turangi.

Mount Egmont

The next day, I got up at 5:30 and took a bus to the Tongariro Crossing. This is a one day hike that takes about 6 hours, and goes up over a high pass of volcanic terrain. It passes several craters, a volcano, and some clear blue lakes. One of the volcanoes it passes is a forbidding looking cone that is familiar to many as “Mount Doom” in the Lord of the Rings movies.

The other major activity in the Turangi area is trout fishing. The rivers in the area are stocked with rainbow trout, and fishing them is a favorite NZ activity. I looked into fishing while I was in the area, but would have needed a guide to find the good fishing spots, and given how much guides cost, it wouldn’t have made sense to go fishing for just one day. I did, however, visit the National Trout Center, which had some great displays on trout fishing, a hatchery, a trout stream/pond, and a nice viewing area where I could look at some large trout through a glass window into a stream.

Red Crater

Rainbow Trout

After visiting the Trout Center, I drove north to Rotorua, and after much deliberation I went wet zorbing. Zorbing involves getting in an inflated plastic ball with a bucket’s worth of water, and being pushed down a hill. It was loads of fun, and I ended up going down the hill twice. There’s also a “challenge” where you try to run the length of the hill inside the zorb, and I noticed that my Kung Fu teacher from Pomona was one of the few people to complete the challenge. What a small world!

On the 17th, I went “black water rafting” in the Waitomo glow worm caves. We were fitted out with wetsuits, wet boots, and hard hats with lights. Then, we hopped in a van for a drive to the caves and picked out inflated inner tubes for floating through the cave system, which is filled with water. The caves were a bit claustrophobic at first, but I quickly got over that and got into the experience of jumping off waterfalls, clambering around low caverns, and floating on my tube through pitch dark sections chock full of glow worms. I saw a whole lot more glow worms on this trip than I did in Te Anau, and overall it was one of my favorite activities in New Zealand.

Zorbing!

In a Cave

Bay of Islands

After the caving trek, I drove north to the Bay of Islands, which is on the northern end of the North Island. There, I went on a 4 hour cruise, hoping to be able to swim with the dolphins which inhabit the Bay. The group of dolphins we found, however, had their young with them, so we weren’t able to swim with them for safety reasons (presumably both for the dolphins and the people). All the same, it was fun watching them, whistling at them, and hearing them whistle back. I even saw one leap about 20 feet straight up out of the water, as well as others doing some other acrobatic maneuvers. They were much larger than I anticipated, and really just all-around interesting animals. Later, while wandering around Paihia, I saw some California quail, the same kind we have on the ranch back home. Apparently they were the only upland game bird that did very well in New Zealand’s climate.

Dolphins

The next day, I drove from the Bay of Islands back down to Auckland by way of the Waipoua Forest. There, I saw the tallest and the biggest trees in New Zealand. These are giant Kauri trees; impressive trunks that rise high up through the forest and terminate in tiny (by comparison) tufts of branches.

Back in Auckland

When I got to Auckland, I checked into a carpark site just north of town. In reception, I overheard someone frantically calling around canceling his mobile phone, credit cards, etc, having had his campervan get broken into at the Auckland zoo. I decided not to visit the zoo the next day, and resolved to be very careful parking in Auckland. The next day, I drove into Auckland, parked at a secure carpark, and went shopping for Christmas presents. Afterwards, I went to the maritime museum, which has an impressive collection of boats, South Pacific Islander craft, and related items. Auckland is known as the City of Sails for the large number of boats in its harbors and rich maritime history, and the museum really conveyed how much people there love the sea.

Sky Tower

Small Craft

That night, I took an elevator up Sky Tower to get a view of the Auckland nightscape from up high. The top of Sky Tower has viewing platforms where you can lean against very sturdy glass walls and look down at the city from 220m up. It’s a very impressive view.

Auckland at Night

The next day, I got up and packed, cleaning out the van and throwing out two trash bags full of stuff in the process. Then, I drove over to the rental place and dropped off my car. They were nice enough to watch my bags while I walked up to the Parnell, ate some fish and chips and did some last-minute Christmas shopping. Then, I caught a cab from the rental place to the airport, where I did some last-minute postcard writing and caught my long, long flight back to the US, bidding New Zealand a fond farewell as I watched the country drop away into the sunset from my window seat.

Also…

Check out some photos from my trip: New Zealand Photoset on Flickr.

A Peek Inside

Andrew — Sat, 27 Jan 2007 19:59:39 +0000

The Shomstoppers present: Jolly Boots of Doom.

Undergraduate Computer Science Reading List

Andrew — Fri, 29 Dec 2006 01:17:16 +0000

My friend and coworker Buzz Andersen posited awhile ago that there exists a programming canon, a body of literature with which software developers and computer scientists are expected to be familiar.

Having interviewed a lot of people (and I mean a lot) and been interviewed myself a few times, Buzz’s point resonated with me. I’ve noticed that many recent university graduates have not been exposed to material that is considered fundamental to computer science and programming by many in the industry. A lot of schools have sacrificed the teaching of this material in favor of more IT development oriented courses, and really done their students a disservice in the process.

Since I’d rather see more kick-ass programmers and computer scientists than fewer, here’s my list of books that I think should be on the bookshelves of all computer science undergraduates. These books are, of course, my own favorites, and they exclude a lot of the books I’d include in a list of graduate-level books. Feel free to chime in in the comments if you think I’ve missed something important.

Note: The cover images are all linked to Amazon, all other links point at the homepage of the books on the web (or Wikipedia).

The Fundamentals
	Introduction to Algorithms by Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, Clifford Stein This book is a bit dense as an introduction, but serves as an excellent reference. Supplement it with a good algorithms and data structures book, such as Bruno R. Preiss’ excellent and free Data Structures and Algorithms with Object-Oriented Design Patterns.
	The Art of Computer Programming by Donald Knuth This is a book most people (including myself) have probably browsed through or skimmed, but never read. Nevertheless, it sits in the background in many CS programs, informing the way introductory computer science is taught. Alternatives for introductions to CS vary from school to school, but many use SICP. My personal favorite is the text used at Harvey Mudd College, “Computer Science: Abstraction to Implementation”, aka “Computer Science for Smart People.”
	The C Programming Language (2nd Edition) by Brian W. Kernighan, Dennis Ritchie, Dennis M. Ritchie This is the canonical introduction to C, the lingua franca of systems programming, and a favorite source of interview questions.
	Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig AI is a fascinating field, and this book covers a lot of ground. Incidentally, Peter Norvig is the Director of Research at Google.
	Operating System Concepts (aka The Dino Book) by Abraham Silberschatz, Peter Baer Galvin, Greg Gagne I work in an OS group, so OS books are important to me. This is the book my OS class at Mudd used, and I still refer to it at work.
	Modern Operating Systems by Andrew S. Tanenbaum An alternative to the Dino Book.
	Computer Networks by Andrew S. Tanenbaum If you can’t joke about what the “U” in UDP stands for, you need to pick up a book on networking. Any good book will do.
Compilers and Programming Languages
	Compilers (aka The Dragon Book) by Alfred V. Aho There’s a new edition of this book coming out soon. For many years this was the standard in teaching compilers, though recently “Engineering a Compiler” has become more popular.
	Engineering a Compiler by Keith Cooper, Linda Torczon Often plugged as the modern alternative to the Dragon Book, this book has proven popular in recent years as the Aho book has grown increasingly out of date.
	The Little Schemer by Daniel P. Friedman A whimsical introduction to Scheme and functional programming. In my opinion, CS graduates should at least be familiar with one functional programming language.
Software Engineering
	The Mythical Man Month by Frederick P. Brooks If you want to understand why throwing more engineers at a project may actually make the project take longer, this is the book to read.
	Design Patterns by Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides (aka The Gang of Four) Also known as the GoF (Gang of Four) book. There are also a variety of “Design Patterns in X” books for various languages, which may help make the various abstractions clearer.
UNIX
	Advanced Programming in the UNIX Environment by W. Richard Stevens and Stephen A. Rago Recently updated by Rago, the 2nd edition of this book is truly excellent. At the very least this is a required reference if you’re programming on a UNIX-based system or just want to understand the design of the API’s.
	Unix Network Programming, Vol. 1 by W. Richard Stevens In much the way APUE is the classic for learning UNIX programming, this is the classic for learning network programming.
	Unix Network Programming, Vol. 2 by W. Richard Stevens Whereas the first volume teaches you how to get your computer to talk to others on a network, this one teaches you about interprocess communication (or how to have programs on a local computer communicate).
	Design of the UNIX Operating System by Maurice J. Bach Contains great descriptions of many of the parts of the old AT&T Unix System V. Given the increasing complexity of Unix systems, this a good introduction to the design of an actual operating system from when such things were much simpler.
	Lions’ Commentary on Unix by John Lions SCO is not entirely evil, having given permission for the republication of this book in 1996. This book contains the source code for the 6th Edition UNIX kernel with excellent commentary on the code and a description of the PDP-11 architecture on which it ran.
Theory
	Introduction to the Theory of Computation by Michael Sipser The theory of computation is all about computability and complexity. In other words, it asks what can be computed, and at what cost? This comes up in programming surprisingly often, as tasks that are deceptively simple for people (like finding the shortest route between several stops on a map) can be quite complex for computers to perform. This book serves as a good introduction to the field.
	Computers and Intractability : A Guide to the Theory of NP-Completeness by M. R. Garey, D. S. Johnson Another introduction to the theory of computation, with lists of common hard problems.

Other lists of readings in computer science and software development:

Rolling Garfield Ninja Turtles

Andrew — Thu, 26 Oct 2006 09:30:25 +0000

So, a friend pointed out today that when you search for “Rolling Stones” on Google, you may not get the results you were expecting:

Apparently, to Google, Garfield.com is the canonical source for information on the Rolling Stones. Also, Garfield.com and NinjaTurtles.com are apparently the same site:

Hmm, I wonder what the top result is for “Ninja Turtles“. Let’s see:

Yup, that Garfield cat sure is popular. For completeness’ sake, I’ll point out now that searches for both “Teenage Mutant Ninja Turtles” and even “Ninja Stones” also show Garfield.com as the top hit. If you find more, let me know. In the mean time, I’m really curious as to why this is happening, so let’s try to figure that out.

So, why does Google think Garfield.com and NinjaTurtles.com are the same site? Well, might they be hosted on the same machine?

$ host www.ninjaturtles.com
www.ninjaturtles.com has address 216.204.128.19
$ host www.garfield.com
www.garfield.com is an alias for ucomics.com.
ucomics.com has address 198.247.208.81
ucomics.com has address 198.247.208.121
ucomics.com has address 198.247.208.122
ucomics.com has address 198.247.208.123
ucomics.com has address 63.208.55.161
ucomics.com has address 63.208.55.201
ucomics.com has address 63.208.55.202
ucomics.com has address 63.208.55.203

Nope. Neither is anything about the Rolling Stones. Okay, just for fun, let’s look at the robots.txt file on Garfield.com:

# robots.txt for http://garftest.uclick.com/

User-agent: *
Dissallow: /

That’s weird. First of all, they misspelled disallow as “dissallow”. A quick check of other sites with disallow misspelled in their robots.txt files shows that Google ignores the misspelled disallow directive. So, the file should have no affect on indexing.

Maybe Google’s is using a hash of the domain name to identify identical sites? Trying 16 common hashing algorithms (MD5, RIPEMD-160, Tiger, SHA-1, SHA-256, Adler32, CRC32, etc.) of various permutations of the domain names with and without punctuation, I didn’t find any collisions. That doesn’t rule out hash collisions using different (say, homegrown) hashing algorithms, but I’ll move on for now.

Next, scanning the results returned for Garfield on various search engines, I don’t see any evidence of Googlebombing the terms “Garfield”, “Rolling Stones”, or “Teenage Mutant Ninja Turtles”.

So, I give up. Maybe Garfield’s prominence in Google results is an homage to Eugene Garfield, a citation analysis pioneer cited in Larry and Sergey’s PageRank paper from 1998.

Okay, that’s a stretch, but I’m stumped. Anyone else have any ideas?