Comments on: robots.txt Adventure

By: robots.txt analysiert | Webmaster

robots.txt analysiert | Webmaster — Wed, 26 Sep 2007 08:08:05 +0000

[...] Eine interessante Untersuchung von Andrew Wooster. Der ließ 4,6 Millionen Domains von einem selbst gebastelten Spider ansteuern, um jeweils die Datei robots.txt einzusammeln und zu analysieren. Dabei kamen nicht nur statistische Daten über Statuscodes und Mime-Typen zu Tage, auch allerlei Merkwürdigkeiten wurden ausgemacht, die auf ein sonderbares Verständnis der Datei schließen lassen. So finden sich Texte aller Art, Keywords, Logs, Listen und sogar ASCII-Kunst in einer Datei, die sich ausschließlich an Bots und Spider richtet. [...]

By: Jason King

Jason King — Tue, 25 Sep 2007 01:37:32 +0000

Oh that’s amusing, interesting and useful too.

I frequently do health checks on other peoples’ websites but it hadn’t occured to me to check they’ve written their robots file correctly. I’ll add that to my list of checks.

By: Max Design - standards based web design, development and training » Some links for light reading (25/9/07)

Mon, 24 Sep 2007 21:51:33 +0000

[...] robots.txt Adventure [...]

By: Links on a fickle monday

Links on a fickle monday — Mon, 24 Sep 2007 18:56:18 +0000

[...] Interesting web surveys: robots.txt and http headers (via Simon Willison). [...]

By: Andrew

Andrew — Mon, 24 Sep 2007 16:51:27 +0000

Sorry about that Sean. I’ve fixed it in the article.

By: Sean Conner

Sean Conner — Mon, 24 Sep 2007 08:54:35 +0000

Just one small quibble: it’s “c-o-n-n-E-r”.

By: Nick Gully

Nick Gully — Sun, 23 Sep 2007 18:26:32 +0000

I think you’re missing some important rules for robots to follow:
# A robot may not injure a human being or through inaction allow a human being to come to harm.
# A robot must obey the orders given it by human beings, except where such orders would conflict with the First Law
# A robot must protect its own existence, as long as such protection does not conflict with the First or Second Laws.

By: egorych

egorych — Sun, 23 Sep 2007 10:45:05 +0000

Hey, I’ve translated this article into Russian (of course you’ve got some more links :)).
This is great. I’m surprised how many sites from Dmoz have such stupid errors. They are likely to be good sites, aren’t they? It’s so hard to get into dmoz now…

Good job.

By: links for 2007-09-23 « Simply… A User

links for 2007-09-23 « Simply… A User — Sun, 23 Sep 2007 00:30:59 +0000

[...] nextthing.org Â» robots.txt Adventure (tags: web robots.txt http search spider robots standards internet google genius analysis **) [...]

By: James

James — Sat, 22 Sep 2007 13:36:15 +0000

You missed the blog in http://www.webmasterworld.com/robots.txt