Xenu LinkSleuth Options
One of the people on my own staff used Xenu to spider Xenite.Org yesterday. To say I was not pleased is an understatement. I've now blocked Xenu in my robots.txt and I hope the software honors that file because I don't see how else to stop people from hammering my server.
Xenu grabbed 2400 files in less than a minute. I had to manually update my server log file so that the traffic didn't artificially inflate my daily statistics update.
Xenu uses the prefetching protocol that Google Web Accelerator uses to slam servers. It's just not a server-friendly methodology and I consider use of Xenu to be totally "black hat", totally unethical, and completely unacceptable.
|