sailing   wooden boats

Enter what you want to search for, to search for an "exact phrase" by enclosing it in quotes. You may search for a combination of words and phrases


Learn more about our "How to Tie Knots" DVD/CD
July 20th 2024 - 02:55
December 12th, 2006
December 12th, 2006

Attack of the Bots

For those of you that know much About Search Engine Optimization you will have heard of "Bots" - the automated programs that crawl the web and fill the search engines with the data they use when people search for things... over the last few days the server on which resides has been pummeled by thousands of these - to the point where it actually crashed the server several times!

Now, having lots of bots visit your web site is generally a good thing. Something, in fact, that many web designers dream About because it means that your site is "popular" and being placed well within the search engines. However when they start to descend on the server at a rate of 50-60 per second things arenīt quite as happy... You donīt then want to Block them because the bots actually help your rank - so... what to do???

What we did was to add a random delay to all our high traffic web sites IF the visitor was one of these bots (I have some fancy detection code for this) thus slowing down the site ONLY for the bots - from between 5 to 30 seconds. We also stop the display of advertisements to the bots on a few of the sites the have lots of ads. The result: well - the server is alive anyway ;) We also created a report that shows what server traffic is the result of bots - right now hovering around 21% of the total - and holding - even after we added the delay and the restriction to ads.

So all is well in the digital world... until the next catastrophe ;)

Share this:
No Comments
You must login or register to post comments Login/Signup
RSS feed Feed Description
Subscribe to the complete The Bosunīs Mate News RSS news feedAll News RSS feed Complete RSS feed
Subscribe to the The Bosunīs Mate News RSS news feed for this category onlyBosunīs Log RSS feed for: Bosunīs Log
A Rich Site Summary (RSS) feed is an xml data file that provides a summary of the information contained here. It is not designed to be viewed in your browser, but instead by rss reader software. If you do not know what this means - you can safely ignore it, as it is provided for advanced users with rss reader software only.

tall ships
Copyright © 1987-2024 The Bosun's Mate
web design, virtual tours and maintenance donated: