Attack of the Bots Part 2

I thought a follow-up to our last report About the search engine robots descending en mass upon out web sites was in order. We have resolved the problem and some people may be interested in the solution...

We had been looking at the server reports for some time - but none of the indicated anything extraordinary. The CPU was only working at About 20% max and there was over 700Mb of free ram. So although we could see things slowing down - we couldn´t see what was causing the sluggishness.

Finally we found a stats tool called iostat that gave us the answer - it indicated that the hard drive was running pegged at 100% and the CPU was spending most of its time waiting for the data from the HD to arrive.

Solution: add another Hard Drive and move all the databases there. Which we did - and the problem has been solved. In fact you´ll probably notice that this site is now even faster then before as a result.

We also did a bunch of tweaking on the MySQL database to get it to use more RAM which is also a contributing factor - but nothing compared to the database move and the separate physical hard drive.

Anyway - I just though some of you would appreciate knowing wht the result turned out to be.

---