Re: Googlebot went nuts!

Ignoramus13314 wrote:

I have a mirror of wikipedia. I try to keep track of how many
googlebot visits were done to the mirror pages over time. Yesterday,
googlebot went NUTS and retrieved roughly 20 times the # of pages as
usual!

This gets pretty close to what I think is a sane limit of
queries. Does google recognize crawl-delay in robots.txt?

algebra/2005_07_04/count.txt: Googlebot Wiki visits: 1773
algebra/2005_07_05/count.txt: Googlebot Wiki visits: 1667
algebra/2005_07_06/count.txt: Googlebot Wiki visits: 985
algebra/2005_07_07/count.txt: Googlebot Wiki visits: 1065
algebra/2005_07_08/count.txt: Googlebot Wiki visits: 1637
algebra/2005_07_09/count.txt: Googlebot Wiki visits: 1528
algebra/2005_07_10/count.txt: Googlebot Wiki visits: 947
algebra/2005_07_11/count.txt: Googlebot Wiki visits: 1918
algebra/2005_07_12/count.txt: Googlebot Wiki visits: 2633
algebra/2005_07_13/count.txt: Googlebot Wiki visits: 50869
=======
cooldictionary/2005_07_04/count.txt: Googlebot Wiki visits: 197
cooldictionary/2005_07_05/count.txt: Googlebot Wiki visits: 875
cooldictionary/2005_07_06/count.txt: Googlebot Wiki visits: 311
cooldictionary/2005_07_07/count.txt: Googlebot Wiki visits: 663
cooldictionary/2005_07_08/count.txt: Googlebot Wiki visits: 985
cooldictionary/2005_07_09/count.txt: Googlebot Wiki visits: 417
cooldictionary/2005_07_10/count.txt: Googlebot Wiki visits: 582
cooldictionary/2005_07_11/count.txt: Googlebot Wiki visits: 472
cooldictionary/2005_07_12/count.txt: Googlebot Wiki visits: 636
cooldictionary/2005_07_13/count.txt: Googlebot Wiki visits: 2878

I think this may be related what I was describing in the "2 different
google crawlers" thread in this group.

Chris
--
http://www.lowth.com - Open source (GPL) security tools and how-tos.