![]() | |
![]() |
| | Thread Tools | Display Modes |
#1
| |||
| |||
|
|
I have a mirror of wikipedia. I try to keep track of how many googlebot visits were done to the mirror pages over time. Yesterday, googlebot went NUTS and retrieved roughly 20 times the # of pages as usual! This gets pretty close to what I think is a sane limit of queries. Does google recognize crawl-delay in robots.txt? algebra/2005_07_04/count.txt: Googlebot Wiki visits: 1773 algebra/2005_07_05/count.txt: Googlebot Wiki visits: 1667 algebra/2005_07_06/count.txt: Googlebot Wiki visits: 985 algebra/2005_07_07/count.txt: Googlebot Wiki visits: 1065 algebra/2005_07_08/count.txt: Googlebot Wiki visits: 1637 algebra/2005_07_09/count.txt: Googlebot Wiki visits: 1528 algebra/2005_07_10/count.txt: Googlebot Wiki visits: 947 algebra/2005_07_11/count.txt: Googlebot Wiki visits: 1918 algebra/2005_07_12/count.txt: Googlebot Wiki visits: 2633 algebra/2005_07_13/count.txt: Googlebot Wiki visits: 50869 ======= cooldictionary/2005_07_04/count.txt: Googlebot Wiki visits: 197 cooldictionary/2005_07_05/count.txt: Googlebot Wiki visits: 875 cooldictionary/2005_07_06/count.txt: Googlebot Wiki visits: 311 cooldictionary/2005_07_07/count.txt: Googlebot Wiki visits: 663 cooldictionary/2005_07_08/count.txt: Googlebot Wiki visits: 985 cooldictionary/2005_07_09/count.txt: Googlebot Wiki visits: 417 cooldictionary/2005_07_10/count.txt: Googlebot Wiki visits: 582 cooldictionary/2005_07_11/count.txt: Googlebot Wiki visits: 472 cooldictionary/2005_07_12/count.txt: Googlebot Wiki visits: 636 cooldictionary/2005_07_13/count.txt: Googlebot Wiki visits: 2878 |
![]() |
| Thread Tools | |
| Display Modes | |
| |