Re: Googlebot becoming obnoxious

__/ [Ignoramus25589] on Friday 14 October 2005 05:31 \__

It crawls my websites algebra.com and cooldictionary.com way too
fast.

Today, for instance, algebra.com received 67,000 accesses from
googlebot.

They also misbehave and do not honor Crawl-delay setting.

I am not going to ban googlebot from crawling my site, but I am
becoming somewhat apprehensive of them.

I want to limit their accesses to, say, no more than every 5
seconds. How can I do it, realistically speaking.

i

(Mikkel, I am aware of the subject line which will be bound to my name in
SEPR's... ouch!)

Someone in this newsgroup asked a similar question a month or two back. It
is not quite possible to limit the demands that reach from Google if I
recall correctly. That was the conclusion reached, I think. My recollection
tells me that the best move would be to contact Google saying that your
dedicated server is knocked on too hard.

001 650 330 0100

Call them and get an E-mail address and then be in touch with a real person
promptly.

Hope it helps,

Roy


--
Roy S. Schestowitz | Have you hugged your penguin today?
http://Schestowitz.com | SuSE Linux | PGP-Key: 74572E8E
6:10am up 49 days 18:24, 4 users, load average: 0.82, 0.84, 0.73
http://iuron.com - next generation of search paradigms

Roy Schestowitz wrote:

They also misbehave and do not honor Crawl-delay setting.

I have observed the opposite. They DO behave well in terms of robots.txt
directives handling. The key is: the robot.txt is cached, and I do not
think they update it as fast as they say they do. I would say the real
life cache span for Google is at least 48 hours or even more, so if you
added crawl-delay recently, it may not be in effect as yet.

Did you type it right?

User-Agent: Googlebot
Crawl-Delay: 30

It is not quite possible to limit the demands that reach from Google if I
recall correctly. That was the conclusion reached, I think.

You can' staisfy Google's thirst for knowledge (or world search engine
domination - pick one), but you can make them consume your bandwidth
slower. Like I said above, they do follow crawl-delay directive, but it
takes a long delay (no pun) to take the directive in.

--
Cheers,
Dmitri
See Site Sig Below
-------------------------------------

--
##-----------------------------------------------#
Article posted with Web Developer's USENET Archiv
http://www.1-script.com/forum
Web and RSS gateway to your favorite newsgroup -
alt.internet.search-engines - 15980 messages and counting
##-----------------------------------------------##