Re: I am giving up wikipedia mirror

Ignoramus23035 wrote:

About 8 months ago I set a up a wikipedia mirror. I also let search
engines crawl it. It return, I got about $10 per day adsense earnings
and an incredible amount of hassle. Googlebot is completely out of
control and was mercilessly hammering my website. It does around 4
queries per second. I think that I pay in bandwidth about as much as I
make, plus I have a big headache.

So, I decided to keep wikipedia mirror (I use it as content for some
of my chapters), but I will no longer let search engines, especially
the badly behaving googlebot, index them.

Last night, I made changes for robots.txt, so far no effect.

I tried using sitemaps to tell googlebot not to crawl page more than
1x per months, but that made it only worse and bolder.

i

I would not give up on it. Not for AdSense reasons but the IBLs. If you
suddenly drop a significant number of backlinks to your homepage (or other
pages for that matter), it might negatively affect your SERPs. Besides,
the system is up and running, so why touch it? Googlebot sometimes does
hammer on sites that show static URLs that look suspiciously dynamic to
them. Basically it looks as if they are trying to determine if the site is
actually dynamic and just how much load it can take. In my experience
Googlebot honors the Crawl-Delay directive in robots.txt, so just add this
to your robots.txt


User-Agent: Googlebot
Crawl-Delay: 5

and then give it 48 hours to be discovered and put in effect. make it 30
seconds or even more if you really want to slow it down.

--
Cheers,
Dmitri
See Site Sig Below
------------------------------------


--
##-----------------------------------------------##
Article posted with Web Developer's USENET Archive
http://www.1-script.com/forums
Web and RSS gateway to your favorite newsgroup -
alt.internet.search-engines,alt.www.webmaster - messages and counting!
##-----------------------------------------------##

Ignoramus14359 wrote:

I already tried crawl-delay, to no effect.

Are you sure it's actually Googlebot and not another rogue bot that signs
as Googlebot? Catch its IP address, post it here. Maybe one of kind souls
here that have access ('cause I don't) to Fantomaster's bot IP database
http://searchbotbase.com/ can run that IP against it to see if it is, in
fact, Google's Googlebot

--
Cheers,
Dmitri
See Site Sig Below
-------------------------------------



--
##-----------------------------------------------##
Article posted with Web Developer's USENET Archive
http://www.1-script.com/forums
Web and RSS gateway to your favorite newsgroup -
alt.internet.search-engines,alt.www.webmaster - messages and counting!
##-----------------------------------------------##