Re: I am giving up wikipedia mirror

Ignoramus23035 wrote:

About 8 months ago I set a up a wikipedia mirror. I also let search
engines crawl it. It return, I got about $10 per day adsense earnings
and an incredible amount of hassle. Googlebot is completely out of
control and was mercilessly hammering my website. It does around 4
queries per second. I think that I pay in bandwidth about as much as I
make, plus I have a big headache.

So, I decided to keep wikipedia mirror (I use it as content for some
of my chapters), but I will no longer let search engines, especially
the badly behaving googlebot, index them.

Last night, I made changes for robots.txt, so far no effect.

I tried using sitemaps to tell googlebot not to crawl page more than
1x per months, but that made it only worse and bolder.

Are you using static pages? Do you update all files, when you update
wikipedia and how often?

I'm not sure how googlebot exactly judges, if a page is updated frequently
and needs to be indexed more often. An often changing (or even absent)
Last-Modified header might fool googlebot, even if the content is
unchanged.

If your mirror tool does not try to keep the Last-Modified dates unchanged
whenever possible, you could try to tune it a bit here.


--
Benjamin Niemann
Email: pink at odahoda dot de
WWW: http://www.odahoda.de/