Quote:
I just noticed that the Baidu Spider has started putting "findmore.org"
as the referer in their header.
My stats page is Disallowed in robots.txt so referer spam doesn't work,
but I still dislike the idea that these guys are trying it. The spider
is crawling at a reasonable rate (just a couple of hits per hour at the
most), so bandwidth isn't an issue.
Any thoughts on this? Is putting a referer in the spider crawl spammy?
Should I ban these guys from my website? |
Have no idea on your question.
What I can tell you is baidu is a Chinese search engine. If your don't
expect Chinese visitors to visit your website, maybe you can ban it.