Yahoo is crawling like hell

Yesterday I had 1957 hits from Yahoo bot coming from
688 different IP adresses!
Maybe other bots work similar.
Of course this destroys my "Unique Visitors" statistic.
How do you solve this problem?

Thanks,
Gerhard

I wish there was some kind of Stats program that lets you specificly
seperate robots stats from the rest of the stats, and see which robot came
when, what it asked for, what it had problems getting etc etc.

I have seen AW stats come close, but not detailed enough info is shown.


"Gerhard Buergmann" <gerhard (AT) puon (DOT) at> wrote

Yesterday I had 1957 hits from Yahoo bot coming from
688 different IP adresses!
Maybe other bots work similar.
Of course this destroys my "Unique Visitors" statistic.
How do you solve this problem?

Thanks,
Gerhard

Gerhard Buergmann wrote:

Yesterday I had 1957 hits from Yahoo bot coming from
688 different IP adresses!
Maybe other bots work similar.
Of course this destroys my "Unique Visitors" statistic.
How do you solve this problem?

Thanks,
Gerhard

Yahoo Slurp supports "crawl-delay" in the robots.txt. It's a bit of a
long shot but perhaps by slowing it down it won't hit your site so hard
and therefore won't need to come in from such a wide range of IP
addresses.