Re: Yahoo Slurp went NUTS!

__/ [ www.1-script.com ] on Friday 28 April 2006 01:06 \__

Is it just my turn today or you guys also see an enormous increase in
Yahoo's crawling?

For the past week they have been crawling a lot. I haven't checked all my
sites and I only noticed that last night. Earlier this month it was MSN that
ate up over a gig in just a couple of weeks. Google is only third in terms
of activity. Imagine that...

I would not be surprised if they will end up requesting 100,000 pages
before the day's end (on a 500,000+ pages site). What are they, on a
mission to re-crawl it all in less than a week or something?

You have over a quarter of a million pages on your site, according to what I
can see. Could you slow down the delivery pages selectively (i.e.
deliberately slowing it down for Yahoo)? The quicker you serve pages, the
better capacity is assumed and the more greedy crawlers can become, I think.

It could also be that they are trying to beef up their index in order to
beat Google in their "mine's (index) bigger" race.

One thing I noticed last week is that the number of pages Yahoo retain
(saturation) has increased significantly. The number of referrals increased
as well. Perhaps they go for depth and breath rather than highly-regarded
pages.

What do you guys see in your stats?

Too much stuff for conclusions to be reached within a decent amount of time.
Measuremap, which is a stats tool for blogs (recently bought by Google),
attempts to summarise detailed statistics and translate them into simpler,
digestable, informative lumps. Maybe it's a smart way to go. Imagine a stats
package that indicates (using flag) where there are changes in general
changes in behaviour (trends). I don't know Urhcin beyond its demos...


Hope it helps,

Roy

--
Roy S. Schestowitz
http://Schestowitz.com | SuSE Linux ¦ PGP-Key: 0x74572E8E
8:40am up 15:45, 12 users, load average: 0.42, 0.37, 0.41
http://iuron.com - next generation of search paradigms

Roy Schestowitz wrote:

For the past week they have been crawling a lot. I haven't checked all
my
sites and I only noticed that last night. Earlier this month it was MSN
that
ate up over a gig in just a couple of weeks. Google is only third in
terms
of activity. Imagine that...

Tell me about it! This month I fed Yahoo Slurp alone 16.58 GB! My entire
human traffic according to Urchin was 20.8 GB this month, and that
includes pictures that bots don't download. Total bot traffic, however
(only including top 10) was 25.19GB so far this month. It is clearly
becoming bots feeding frenzy!



--
Cheers,
Dmitri
See Site Sig Below
-------------------------------------



--
##-----------------------------------------------##
Article posted with Web Developer's USENET Archive
http://www.1-script.com/forums/
Web and RSS gateway to your favorite newsgroup -
alt.internet.search-engines - 30138 messages and counting!
##-----------------------------------------------##

Worse here. The bots take over 75% of my bandwidth.

__/ [ Phil Payne ] on Sunday 30 April 2006 11:34 \__

__/ [ www.1-script.com ] on Friday 28 April 2006 21:08 \__

Roy Schestowitz wrote:

For the past week they have been crawling a lot. I haven't checked all
my
sites and I only noticed that last night. Earlier this month it was MSN
that
ate up over a gig in just a couple of weeks. Google is only third in
terms
of activity. Imagine that...

Tell me about it! This month I fed Yahoo Slurp alone 16.58 GB! My entire
human traffic according to Urchin was 20.8 GB this month, and that
includes pictures that bots don't download. Total bot traffic, however
(only including top 10) was 25.19GB so far this month. It is clearly
becoming bots feeding frenzy!

Worse here. The bots take over 75% of my bandwidth.

When you spread breadbrumbs (to analogize pages) all over your garden, the
crows will come. I think moderation in terms of scale is one way to prevent
crawlers from exploring. Another possibility is to indicate that pages are
static in the sense that they are no longer being extended. See:

http://www.i18nguy.com/markup/metatags.html

More crawling does not /necessarily/ mean more visitors. It's where crawling
efforts are concentrated (affect cycles/frequency of update) that matter.

With kind regards,

Roy

--
Roy S. Schestowitz | "Error, no keyboard - press F1 to continue"
http://Schestowitz.com | SuSE Linux ¦ PGP-Key: 0x74572E8E
11:40am up 2 days 18:37, 14 users, load average: 1.01, 1.09, 0.88
http://iuron.com - next generation of search paradigms