spidering Amazon website

Having a large list of books, I wanted to spider data from Amazon
(isbns, reviews...)
Do you know if Amazon accepts kindly this behavior?
And, just in case, which timeout between requests do you think is
acceptable?


Thanks for any info!

Alessandro

On Thu, 6 Dec 2007 07:03:52 -0800 (PST), "alexxx.magni (AT) gmail (DOT) com"
<alexxx.magni (AT) gmail (DOT) com> wrote:

Having a large list of books, I wanted to spider data from Amazon
(isbns, reviews...)
Do you know if Amazon accepts kindly this behavior?
And, just in case, which timeout between requests do you think is
acceptable?


Thanks for any info!

Alessandro

I'm thinking you might have got this wrong somehow... what is it you
want to do exactly?

BB
--

http://www.kruse.co.uk/
http://www.fat-odin.com/
http://www.here-be-posters.co.uk/

"alexxx.magni (AT) gmail (DOT) com" <alexxx.magni (AT) gmail (DOT) com> wrote in news:df50f25d-
6845-4c06-a900-e39801e869f1 (AT) e23...oglegroups.com:

Having a large list of books, I wanted to spider data from Amazon
(isbns, reviews...)
Do you know if Amazon accepts kindly this behavior?
And, just in case, which timeout between requests do you think is
acceptable?


Thanks for any info!

Alessandro

Why not just run your software and see if the bot UA is denied or allowed?

There are tons and tons of bots that are spidering sites, in which the
spidered sites fail to insall any prevention against spidering.

thank you for the reply...
The bot is running already, their robots.txt does not deny it:
my problem was to find in which way to let it behave correctly, not
too hammer the site too much (although, being such a large site as
Amazon, I dont think I'm doing too harm...).
Currently I wait a random time between 3 and 30 seconds between
requests - I dont know if it's too high / too low


Alessandro


On Dec 7, 5:32 am, Don <lostinsp... (AT) 123-universe (DOT) com> wrote:

"alexxx.ma... (AT) gmail (DOT) com" <alexxx.ma... (AT) gmail (DOT) com> wrote in news:df50f25d-
6845-4c06-a900-e39801e86... (AT) e23g2000prf (DOT) googlegroups.com:

Having a large list of books, I wanted to spider data from Amazon
(isbns, reviews...)
Do you know if Amazon accepts kindly this behavior?
And, just in case, which timeout between requests do you think is
acceptable?

Thanks for any info!

Alessandro

Why not just run your software and see if the bot UA is denied or allowed?

There are tons and tons of bots that are spidering sites, in which the
spidered sites fail to insall any prevention against spidering.

"alexxx.magni (AT) gmail (DOT) com" <alexxx.magni (AT) gmail (DOT) com> wrote in
news:66592aa2-8467-4416-b032-f16dd9e17405 (AT) j20g2000hsi (DOT) googlegroups.com:

thank you for the reply...
The bot is running already, their robots.txt does not deny it:
my problem was to find in which way to let it behave correctly, not
too hammer the site too much (although, being such a large site as
Amazon, I dont think I'm doing too harm...).
Currently I wait a random time between 3 and 30 seconds between
requests - I dont know if it's too high / too low


Alessandro


On Dec 7, 5:32 am, Don <lostinsp... (AT) 123-universe (DOT) com> wrote:
"alexxx.ma... (AT) gmail (DOT) com" <alexxx.ma... (AT) gmail (DOT) com> wrote in
news:df50f25d-
6845-4c06-a900-e39801e86... (AT) e23g2000prf (DOT) googlegroups.com:

Having a large list of books, I wanted to spider data from Amazon
(isbns, reviews...)
Do you know if Amazon accepts kindly this behavior?
And, just in case, which timeout between requests do you think is
acceptable?

Thanks for any info!

Alessandro

Why not just run your software and see if the bot UA is denied or
allowed?

There are tons and tons of bots that are spidering sites, in which
the
spidered sites fail to insall any prevention against spidering.

There's a delay inclusion in robots.txt

User-agent: *
Crawl-delay: xx

Here's a Yahoo link which provides that explanation
http://help.yahoo.com/l/us/yahoo/sea.../slurp-03.html

where did you find it??
www.amazon.com/robots.txt:

# Disallow all crawlers access to certain pages.

User-agent: *
Disallow: /exec/obidos/account-access-login
Disallow: /exec/obidos/change-style
Disallow: /exec/obidos/flex-sign-in
Disallow: /exec/obidos/handle-buy-box
Disallow: /exec/obidos/tg/cm/member
Disallow: /gp/cart
Disallow: /gp/flex
Disallow: /gp/product/e-mail-friend
Disallow: /gp/product/product-availability
Disallow: /gp/product/rate-this-item
Disallow: /gp/sign-in
Disallow: /gp/reader
Disallow: /gp/sitbv3/reader
Disallow: /gp/richpub/syltguides/create
Disallow: /gp/customer-media
Disallow: /gp/gfix
Disallow: /gp/associations/wizard.html
Disallow: /gp/dmusic/order
Disallow: /gp/legacy-handle-buy-box.html
Disallow: /gp/aws/ssop
Disallow: /gp/yourstore
Disallow: /gp/gift-central/organizer/add-wishlist
Disallow: /gp/gurupamacro
Disallow: /gp/vote
Disallow: /gp/music/wma-pop-up
Disallow: /gp/customer-images


Alessandro

Don wrote:

"alexxx.magni (AT) gmail (DOT) com" <alexxx.magni (AT) gmail (DOT) com> wrote in
news:66592aa2-8467-4416-b032-f16dd9e17405 (AT) j20g2000hsi (DOT) googlegroups.com:

thank you for the reply...
The bot is running already, their robots.txt does not deny it:
my problem was to find in which way to let it behave correctly, not
too hammer the site too much (although, being such a large site as
Amazon, I dont think I'm doing too harm...).
Currently I wait a random time between 3 and 30 seconds between
requests - I dont know if it's too high / too low


Alessandro


On Dec 7, 5:32 am, Don <lostinsp... (AT) 123-universe (DOT) com> wrote:
"alexxx.ma... (AT) gmail (DOT) com" <alexxx.ma... (AT) gmail (DOT) com> wrote in
news:df50f25d-
6845-4c06-a900-e39801e86... (AT) e23g2000prf (DOT) googlegroups.com:

Having a large list of books, I wanted to spider data from Amazon
(isbns, reviews...)
Do you know if Amazon accepts kindly this behavior?
And, just in case, which timeout between requests do you think is
acceptable?

Thanks for any info!

Alessandro

Why not just run your software and see if the bot UA is denied or
allowed?

There are tons and tons of bots that are spidering sites, in which
the
spidered sites fail to insall any prevention against spidering.


There's a delay inclusion in robots.txt

User-agent: *
Crawl-delay: xx

Here's a Yahoo link which provides that explanation
http://help.yahoo.com/l/us/yahoo/sea.../slurp-03.html

"alexxx.magni (AT) gmail (DOT) com" <alexxx.magni (AT) gmail (DOT) com> wrote in news:bffac00f-
df5b-4458-804e-48b0bca90d0c (AT) w28...oglegroups.com:

where did you find it??
www.amazon.com/robots.txt:

# Disallow all crawlers access to certain pages.

User-agent: *
Disallow: /exec/obidos/account-access-login
snip

</snip>

Disallow: /gp/customer-images


Alessandro

Don wrote:
"alexxx.magni (AT) gmail (DOT) com" <alexxx.magni (AT) gmail (DOT) com> wrote in
news:66592aa2-8467-4416-b032-f16dd9e17405
@j20g2000hsi.googlegroups.com:

thank you for the reply...
The bot is running already, their robots.txt does not deny it:
my problem was to find in which way to let it behave correctly, not
too hammer the site too much (although, being such a large site as
Amazon, I dont think I'm doing too harm...).
Currently I wait a random time between 3 and 30 seconds between
requests - I dont know if it's too high / too low


Alessandro


On Dec 7, 5:32 am, Don <lostinsp... (AT) 123-universe (DOT) com> wrote:
"alexxx.ma... (AT) gmail (DOT) com" <alexxx.ma... (AT) gmail (DOT) com> wrote in
news:df50f25d-
6845-4c06-a900-e39801e86... (AT) e23g2000prf (DOT) googlegroups.com:

Having a large list of books, I wanted to spider data from Amazon
(isbns, reviews...)
Do you know if Amazon accepts kindly this behavior?
And, just in case, which timeout between requests do you think is
acceptable?

Thanks for any info!

Alessandro

Why not just run your software and see if the bot UA is denied or
allowed?

There are tons and tons of bots that are spidering sites, in which
the
spidered sites fail to insall any prevention against spidering.


There's a delay inclusion in robots.txt

User-agent: *
Crawl-delay: xx

Here's a Yahoo link which provides that explanation
http://help.yahoo.com/l/us/yahoo/sea.../slurp-03.html

I was merely suggesting that since their robots was absent of a crawl
delay that it was not an issue for their websites.