reply urgent

google spiders all pages of a website?? actually i use some pages for
just returning values or some specific purpose.They are not intended
to be spidered. While analyzing a site, do the optimizers analyze
individually all pages or only default or index page??
thx

"CreativeMind" <aftab.pucit (AT) gmail (DOT) com> wrote

google spiders all pages of a website?? actually i use some pages for
just returning values or some specific purpose.They are not intended
to be spidered. While analyzing a site, do the optimizers analyze
individually all pages or only default or index page??
thx

You need to upload a robots.txt file to your server.

The content should be something like:-

User-Agent: *

Disallow: /images/

which I use on a site to stop Google and others spidering my images folder.

hth
--
Rich
http://www.richdavies.com
http://www.richdavies.com/tomtom.htm


--
Posted via a free Usenet account from http://www.teranews.com

CreativeMind <aftab.pucit (AT) gmail (DOT) com> wrote:

google spiders all pages of a website?? actually i use some pages for
just returning values or some specific purpose.They are not intended
to be spidered.

The best option is to password protect such pages. There are plenty of
"spiders" that don't check robots.txt

--
John Bokma http://johnbokma.com/

You need to upload a robots.txt file to your server.

The content should be something like:-

User-Agent: *

Disallow: /images/

No.

The blank line after User-agent: delimits the record. It has to be:

User-Agent: *
Disallow: /images/

On Feb 20, 3:40*am, CreativeMind <aftab.pu... (AT) gmail (DOT) com> wrote:

google spiders all pages of a website?? actually i use some pages for
just returning values or some specific purpose.They are not intended
to be spidered. While analyzing a site, do the optimizers analyze
individually all pages or only default or index page??
thx

Whatever you do, when a visitor arrives at a page that was not
intented to attract him, make sure he can easily go to one that is.
Avoid redirects. Use hypelinkes text or graphics.