banning spiders from cgi-bins etc

Is there really any point? I'm going through the help files for Robot
Manager and wondering what the oint of using it is apart from
registering spider visits and indexings etc. Does our technical team
think there's any genuine reason for disallowing spider access to a
file? Isn't it a whole lot easier to just not have files with
sensitive info on the web in the first place?

BB
--
www.kruse.co.uk SEO (AT) kruse (DOT) demon.co.uk
home of SEO that's shiny!
--

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> wrote

Is there really any point? I'm going through the help files for Robot
Manager and wondering what the oint of using it is apart from
registering spider visits and indexings etc. Does our technical team
think there's any genuine reason for disallowing spider access to a
file? Isn't it a whole lot easier to just not have files with
sensitive info on the web in the first place?

BB
--
www.kruse.co.uk SEO (AT) kruse (DOT) demon.co.uk
home of SEO that's shiny!
--

There are other reasons as well - do you really want spiders downloading
your .exe/.zip files if you sell software?

--
John Dingley

On Sat, 8 Jan 2005 09:19:02 -0000, "John Dingley"
<john (AT) notrealurl (DOT) dot.com> wrote:

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> wrote in message
news:biput0lku2isb2k3r1a88hni0gbihhtlgd (AT) 4ax (DOT) com...
Is there really any point? I'm going through the help files for Robot
Manager and wondering what the oint of using it is apart from
registering spider visits and indexings etc. Does our technical team
think there's any genuine reason for disallowing spider access to a
file? Isn't it a whole lot easier to just not have files with
sensitive info on the web in the first place?

BB
--
www.kruse.co.uk SEO (AT) kruse (DOT) demon.co.uk
home of SEO that's shiny!
--

There are other reasons as well - do you really want spiders downloading
your .exe/.zip files if you sell software?

Good point. Where would they be downloading them to though? The
engine's index? And if they were behind some kind of secure system how
would a robot negotiate that anyhow?

BB


--
www.kruse.co.uk SEO (AT) kruse (DOT) demon.co.uk
home of SEO that's shiny!
--

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> wrote

On Sat, 8 Jan 2005 09:19:02 -0000, "John Dingley"
john (AT) notrealurl (DOT) dot.com> wrote:

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> wrote in message
news:biput0lku2isb2k3r1a88hni0gbihhtlgd (AT) 4ax (DOT) com...
Is there really any point? I'm going through the help files for Robot
Manager and wondering what the oint of using it is apart from
registering spider visits and indexings etc. Does our technical team
think there's any genuine reason for disallowing spider access to a
file? Isn't it a whole lot easier to just not have files with
sensitive info on the web in the first place?

BB
--
www.kruse.co.uk SEO (AT) kruse (DOT) demon.co.uk
home of SEO that's shiny!
--

There are other reasons as well - do you really want spiders downloading
your .exe/.zip files if you sell software?

Good point. Where would they be downloading them to though? The
engine's index? And if they were behind some kind of secure system how
would a robot negotiate that anyhow?

BB


--
www.kruse.co.uk SEO (AT) kruse (DOT) demon.co.uk
home of SEO that's shiny!
--

Yes - in google's case it would stop at 100K but some do the whole lot.

Plus if your site has cgi scripts etc you really don't want these spidered
either, just uses up your bandwidth

On Sat, 8 Jan 2005 12:23:52 -0000, "John Dingley"
<john (AT) notrealurl (DOT) dot.com> wrote:

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> wrote in message
news:ekavt0trjcfcth1hhchthkos6gerv93hbc (AT) 4ax (DOT) com...
On Sat, 8 Jan 2005 09:19:02 -0000, "John Dingley"
john (AT) notrealurl (DOT) dot.com> wrote:

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> wrote in message
news:biput0lku2isb2k3r1a88hni0gbihhtlgd (AT) 4ax (DOT) com...
Is there really any point? I'm going through the help files for Robot
Manager and wondering what the oint of using it is apart from
registering spider visits and indexings etc. Does our technical team
think there's any genuine reason for disallowing spider access to a
file? Isn't it a whole lot easier to just not have files with
sensitive info on the web in the first place?

BB
--
www.kruse.co.uk SEO (AT) kruse (DOT) demon.co.uk
home of SEO that's shiny!
--

There are other reasons as well - do you really want spiders downloading
your .exe/.zip files if you sell software?

Good point. Where would they be downloading them to though? The
engine's index? And if they were behind some kind of secure system how
would a robot negotiate that anyhow?

BB


--
www.kruse.co.uk SEO (AT) kruse (DOT) demon.co.uk
home of SEO that's shiny!
--

Yes - in google's case it would stop at 100K but some do the whole lot.

Plus if your site has cgi scripts etc you really don't want these spidered
either, just uses up your bandwidth

Saving bandwidth is another point.
Any body else?

BB

--
www.kruse.co.uk SEO (AT) kruse (DOT) demon.co.uk
home of SEO that's shiny!
--

Blocking bad robots also makes it just a wee bit more difficult for my
entire web site to show up at a new URL.

It also makes it a wee bit more difficult for e-mail spammers to
search for e-mail addresses on my web sites.

And, of course, offline-browsers usually are *not* clicking my ads.



I just updated my robots.txt with the new set of bad robots documented
here:

http://www.internet-search-engines-f...d-robots.shtml