Re: Warning: robots.txt unreliable in Apache servers

In comp.infosystems.www.authoring.html, "Stan Brown" wrote:

Wouldn't it be more effective to have any URL containing http://.*//
return a 403 Forbidden or a 404 Not Found? This could be done in
.htaccess or perhaps httpd.conf. I may be having a failure of
imagination, but I can't think of any legitimate reason for such a
link.

That would also be effective, but maybe it's better to do something useful
with the URL if you can.

Most servers will redirect to a URL with a trailing slash when the name of a
directory is requested. Why not treat multiple slashes in a similar way?

Besides, it might help in terms of page rank.

[[Crossposted to alt.internet.search-engines, with apologies to Nick]]

--
phil [dot] ronan @ virgin [dot] net
http://vzone.virgin.net/phil.ronan/