Can search engines find all files in a Web site, even if there are no links to them?

I use certain subdirectories (i.e., www.mydomain/proofs/) in my Web site to
put early drafts and proofs of Web pages for my clients to review. I don't
want search engines to find those Web pages, because they are drafts, and I
don't want to password protect them because my clients want quick access to
them. Thus I've made sure that I never link to those pages from my regular
site. I've been under the understanding that a search engine can't find Web
pages unless it finds a link to the Web pages, from other Web pages.

However, these pages end up in the search engines anyway. How is it that
search engines can find these pages, if nobody submits the URL to the search
engine, and no links lead to the page? Can search engines get a directory of
all the files that are within a certain domain, and then index them that
way? My Web hosting company claims that the names of files on a Web site
cannot be accessed by search engines, but that is the only way I can figure
out that the search engines are finding these pages.

How do search engines find pages that have no links going to them? Your help
is appreciated!!

--
DK

DK wrote:

I use certain subdirectories (i.e., www.mydomain/proofs/) in my Web
site to
put early drafts and proofs of Web pages for my clients to review. I
don't
want search engines to find those Web pages, because they are drafts,
and I
don't want to password protect them because my clients want quick
access to
them. Thus I've made sure that I never link to those pages from my
regular
site. I've been under the understanding that a search engine can't find
Web
pages unless it finds a link to the Web pages, from other Web pages.

However, these pages end up in the search engines anyway. How is it
that
search engines can find these pages, if nobody submits the URL to the
search
engine, and no links lead to the page? Can search engines get a
directory of
all the files that are within a certain domain, and then index them
that
way? My Web hosting company claims that the names of files on a Web
site
cannot be accessed by search engines, but that is the only way I can
figure
out that the search engines are finding these pages.

How do search engines find pages that have no links going to them? Your
help
is appreciated!!

Well, if you were using Google Toolbar with PR enabled and visited that
page, they'd know about the fact that the page exists. (check their
privacy disclaimer - they warned you!) So, your question should be
re-stated: "how do I ensure that some parts of my site are not listed
although the search engine knows it exists".
For something that it not a sensitive data but just something you don't
want to show to the world (like unfinished work that will be published
later) you can just exclude that directory (or even by file, individually)
by adding this to your robots.txt file:

User-Agent: *
Disallow: /cgi-bin/ # this is a service CGI directory
Disallow: /tmp/ # I'm not ready to publish it yet

Add all those directories that you do not want visible.

The robots.txt. obviously, should be uploaded into your site's root
document directory.

HOWEVER with that said, you should realize that you just gave all the bad
guys of the world the map to resources you don't want them to see! So, if
it's a sensitive data, it has to be password-protected. That would repel
the robots, too.


--
Cheers,
Dmitri
See Site Sig Below
-------------------------------------


--
##-----------------------------------------------##
Article posted with Web Developer's USENET Archive
http://www.1-script.com/forums
Web and RSS gateway to your favorite newsgroup -
alt.www.webmaster,alt.internet.search-engines - messages and counting!
##-----------------------------------------------##

On Fri, 18 Nov 2005 22:35:16 +0100, DK <none (AT) NoSpam (DOT) com> wrote:

How do search engines find pages that have no links going to them? Your
help is appreciated!!

Google toolbar.

OK. I will be less brief this time

It is absolutely enough that the page was visited by someone using
Google toolbar. Toolbar asks G for a PageRank, submitting URL. At
this moment Google knows URL and can index the page.

Best,
Borek
--
http://www.chembuddy.com
http://www.chembuddy.com/?left=BATE&...ion_equilibria
http://www.chembuddy.com/?left=CASC&...n_calcul ator

"Borek" <borek (AT) parts (DOT) bpp.to.com.remove.pl> wrote

On Fri, 18 Nov 2005 22:35:16 +0100, DK <none (AT) NoSpam (DOT) com> wrote:

How do search engines find pages that have no links going to them? Your
help is appreciated!!

Google toolbar.

OK. I will be less brief this time

It is absolutely enough that the page was visited by someone using
Google toolbar. Toolbar asks G for a PageRank, submitting URL. At
this moment Google knows URL and can index the page.

Best,
Borek

So, one can assume that anyone loading a page in a browser with google
toolbar, page rank on, will auto submit to google? No need for submission
sites, services or even visiting googley's own addurl page anymore. Cool.

Hmmm ... c:/allis/sites/stoopid-stuff/*.htm .... hmmmm....

Oh no!
c:/allis/remove/googletoolbar

On Fri, 18 Nov 2005 14:52:08 -0800, "Allis" <allis@allis> wrote:

"Borek" <borek (AT) parts (DOT) bpp.to.com.remove.pl> wrote in message
newsp.s0gjaxrd584cds (AT) borek (DOT) ..
On Fri, 18 Nov 2005 22:35:16 +0100, DK <none (AT) NoSpam (DOT) com> wrote:

How do search engines find pages that have no links going to them? Your
help is appreciated!!

Google toolbar.

OK. I will be less brief this time

It is absolutely enough that the page was visited by someone using
Google toolbar. Toolbar asks G for a PageRank, submitting URL. At
this moment Google knows URL and can index the page.

Best,
Borek

So, one can assume that anyone loading a page in a browser with google
toolbar, page rank on, will auto submit to google? No need for submission
sites, services or even visiting googley's own addurl page anymore. Cool.

Been that way for some time.

Hmmm ... c:/allis/sites/stoopid-stuff/*.htm .... hmmmm....

Oh no!

Too late! Oh, we laughed at that one!

c:/allis/remove/googletoolbar

Better late than never.

BB

--
www.kruse.co.uk/ seo (AT) kruse (DOT) demon.co.uk
The buffalo have gone

Allis wrote:

"Borek" <borek (AT) parts (DOT) bpp.to.com.remove.pl> wrote in message
newsp.s0gjaxrd584cds (AT) borek (DOT) ..
On Fri, 18 Nov 2005 22:35:16 +0100, DK <none (AT) NoSpam (DOT) com> wrote:

How do search engines find pages that have no links going to them? Your
help is appreciated!!

It is absolutely enough that the page was visited by someone using
Google toolbar. Toolbar asks G for a PageRank, submitting URL. At
this moment Google knows URL and can index the page.

c:/allis/remove/googletoolbar

Won't help if one of your clients uses that toolbar when checking the
pages.

--
Els http://locusmeus.com/
Sonhos vem. Sonhos vão. O resto é imperfeito.
- Renato Russo -

"Els" <els.aNOSPAM (AT) tiscali (DOT) nl> wrote

Allis wrote:
"Borek" <borek (AT) parts (DOT) bpp.to.com.remove.pl> wrote in message
newsp.s0gjaxrd584cds (AT) borek (DOT) ..
On Fri, 18 Nov 2005 22:35:16 +0100, DK <none (AT) NoSpam (DOT) com> wrote:

How do search engines find pages that have no links going to them?
Your
help is appreciated!!

It is absolutely enough that the page was visited by someone using
Google toolbar. Toolbar asks G for a PageRank, submitting URL. At
this moment Google knows URL and can index the page.

c:/allis/remove/googletoolbar

Won't help if one of your clients uses that toolbar when checking the
pages.

Els ... I wuv you girl!
Absolutely!

Allis
http://themooseisloose.net/outkast.html

"Allis" <allis@allis> wrote:

"Borek" <borek (AT) parts (DOT) bpp.to.com.remove.pl> wrote in message
newsp.s0gjaxrd584cds (AT) borek (DOT) ..
On Fri, 18 Nov 2005 22:35:16 +0100, DK <none (AT) NoSpam (DOT) com> wrote:

How do search engines find pages that have no links going to them?
Your help is appreciated!!

Google toolbar.

OK. I will be less brief this time

It is absolutely enough that the page was visited by someone using
Google toolbar. Toolbar asks G for a PageRank, submitting URL. At
this moment Google knows URL and can index the page.

Best,
Borek

So, one can assume that anyone loading a page in a browser with google
toolbar, page rank on, will auto submit to google? No need for
submission sites, services or even visiting googley's own addurl page
anymore. Cool.

So what else is new? :-D.

Also, if you don't want your pages to end up in SEs, put them behind a
password.


--
John Perl SEO tools: http://johnbokma.com/perl/
or have them custom made
Experienced (web) developer: http://castleamber.com/

Mark Goodge <usenet (AT) listmail (DOT) good-stuff.co.uk> wrote:

On 19 Nov 2005 05:12:57 GMT, John Bokma put finger to keyboard and
typed:

....

Also, if you don't want your pages to end up in SEs, put them behind a
password.

You can use robots.txt to exclude them from pretty much all of the
major search engines.

Pretty much all might not be enough. Also note that Google for example
caches robots.txt, and hence might miss your change, and presto.

If all you want is to avoid drafts being indexed
(as opposed to keeping the material confidential), then that's fine.

I wouldn't rely on it.

Drafts I keep on my own computer :-D.

--
John Perl SEO tools: http://johnbokma.com/perl/
or have them custom made
Experienced (web) developer: http://castleamber.com/

How do search engines find pages that have no links going to them? Your
help is appreciated!!

Are you 100% sure there aren't any old links / redirections?
A good sitemap / website analyser utility can probably tell you.
[if you're interested, I can send you a download link to the beta of:
http://www.micro-sys.dk/products/sit...reenshots.php]


best regards
Thomas