Can search engines find all files in a Web site, even if there are no links to them?

Borek wrote:

On Fri, 18 Nov 2005 22:35:16 +0100, DK <none (AT) NoSpam (DOT) com> wrote:

How do search engines find pages that have no links going to them? Your
help is appreciated!!

Google toolbar.

Or Opera with advertising.
Or if there's a link to an external site which is clicked by someone - your
site will appear is the referer log of this external site, and it's up to
its webmaster to do anything they like with you URL, e.g. post their
referer logs on the web for search engines to crawl.

robots.txt might be a sufficient solution. If you want something more
reliably, use password protection and give the password only to your
clients.

--
Benjamin Niemann
Email: pink at odahoda dot de
WWW: http://www.odahoda.de/

Hi,

DK wrote:

How do search engines find pages that have no links going to them?

I do not know how your pages were found by search engines, but...
maybe you can get some idea from your access logs, to see if/what bots
accessed your drafts pages.
Some web statistics for some websites are on public access pages
and your pages could have been found from stats like these,
from your website
or from other websites (with your pages as referrers if they have
click-able and click-ed links).
If you do not want your draft pages to have restricted access (password
protected),
then you can hope for the best by disallowing access to compliant
robots from the robots.txt file,
also you can put big titles or markings on the pages
indicating that they are drafts, temporary etc.

Els wrote:

Allis wrote:

Borek wrote...

DK wrote:

How do search engines find pages that have no links going to them?

It is absolutely enough that the page was visited by someone using
Google toolbar. Toolbar asks G for a PageRank, submitting URL. At
this moment Google knows URL and can index the page.

c:/allis/remove/googletoolbar

Won't help if one of your clients uses that toolbar when checking the
pages.

Even if they don't, the next page that your visitor goes to might get
the URL from its referrer logs, and some sites put referrer statistics
on pages that search engines then index.

See [ http://en.wikipedia.org/wiki/Referer_spam ].

Now think about he case where that next page that gets the referrer
info happens to be Google...

Guy Macon wrote:

Els wrote:
Allis wrote:
Borek wrote...
DK wrote:

How do search engines find pages that have no links going to them?

It is absolutely enough that the page was visited by someone using
Google toolbar. Toolbar asks G for a PageRank, submitting URL. At
this moment Google knows URL and can index the page.

c:/allis/remove/googletoolbar

Won't help if one of your clients uses that toolbar when checking
the pages.

Even if they don't, the next page that your visitor goes to might get
the URL from its referrer logs,

No, that would only be the case if there were a link to that next
page. I don't know about other developers, but I don't put external
links on drafts for clients' viewing. That would be plain stupid if
you ask me...

--
Els http://locusmeus.com/
Sonhos vem. Sonhos vão. O resto é imperfeito.
- Renato Russo -

Els wrote:

I don't know about other developers, but I don't put external
links on drafts for clients' viewing. That would be plain
stupid if you ask me...

Why? If I am a customer and I am looking at a draft in order to
approve it, I would want to check that any external links are
error-free before giving my approval. I might also send a link
to others for comments. and I wouldn't consider the URL to be
particlarly sensitive/secret information. Then again, every web
developer I have ever hired protected drafts with passwords, not
with security through obsurity.

Guy Macon wrote:

Els wrote:

I don't know about other developers, but I don't put external
links on drafts for clients' viewing. That would be plain
stupid if you ask me...

Why? If I am a customer and I am looking at a draft in order to
approve it, I would want to check that any external links are
error-free before giving my approval.

Sounds like the later stages of development. Not when the pages are
still being designed and the content still consists of Lorem Ipsums.

I might also send a link
to others for comments. and I wouldn't consider the URL to be
particlarly sensitive/secret information.

If the URL isn't secret/sensitive info, there wouldn't be a problem
with Google indexing it I think?

Then again, every web
developer I have ever hired protected drafts with passwords, not
with security through obsurity.

I only do that on sensitive information, not on design drafts really.

--
Els http://locusmeus.com/
Sonhos vem. Sonhos vão. O resto é imperfeito.
- Renato Russo -

Els <els.aNOSPAM (AT) tiscali (DOT) nl> wrote:

I only do that on sensitive information, not on design drafts really.

Those are not always mutually exclusive.

Oh, and for the external links: one could use a redirecion script.

--
John Perl SEO tools: http://johnbokma.com/perl/
or have them custom made
Experienced (web) developer: http://castleamber.com/

Thanks, everyone for the comments. I checked, and my client uses Google
Toolbar with PageRank enabled, so that explains why these pages were being
indexed, when they otherwise would not have been. Thanks for all your help
and for figuring out the solution!!

--
DK