Will Googlebit crawl non-indexed files?

Hi, I have a database that generates stories and an associated image
that I wish to have indexed by Google and Google Images. Here is an
example: http://www.nufocusinc.com/sightings/...ons_KAN_3.html

These pages are autogenerated in the /sightings/ directory and I do
not want to have to maintain an index.html file that lists all the
files.

- My site root is successfully being crawled.

- My directory listing access in Apache for /sightings/ is forbidden
as in:
http://www.nufocusinc.com/sightings/

- I have added a link in my home page to a dummy file:
http://www.nufocusinc.com/sightings/google.html

So are questions are:
1) Do I have to allow directory listing access to that directory for
googlebot?
2) Is the dummy file idea of any use?
3) Do I have to have an index.html file in /sightings/ ?

My goal is that the pages that are placed in the /sightings/ directory
are crawled and indexed by googlebot.

google will not spider through the newly generated content but may be
possible after some specific period. Actually i have the same case.i m
also looking for ur question's answer.


On Feb 21, 5:35 pm, dgarr... (AT) nufocusinc (DOT) com wrote:

Hi, I have a database that generates stories and an associated image
that I wish to have indexed by Google and Google Images. Here is an
example:http://www.nufocusinc.com/sightings/...ons_KAN_3.html

These pages are autogenerated in the /sightings/ directory and I do
not want to have to maintain an index.html file that lists all the
files.

- My site root is successfully being crawled.

- My directory listing access in Apache for /sightings/ is forbidden
as in:
http://www.nufocusinc.com/sightings/

- I have added a link in my home page to a dummy file:http://www.nufocusinc.com/sightings/google.html

So are questions are:
1) Do I have to allow directory listing access to that directory for
googlebot?
2) Is the dummy file idea of any use?
3) Do I have to have an index.html file in /sightings/ ?

My goal is that the pages that are placed in the /sightings/ directory
are crawled and indexed by googlebot.

dgarrard (AT) nufocusinc (DOT) com schreef:

Hi, I have a database that generates stories and an associated image
that I wish to have indexed by Google and Google Images. Here is an
example: http://www.nufocusinc.com/sightings/...ons_KAN_3.html

These pages are autogenerated in the /sightings/ directory and I do
not want to have to maintain an index.html file that lists all the
files.

- My site root is successfully being crawled.

- My directory listing access in Apache for /sightings/ is forbidden
as in:
http://www.nufocusinc.com/sightings/

- I have added a link in my home page to a dummy file:
http://www.nufocusinc.com/sightings/google.html

So are questions are:
1) Do I have to allow directory listing access to that directory for
googlebot?

NO

2) Is the dummy file idea of any use?

NO

3) Do I have to have an index.html file in /sightings/ ?

YES. Preferably with links. The bot follows links, nothing else. It
might pick up textual placed URl's and try to access them.

My goal is that the pages that are placed in the /sightings/ directory
are crawled and indexed by googlebot.

Then make a page where the bot can pick them up.

--
Webdesign: http://vision2form.nl/webontwerp/
Korte handleiding zoekmachine optimalisatie / gevonden worden:
http://vision2form.nl/webontwerp/gevonden-worden.html
Lifestyle - wonen reizen en genieten : http://vision4living.com

dgarrard (AT) nufocusinc (DOT) com wrote:

Hi, I have a database that generates stories and an associated image
that I wish to have indexed by Google and Google Images. Here is an
example: http://www.nufocusinc.com/sightings/...ons_KAN_3.html

You might want to name your files smarter, for example:

/sightings/orange-chat-aurifrons.html

or better:

/sightings/orange-chat-epthianura-aurifrons.html

If the 3 is required (for ID, or because there are more orange chat
sightings):

/sightings/orange-chat-epthianura-aurifrons-3.html


* Don't use _
* Don't use CAPS

in the filename, though


Use h1 instead of h2 if it's the first heading on the page (which it
is).

You might want to link to other pages then the "click here" software,
even if your site is just set up to feed many pages to Google in order
to promote the software.

You might want to relate to similar or related species, for example. Or
an overview.

These pages are autogenerated in the /sightings/ directory and I do
not want to have to maintain an index.html file that lists all the
files.

Simple answer: generate that page. It's not that hard.

- My site root is successfully being crawled.

- My directory listing access in Apache for /sightings/ is forbidden
as in:
http://www.nufocusinc.com/sightings/

- I have added a link in my home page to a dummy file:
http://www.nufocusinc.com/sightings/google.html

So, google will find a page with hardly any content, which looks spammy,
and no links.

So are questions are:
1) Do I have to allow directory listing access to that directory for
googlebot?

What you need is links to every page in sightings. The best way to do
this is by making a index page which several category pages instead of
an index page which links to each and every other page. On the index
page you link to, say 25, category pages, which should be of use to your
visitors so they can pick fast the right category for a bird. On each
category page you put links to birds in that category.

2) Is the dummy file idea of any use?

No. Google sees no links on it. It *can't* find pages unless there is a
link somewhere to a page.

3) Do I have to have an index.html file in /sightings/ ?

Yes

My goal is that the pages that are placed in the /sightings/ directory
are crawled and indexed by googlebot.

So you need links to those pages. If you configure Apache to auto
generate a index for you (one link for each file) you have the worst way
of linking, but it's better than nothing (i.e. having a google.html page
with no links and some keywords as content, even the filename is
pointless and misleading).

--
John Bokma http://johnbokma.com/