Need Advice on Site Map or Putting Page URLs close to Hoem Page

Several questions about getting all of my web site's pages spidered
and indexed.

1. After reading messages in this newsgroup, I would guess it would
help my chances of getting pages spidered and indexed if they are as
close to the home page as possible. In other orders have as few links
as possible between the home page and the content page.

Is this true? (I counted some of the links on one site, and for many
of the pages, you have to click through 9 links to get to them. I am
working on a better organization, but in some cases, which I will not
go into, I may not be able to reduce the number of links between the
home page and a page I want spidered.)

2. Next I need to make a site map for my web sites. The only problem
is two of the sites have over a thousands of pages. So for ease of
use to the web visitor, I have a collapsible way of presenting a site
map.

For example, the first page of the site map shows something like
this:

Home
Cars
Boats
Planes

Then a visitor can click on a topic, like Cars , and the visitor sees
a new page like this:

Home
Cars
Chevy
Ford
Chrysler
Boats
Planes

And they can keep clicking to review the sections they want, and
eventually they get to the web pages and urls.

Will this type of site map work well to get pages indexed?

3. Or should I create a site map that lists every page?

In other words would this be a better site map for Search Engine
spidering and indexing:

Chevy Corvette
Chevy Malibu
Ford Mustang
Chrysler PT Cuiser
etc.

(By the way, I will still offer the "organized" site map since I think
this will be easier to use for human visitors.)

4. I have also heard that having links from every page on the site to
every other page helps. Someone suggested to me to use the following
as a means to do this:

"On every page, I have this code at the end of the page:

<div style="display : none"><a href="page listing urls">Overview for
our SearchEngines</a> </div>

The "page listing urls" of course points to a page where all urls are
listed."

Is this a good practice? And will the spiders follow the link to "url
of listing" if it is inside <div style="display : none"> </div>?

5. Finally, assuming the answer to number 4 is yes and I should do a
page listing of urls, how do other people handle long lists of urls,
as in thousands of urls? My database software limits me to 500 lines
in a list, so I can only display 500 urls at a time. It also gives me
a "Next" link or button to go to the next 500 items. Will this be
sufficient for search engines, or should I try to figure a workaround
to get all of the pages on my sites onto one massive web page?

Thanks for your thoughts?

On Mon, 03 May 2004 16:05:07 GMT, Al F <bscinc3000 (AT) Yahoo_NoSpam (DOT) com>
wrote:

Several questions about getting all of my web site's pages spidered
and indexed.

1. After reading messages in this newsgroup, I would guess it would
help my chances of getting pages spidered and indexed if they are as
close to the home page as possible. In other orders have as few links
as possible between the home page and the content page.

Is this true? (I counted some of the links on one site, and for many
of the pages, you have to click through 9 links to get to them. I am
working on a better organization, but in some cases, which I will not
go into, I may not be able to reduce the number of links between the
home page and a page I want spidered.)

Well, according to an old rule of thumb for web design/site layouts
thoughts - you want to have most of the content within two to three
clicks from the main page for the person coming to the information
being sought.

Some people will drill their way down deeper into a page but others
may start thinking "where is suchasuch information at on this site? Do
they even have anything about suchasuch?" and possibly give up trying
to find it after 5 or so clicks.

Not knowing how your site is laid out - one suggestion from my side,
since you mentioned that some sections are just "deep" and may not be
able to made "less deep", is to try to cross-link to those deeper
pages from the "closer to the top" pages. Such as through the linking
method Cat uses on her site within the body of content or perhaps
"Some Related Pages:" type listing at the bottom of the page. May not
be able to link all the deeper content pages that way but may help out
on some of them. I would also list, no matter how deep the pages are,
an easy way back to the main page and, possibly, to the site map
itself that people can use for navigation means.

In terms of Search Engine Spiders, I read differing opinions about
"how deep a spider will go"; some say the spiders won't go much beyond
three levels deep - others say spiders slack off around 5 levels deep
.... then some say it doesn't matter as long as there links for it to
follow to that deeply shared content.

2. Next I need to make a site map for my web sites. The only problem
is two of the sites have over a thousands of pages. So for ease of
use to the web visitor, I have a collapsible way of presenting a site
map.

For example, the first page of the site map shows something like
this:

Home
Cars
Boats
Planes

Then a visitor can click on a topic, like Cars , and the visitor sees
a new page like this:

Home
Cars
Chevy
Ford
Chrysler
Boats
Planes

And they can keep clicking to review the sections they want, and
eventually they get to the web pages and urls.

Will this type of site map work well to get pages indexed?

Is it scripted for this expanding thought - does it take the person to
a new page for the expansion of sections? If scripted, then that leads
back to the debate about search engines spiders sometimes having
difficulty with javascript or such thoughts.

3. Or should I create a site map that lists every page?

In other words would this be a better site map for Search Engine
spidering and indexing:

Chevy Corvette
Chevy Malibu
Ford Mustang
Chrysler PT Cuiser
etc.

(By the way, I will still offer the "organized" site map since I think
this will be easier to use for human visitors.)

What I did on my sites is have a simple site map that is divided into
sections of thought; starting from the index.html and on through.
Below is a bad paraphrasing:

* Main Page
.... Lists more recent content <-- anchor text back to main page
* Indexes
.... Generic listing of all content on site <--- anchor text to
"articles listing" page
..... Yearly Collections:
........ Year 1: brief spiel <--- anchor text to that index/list of
contents
........ Year 2: brief spiel <--- anchor text to that index/list of
contents
........ Year 3: brief spiel <--- anchor text to that index/list of
contents
........ Year 4: brief spiel <--- anchor text to that index/list of
contents
........ etc.
* About the site: information, guidelines, etc.
.... Page 1 if needed <-- anchor text to that page
.... Page 2 if needed <-- anchor text to that page
.... etc.
* Contact Information
.... Online form <-- anchor text to that page
.... Page 1 if needed <-- anchor text to that page
.... etc.
* Resources shared: links to other sites shared
.... Page 1 <-- anchor text to that page
.... Page 2 if needed <-- anchor text to that page
.... etc.
* Recommendations: Amazon.com and such links shared
.... Affiliate 1 <-- anchor text to that page
.... Affiliate 2 <-- anchor text to that page
.... etc.
* Site Map: this page

The "generic table of contents" lists all the inner page's links - so
could be viewed as a more detailed site map. I am not really concerned
about length of it - it is more so there for my information and
reference versus being provided for search engine spiders or people to
scroll their way through. Most people prefer the simpler inner index
pages that have fewer URLs [50 or under] listed in a way I can share a
brief teaser paragraph about those links.

4. I have also heard that having links from every page on the site to
every other page helps. Someone suggested to me to use the following
as a means to do this:

"On every page, I have this code at the end of the page:

div style="display : none"><a href="page listing urls">Overview for
our SearchEngines</a> </div

Display none means the text is hidden so that would be viewed as a
hidden link which some view negatively as a practice.

There are so many simpler ways to share links to inner pages, in my
opinion, that hidden links is not really necessary method to employ.

If nothing else, share a link to the "organized site map" and let the
spider follow that. Then you have the link visible for users of your
site to be able to use also.

The "page listing urls" of course points to a page where all urls are
listed."

Is this a good practice? And will the spiders follow the link to "url
of listing" if it is inside <div style="display : none"> </div>?

Some sites have been said to been penalized for using hidden text/link
methods. If you know about the trick - search engines likely know
about it too so they aren't in the dark.

5. Finally, assuming the answer to number 4 is yes and I should do a
page listing of urls, how do other people handle long lists of urls,
as in thousands of urls? My database software limits me to 500 lines
in a list, so I can only display 500 urls at a time. It also gives me
a "Next" link or button to go to the next 500 items. Will this be
sufficient for search engines, or should I try to figure a workaround
to get all of the pages on my sites onto one massive web page?

A thousand URLs on one page may risk having the lower shared URLs not
followed; Google advises to keep listing of links under 100 per page.

What I would do, while bearing in mind that I do not claim to do SEO
as a living but merely for myself, is list main section headers that
then list or lead to sub-section "main" pages. Link from the site map
to the sub-section main/index/table of contents page that holds a list
of links for that section.

Thanks for your thoughts?

Hehehe, I like how you put that as a question. )

Don't know if my 2 cents will help you out on deciding which way to go
or not.

Carol

Thanks for your post. Here are some answer to your questions or
issues you raised.

On Mon, 03 May 2004 17:30:03 GMT, C.W. <from_you (AT) nomail (DOT) com> wrote:

....snip

Well, according to an old rule of thumb for web design/site layouts
thoughts - you want to have most of the content within two to three
clicks from the main page for the person coming to the information
being sought.

This may be true for a person, but my inquiry has to do with search
engines more than anything. As an aside, information or subjects can
be very complex, not allowing for a mere three clicks to get to it.
Thus I would need a search capability on my site to make it easy for
my human visitors..

....snip

- one suggestion from my side,
since you mentioned that some sections are just "deep" and may not be
able to made "less deep", is to try to cross-link to those deeper
pages from the "closer to the top" pages.

I will review my organization and try to make it as flat as possible,
but I am sure I will not be able to get all pages within 5 clicks of
the home page. I'll try cross lining to see if I can "move them up."

...snip

Is it scripted for this expanding thought - does it take the person to
a new page for the expansion of sections?
....snip

The site map is not scripted, just plain html code.

....snip

What I did on my sites is have a simple site map that is divided into
sections of thought; starting from the index.html and on through.
Below is a bad paraphrasing:

* Main Page
... Lists more recent content <-- anchor text back to main page
* Indexes
... Generic listing of all content on site <--- anchor text to
"articles listing" page
.... Yearly Collections:
....snip

I like this approach, kind of like a table of contents and summary
combined..

....snip

div style="display : none"><a href="page listing urls">Overview for
our SearchEngines</a> </div

Display none means the text is hidden so that would be viewed as a
hidden link which some view negatively as a practice.

There are so many simpler ways to share links to inner pages, in my
opinion, that hidden links is not really necessary method to employ.

If nothing else, share a link to the "organized site map" and let the
spider follow that. Then you have the link visible for users of your
site to be able to use also.

I do not know if search engines penalize you for this type of code. I

know they say they are against "tricks." Is this a trick by their
definition? I don't know. It looks innocent enough, but I guess
unless anyone knows it is safe, I ought to be safe and not include it.
I'll follow your advice and use my human map as a visible link at the
bottom of every page.

A thousand URLs on one page may risk having the lower shared URLs not
followed; Google advises to keep listing of links under 100 per page.

....snip

How do all of these directory sites or anyone get away with large
nubmer of links to a page? Google must have some way to handle this.
That having been said, it is probably still good advice not to fill a
page with tousands of links.

How does a big electronics firm or parts web site get all of their
parts into search engines? They can not have a site map that points
to every part and is only three clicks from the home page and is 100
or less links per page. If they had a 100 links per page, and 100
page linked on a site map, that is only 10, 000 links that are within
2 clicks of a home page. Another level of 100 gives you 1,000,000
links. so maybe that is how.

Or perhaps we are missing the point on seo. Perhaps, and please tell
me what you thing of this, it is necessary to have pages within a few
clicks of a home page to reinforce the strength of the subject matter
of the home page or of lower pages. In other words a web site's home
page that claims to be about cars should have car pages within 3
clicks in order to rank higher. But as you descend "lower into a
site, the search engines keep track of subject matter, so that a car
parts site could get away with having car toys lower in the site and
not be penalized, and the search engines have some way to rank the car
toys different than the home page.

That is not a very coherent explanation, but I have to wonder if my
whole premiss of a site map that points to every single page is not
correct. Perhaps I need a site map, much like your "table of
contents", as I call it, to point out important pages that serve as
mini home pages within a site, kind of like pages that change the
emphasis of the site at that point.

Anyway thanks for your post. It has been thought provoking.

I hope to hear from a few more people on their ideas.

(This is more of a general philosophy post triggered by the post
I am replying to rather than an attempt to addressing the specifics
of that one post...)

Al F <bscinc3000 (AT) Yahoo_NoSpam (DOT) com> says...

This may be true for a person, but my inquiry has to do with search
engines more than anything.

Google and yahoo would love to be able to make it so that the two
are indistinguishable. Anytime we design for the search engine
and not the human we risk future advances in search engine technology
making our work invalid. If, on the other hand, we design in such a
way that both humans and search engines benefit, our optimizations are
more likely to be lasting.

That being said, a search engine is blind, deaf, stupid and fast
compared to a human and that's not likely to change soon.

--
Guy Macon, Electronics Engineer & Project Manager for hire.
Remember Doc Brown from the _Back to the Future_ movies? Do you
have an "impossible" engineering project that only someone like
Doc Brown can solve? My resume is at http://www.guymacon.com/

On Mon, 03 May 2004 21:13:30 GMT, Al F <bscinc3000 (AT) Yahoo_NoSpam (DOT) com>
wrote:

Thanks for your post. Here are some answer to your questions or
issues you raised.

On Mon, 03 May 2004 17:30:03 GMT, C.W. <from_you (AT) nomail (DOT) com> wrote:

...snip

Well, according to an old rule of thumb for web design/site layouts
thoughts - you want to have most of the content within two to three
clicks from the main page for the person coming to the information
being sought.

This may be true for a person, but my inquiry has to do with search
engines more than anything. As an aside, information or subjects can
be very complex, not allowing for a mere three clicks to get to it.
Thus I would need a search capability on my site to make it easy for
my human visitors..

www.atomz.com but it'll cost you.

BB

...snip
- one suggestion from my side,
since you mentioned that some sections are just "deep" and may not be
able to made "less deep", is to try to cross-link to those deeper
pages from the "closer to the top" pages.

I will review my organization and try to make it as flat as possible,
but I am sure I will not be able to get all pages within 5 clicks of
the home page. I'll try cross lining to see if I can "move them up."

...snip

Is it scripted for this expanding thought - does it take the person to
a new page for the expansion of sections?
...snip

The site map is not scripted, just plain html code.

...snip

What I did on my sites is have a simple site map that is divided into
sections of thought; starting from the index.html and on through.
Below is a bad paraphrasing:

* Main Page
... Lists more recent content <-- anchor text back to main page
* Indexes
... Generic listing of all content on site <--- anchor text to
"articles listing" page
.... Yearly Collections:
...snip

I like this approach, kind of like a table of contents and summary
combined..

...snip
div style="display : none"><a href="page listing urls">Overview for
our SearchEngines</a> </div

Display none means the text is hidden so that would be viewed as a
hidden link which some view negatively as a practice.

There are so many simpler ways to share links to inner pages, in my
opinion, that hidden links is not really necessary method to employ.

If nothing else, share a link to the "organized site map" and let the
spider follow that. Then you have the link visible for users of your
site to be able to use also.

I do not know if search engines penalize you for this type of code. I
know they say they are against "tricks." Is this a trick by their
definition? I don't know. It looks innocent enough, but I guess
unless anyone knows it is safe, I ought to be safe and not include it.
I'll follow your advice and use my human map as a visible link at the
bottom of every page.

A thousand URLs on one page may risk having the lower shared URLs not
followed; Google advises to keep listing of links under 100 per page.

...snip

How do all of these directory sites or anyone get away with large
nubmer of links to a page? Google must have some way to handle this.
That having been said, it is probably still good advice not to fill a
page with tousands of links.

How does a big electronics firm or parts web site get all of their
parts into search engines? They can not have a site map that points
to every part and is only three clicks from the home page and is 100
or less links per page. If they had a 100 links per page, and 100
page linked on a site map, that is only 10, 000 links that are within
2 clicks of a home page. Another level of 100 gives you 1,000,000
links. so maybe that is how.

Or perhaps we are missing the point on seo. Perhaps, and please tell
me what you thing of this, it is necessary to have pages within a few
clicks of a home page to reinforce the strength of the subject matter
of the home page or of lower pages. In other words a web site's home
page that claims to be about cars should have car pages within 3
clicks in order to rank higher. But as you descend "lower into a
site, the search engines keep track of subject matter, so that a car
parts site could get away with having car toys lower in the site and
not be penalized, and the search engines have some way to rank the car
toys different than the home page.

That is not a very coherent explanation, but I have to wonder if my
whole premiss of a site map that points to every single page is not
correct. Perhaps I need a site map, much like your "table of
contents", as I call it, to point out important pages that serve as
mini home pages within a site, kind of like pages that change the
emphasis of the site at that point.

Anyway thanks for your post. It has been thought provoking.

I hope to hear from a few more people on their ideas.

Big Bill wrote in
<29je90d68sd64ckl7mjt1cqd108rcqm7m5 (AT) 4ax (DOT) com>

On Mon, 03 May 2004 21:13:30 GMT, Al F <bscinc3000 (AT) Yahoo_NoSpam (DOT) com
wrote:

Thanks for your post. Here are some answer to your questions or
issues you raised.

On Mon, 03 May 2004 17:30:03 GMT, C.W. <from_you (AT) nomail (DOT) com> wrote:

...snip

Well, according to an old rule of thumb for web design/site layouts
thoughts - you want to have most of the content within two to three
clicks from the main page for the person coming to the information
being sought.

This may be true for a person, but my inquiry has to do with search
engines more than anything. As an aside, information or subjects can
be very complex, not allowing for a mere three clicks to get to it.
Thus I would need a search capability on my site to make it easy for
my human visitors..

www.atomz.com but it'll cost you.

You're probably right in this case, Bill (as ever) since the site sounds to
be large but, for others who are taking note of the Atomz suggestion, it's
free for sites of fewer than 500 pages and it is very good, IMHO.

--
PeterMcC
If you feel that any of the above is incorrect,
inappropriate or offensive in any way,
please ignore it and accept my apologies.

Guy Macon <http://www.guymacon.com> Guy Macon <http://www.guymacon.com>
wrote:

That being said, a search engine is blind, deaf, stupid and fast
compared to a human and that's not likely to change soon.

I dunno...human beings seem to be making huge progress in harmonsiation as
regards the third...I'm sure the others will follow

--
eric
www.ericjarvis.co.uk
"live fast, die only if strictly necessary"

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> wrote

On Mon, 03 May 2004 21:13:30 GMT, Al F <bscinc3000 (AT) Yahoo_NoSpam (DOT) com
wrote:

Thanks for your post. Here are some answer to your questions or
issues you raised.

On Mon, 03 May 2004 17:30:03 GMT, C.W. <from_you (AT) nomail (DOT) com> wrote:

...snip

Well, according to an old rule of thumb for web design/site layouts
thoughts - you want to have most of the content within two to three
clicks from the main page for the person coming to the information
being sought.

This may be true for a person, but my inquiry has to do with search
engines more than anything. As an aside, information or subjects can
be very complex, not allowing for a mere three clicks to get to it.
Thus I would need a search capability on my site to make it easy for
my human visitors..

www.atomz.com but it'll cost you.

BB

snip

I have FusionBot, it is free. And works good. You can get the paid one, but
I use the free one and haven't had any problems with it.

Stacey

On Tue, 4 May 2004 11:51:30 -0400, "Stacey"
<stacey (AT) staceyssimplestuff (DOT) com> wrote:

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> wrote in message
news:29je90d68sd64ckl7mjt1cqd108rcqm7m5 (AT) 4ax (DOT) com...
On Mon, 03 May 2004 21:13:30 GMT, Al F <bscinc3000 (AT) Yahoo_NoSpam (DOT) com
wrote:

Thanks for your post. Here are some answer to your questions or
issues you raised.

On Mon, 03 May 2004 17:30:03 GMT, C.W. <from_you (AT) nomail (DOT) com> wrote:

...snip

Well, according to an old rule of thumb for web design/site layouts
thoughts - you want to have most of the content within two to three
clicks from the main page for the person coming to the information
being sought.

This may be true for a person, but my inquiry has to do with search
engines more than anything. As an aside, information or subjects can
be very complex, not allowing for a mere three clicks to get to it.
Thus I would need a search capability on my site to make it easy for
my human visitors..

www.atomz.com but it'll cost you.

BB

snip

I have FusionBot, it is free. And works good. You can get the paid one, but
I use the free one and haven't had any problems with it.

Stacey

Yeah, but for how many pages?

BB