Does Search engine crowl without submission ?

Hello,

My name is Amardeep Ashiyana.
I want to know does search engines crowl websites without submission.
Should website important to submit to Search engines..

I have some website for promotion so I want to know that Search engine
will not crowl without submission..

my websites details are ....

www.emavens.com

www.kashmirmart.com

www.kashmirhotels.net

www.indospan.net

www.eastendapparels.com

www.sanchitech.net

Thanks,
Amardeep
Ashiyana
9811550525

On 13-Jul-2006, "Amardeep Ashiyana" <webdesigningdelhi (AT) gmail (DOT) com> wrote:

My name is Amardeep Ashiyana.
I want to know does search engines crowl websites without submission.
Should website important to submit to Search engines..

If a site known to Google (ranked and being crawled) links
to one of your sites then that site will be crawled. You can use robots.txt
to stop crawling, but Google and other search engines may/often
ignore it.

<ato_zee (AT) hotmail (DOT) com> wrote

On 13-Jul-2006, "Amardeep Ashiyana" <webdesigningdelhi (AT) gmail (DOT) com> wrote:

My name is Amardeep Ashiyana.
I want to know does search engines crowl websites without submission.
Should website important to submit to Search engines..

If a site known to Google (ranked and being crawled) links
to one of your sites then that site will be crawled. You can use
robots.txt
to stop crawling, but Google and other search engines may/often
ignore it.

Google does honor the robots.txt file, as do most other search engines (the
reputable ones).

(If your robots.txt file is being ignored, perhaps it has an error - you can
validate it at http://tool.motoricerca.info/robots-checker.phtml )

Amardeep, ato_zee is right when he says you don't have to submit to search
engines. The search engines will find your site if there are links to it
from other sites. It doesn't hurt to submit to the search engines, but your
time may be better spent on other things.

indie.

Amardeep Ashiyana wrote:

<snip>

This India outsourcing agenda is starting to get a little stale. How
about coming up with a way to stick your schemes in our faces a little
more creatively?

--
Fred
http://www.rezultz-web-site-promotion.com/
search engine marketing ethics and basics

On 13-Jul-2006, "indiecoffeehouse" <nospam (AT) hotmail (DOT) com> wrote:

Google does honor the robots.txt file, as do most other search engines (the
reputable ones).

But the robots.txt file has to be there before or at the time the site is
first created. Once Google and the other SE's have visited and
indexed a site they tend to keep visiting it no matter what you do.
But an ancient site that is never updated can have a good PR
(unusual) and count as another inbound link.

<ato_zee (AT) hotmail (DOT) com> wrote

On 13-Jul-2006, "indiecoffeehouse" <nospam (AT) hotmail (DOT) com> wrote:

Google does honor the robots.txt file, as do most other search engines
(the
reputable ones).

But the robots.txt file has to be there before or at the time the site is
first created. Once Google and the other SE's have visited and
indexed a site they tend to keep visiting it no matter what you do.

That's not true. Every time the crawler comes back to the site it checks the
robots.txt file for changes. And it honors them.
You're not stuck forever with the robots.txt instructions that was there
when the SE first discovered your site.

If you don't have a robots.txt file when the site is first created, but you
add one sometime later, that's OK. When the crawlers come back again they'll
find the new file and and honor it.

But an ancient site that is never updated can have a good PR
(unusual) and count as another inbound link.

What does this have to do with robots.txt?

On 13-Jul-2006, "indiecoffeehouse" <nospam (AT) hotmail (DOT) com> wrote:

But an ancient site that is never updated can have a good PR
(unusual) and count as another inbound link.

What does this have to do with robots.txt?

It has a robots.txt
User-agent: *
Disallow: /

But it's there in search results 6 years later.
It's remains as a text pointer to a new URL in case
anyone comes back on an old bookmark.
The robots.txt was put up to prevent any possibility of
penalization for duplicate site or content.
Now accept it's another inbound link, and recon that
the robots.txt is being ignored.
Still new site is PR5 not that I trust Googles PR tool,
and no 1 on page 1 across all the main relevant
search terms and keywords, so I'm not
worried, if it ain't broke don't fix it.

This India outsourcing agenda is starting to get a little stale. How
about coming up with a way to stick your schemes in our faces a little
more creatively?

Posting URIs in a newsgroup is a funny way to stop them being crawled.

And robots.txt is NOT fetched for every access, but around once per day.