Originally posted by: Newsgroup User
.oO(Amit Chugh)
Quote:
You don't need the robots.txt file if you want everything to be indexed. |
But it makes sense if you want to prevent 404 errors in your logfile.
The same applies to favicons.
Micha
Michael Fesser,
Thanks for correcting me. Forgot about the 404 errors about it. They just fill
up the web server log.
But the file is not needed to improve your site getting searched because the
person has to submit the site and then the crawlers take over your job of
accessing every page (assuming every page is linked to someother page that
ultimately goes back to the page that was submitted to the search engine0
-Amit