Do I need robot file?

Borek wrote:

John Bokma <john (AT) castleamber (DOT) com> wrote:

Would an empty robots.txt file do the same thing? I think MSN
suggests it and that's what I use but is it ok with other SE's?

The one suggested above means: everything is allowed, so is
identical to none. No idea if an empty file works similar.

Even if it works, it seems it is not correct:

http://www.robotstxt.org/wc/norobots.html

states that

"The file consists of one or more records separated by one or more blank
lines (terminated by CR,CR/NL, or NL). Each record contains lines of the
form "<field>:<optionalspace><value><optionalspace> ". The field name is
case insensitive."

So empty file is impossible, as the file MUST contain at least one record.

Alas, the specification contains a contradiction.

http://www.robotstxt.org/wc/norobots.html says "The file consists
of one or more records", which would exclude a file with zero
records - an empty file, but the same specification says "The
presence of an empty "/robots.txt" file ... will be treated as if
it was not present", which implies that an empty file is OK.
Because of this contradiction, I advise against it. We want our
sites to work even if a future web spider decides to follow one
part of the spec strictly and ignore the other part.

BTW, http://www.searchengineworld.com/mis..._txt_crawl.htm
appears to have an error. Compare the section on "DOS Line Enders"
with the spec at http://www.robotstxt.org/wc/norobots.html , which
says that CR,CR/NL, or NL are all allowed.





--
Guy Macon <http://www.guymacon.com/> Guy Macon <http://www.guymacon.com/>
Guy Macon <http://www.guymacon.com/> Guy Macon <http://www.guymacon.com/>
Guy Macon <http://www.guymacon.com/> Guy Macon <http://www.guymacon.com/>
Guy Macon <http://www.guymacon.com/> Guy Macon <http://www.guymacon.com/>

In message <ea0if.39408$5o6.712665 (AT) wagner (DOT) videotron.net>, Clint
<pepmax (AT) videotron (DOT) ca> writes

After a year, my website www.FreeSpiritGallery.ca is finally tracking decent
on the search engines. I have never used a robot file though since I don't
mind the search engines spidering my entire site. My question - is there
any value for me to add in a robot file even if I don't mind my entire site
being indexed?

Clint

The main use is to stop SE for indexing pages we use it to stop Google
and the like from indexing parts of the forum as many just use it as a
link farm this is what we use


User-agent: *
Disallow: /piano-forums/memberlist.php
Disallow: /piano-forums/privmsg.php
Disallow: /piano-forums/profile.php
Disallow: /piano-forums/posting.php
--
BH

Duende <myusenet (AT) sify (DOT) com> wrote:

User-agent: *
Disallow:/

Saves more trafic. :

Not having a site at all seems to be the biggest saver though :-D.


--
John Perl SEO tools: http://johnbokma.com/perl/
or have them custom made
Experienced (web) developer: http://castleamber.com/

Borek <m.borkowski (AT) delete (DOT) chembuddy.these.com.parts> wrote:

On Sun, 27 Nov 2005 02:27:16 +0100, John Bokma <john (AT) castleamber (DOT) com
wrote:

Would an empty robots.txt file do the same thing? I think MSN
suggests it and that's what I use but is it ok with other SE's?

The one suggested above means: everything is allowed, so is identical
to none. No idea if an empty file works similar.

Even if it works, it seems it is not correct:

http://www.robotstxt.org/wc/norobots.html

states that

"The file consists of one or more records separated by one or more
blank lines (terminated by CR,CR/NL, or NL). Each record contains
lines of the form "<field>:<optionalspace><value><optionalspace> ".
The field name is case insensitive."

So empty file is impossible, as the file MUST contain at least one
record.

Same site states that empty is ok (see my previous post). So hmmmm...

--
John Perl SEO tools: http://johnbokma.com/perl/
or have them custom made
Experienced (web) developer: http://castleamber.com/

"John" <nospam (AT) yahoo (DOT) com> wrote

"Clint" <pepmax (AT) videotron (DOT) ca> wrote in message
news:ea0if.39408$5o6.712665 (AT) wagner (DOT) videotron.net...
After a year, my website www.FreeSpiritGallery.ca is finally tracking
decent on the search engines. I have never used a robot file though
since I don't mind the search engines spidering my entire site. My
question - is there any value for me to add in a robot file even if I
don't mind my entire site being indexed?

Clint
This might help some
http://tool.motoricerca.info/robots-checker.phtml
It's a robots.txt validator.

Yup, I put my new robot file through the above validator and it passed.
Thanks

Clint

http://www.FreeSpiritGallery.ca