Do I need robot file?

#1

After a year, my website www.FreeSpiritGallery.ca is finally tracking decent
on the search engines. I have never used a robot file though since I don't
mind the search engines spidering my entire site. My question - is there
any value for me to add in a robot file even if I don't mind my entire site
being indexed?

Clint

#2

"Clint" <pepmax (AT) videotron (DOT) ca> wrote:

Quote:

After a year, my website www.FreeSpiritGallery.ca is finally tracking
decent on the search engines. I have never used a robot file though
since I don't mind the search engines spidering my entire site. My
question - is there any value for me to add in a robot file even if I
don't mind my entire site being indexed?

Yes, it keeps the error_log clean :-D

For a nice default version, see

http://johnbokma.com/robots.txt

Just copy it to your site (change / remove the first line :-) )

--
John Perl SEO tools: http://johnbokma.com/perl/
or have them custom made
Experienced (web) developer: http://castleamber.com/

#3

On Sat, 26 Nov 2005 11:26:30 -0500, "Clint" <pepmax (AT) videotron (DOT) ca>
wrote:

Quote:

After a year, my website www.FreeSpiritGallery.ca is finally tracking decent
on the search engines. I have never used a robot file though since I don't
mind the search engines spidering my entire site. My question - is there
any value for me to add in a robot file even if I don't mind my entire site
being indexed?

Clint

It'll cut down the number of error messages you get in your logs but I
think in practical terms that's about it.
I'd be inclined myself to suggest if it ain't broke don't fix it but
one of the propellerheads we have on here may well override me on
this.
I thought of you the other day, I was watching that movie called um,
Milla's Feeling for Snow, or similar. Glad to hear you're beginning to
get on ok.

BB

--
www.kruse.co.uk/ seo (AT) kruse (DOT) demon.co.uk
The buffalo have gone

#4

"John Bokma" <john (AT) castleamber (DOT) com> wrote

Quote:

"Clint" <pepmax (AT) videotron (DOT) ca> wrote:

After a year, my website www.FreeSpiritGallery.ca is finally tracking
decent on the search engines. I have never used a robot file though
since I don't mind the search engines spidering my entire site. My
question - is there any value for me to add in a robot file even if I
don't mind my entire site being indexed?

Yes, it keeps the error_log clean :-D

For a nice default version, see

http://johnbokma.com/robots.txt

Would an empty robots.txt file do the same thing? I think MSN suggests it
and that's what I use but is it ok with other SE's?

Quote:

Just copy it to your site (change / remove the first line :-) )

--
John Perl SEO tools: http://johnbokma.com/perl/
or have them custom made
Experienced (web) developer: http://castleamber.com/

#5

"John" <nospam (AT) yahoo (DOT) com> wrote:

Quote:

"John Bokma" <john (AT) castleamber (DOT) com> wrote in message
news:Xns971A6E1C2536Dcastleamber (AT) 130 (DOT) 133.1.4...
"Clint" <pepmax (AT) videotron (DOT) ca> wrote:

After a year, my website www.FreeSpiritGallery.ca is finally
tracking decent on the search engines. I have never used a robot
file though since I don't mind the search engines spidering my
entire site. My question - is there any value for me to add in a
robot file even if I don't mind my entire site being indexed?

Yes, it keeps the error_log clean :-D

For a nice default version, see

http://johnbokma.com/robots.txt

Would an empty robots.txt file do the same thing? I think MSN
suggests it and that's what I use but is it ok with other SE's?

The one suggested above means: everything is allowed, so is identical to
none. No idea if an empty file works similar.

--
John Perl SEO tools: http://johnbokma.com/perl/
or have them custom made
Experienced (web) developer: http://castleamber.com/

#6

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> wrote

Quote:

On Sat, 26 Nov 2005 11:26:30 -0500, "Clint" <pepmax (AT) videotron (DOT) ca
wrote:
My question - is there
any value for me to add in a robot file even if I don't mind my entire
site
being indexed?

Clint

It'll cut down the number of error messages you get in your logs but I
think in practical terms that's about it.

There are messages in my logs for sure. I think I will try John's robot
file suggestion - thanks everyone for your comments.

Clint

http://www.FreeSpiritGallery.ca

#7

"Clint" <pepmax (AT) videotron (DOT) ca> wrote:

Quote:

There are messages in my logs for sure. I think I will try John's robot
file suggestion - thanks everyone for your comments.

See also the robots.txt site, especially
<http://www.robotstxt.org/wc/exclusion-admin.html>

To allow all robots complete access

User-agent: *
Disallow:

Or create an empty "/robots.txt" file.

So your idea of creating an empty file is right as well, and saves some
traffic :-)

--
John Perl SEO tools: http://johnbokma.com/perl/
or have them custom made
Experienced (web) developer: http://castleamber.com/

#8

__/ [John Bokma] on Sunday 27 November 2005 01:27 \__

Quote:

"John" <nospam (AT) yahoo (DOT) com> wrote:

"John Bokma" <john (AT) castleamber (DOT) com> wrote in message
news:Xns971A6E1C2536Dcastleamber (AT) 130 (DOT) 133.1.4...
"Clint" <pepmax (AT) videotron (DOT) ca> wrote:

After a year, my website www.FreeSpiritGallery.ca is finally
tracking decent on the search engines. I have never used a robot
file though since I don't mind the search engines spidering my
entire site. My question - is there any value for me to add in a
robot file even if I don't mind my entire site being indexed?

Yes, it keeps the error_log clean :-D

For a nice default version, see

http://johnbokma.com/robots.txt

Would an empty robots.txt file do the same thing? I think MSN
suggests it and that's what I use but is it ok with other SE's?

The one suggested above means: everything is allowed, so is identical to
none. No idea if an empty file works similar.

An empty file should be fine. Keeps the 404's away and implies no exclusions.
I must have read this somewhere because I had an empty file before I began
to list particular pages.

Hope it helps,

Roy

--
Roy S. Schestowitz | Data lacking semantics is currency in an island
http://Schestowitz.com | SuSE Linux | PGP-Key: 0x74572E8E
6:05am up 24 days 1:59, 7 users, load average: 2.23, 1.73, 1.79
http://iuron.com - next generation of search paradigms

#9

"Clint" <pepmax (AT) videotron (DOT) ca> wrote

Quote:

After a year, my website www.FreeSpiritGallery.ca is finally tracking
decent on the search engines. I have never used a robot file though since
I don't mind the search engines spidering my entire site. My question -
is there any value for me to add in a robot file even if I don't mind my
entire site being indexed?

Clint
This might help some

http://tool.motoricerca.info/robots-checker.phtml
It's a robots.txt validator.

#10

On Sun, 27 Nov 2005 02:27:16 +0100, John Bokma <john (AT) castleamber (DOT) com>
wrote:

Quote:

Would an empty robots.txt file do the same thing? I think MSN
suggests it and that's what I use but is it ok with other SE's?

The one suggested above means: everything is allowed, so is identical to
none. No idea if an empty file works similar.

Even if it works, it seems it is not correct:

http://www.robotstxt.org/wc/norobots.html

states that

"The file consists of one or more records separated by one or more blank
lines (terminated by CR,CR/NL, or NL). Each record contains lines of the
form "<field>:<optionalspace><value><optionalspace> ". The field name is
case insensitive."

So empty file is impossible, as the file MUST contain at least one record.

Best,
Borek
--
http://www.chembuddy.com
http://www.chembuddy.com/?left=EBAS&...-stoichiometry
http://www.chembuddy.com/?left=BATE&...ion_equilibria
http://www.chembuddy.com/?left=CASC&...n_calcul ator