robots.txt validator

Does anyone know of a robots.txt checker/validator
that will show exactly what is actually being blocked ?
I know about and have run a couple of the many checkers
but those are really just for syntax and they don't
really say.... hey look dummy... you're blocking your
main index.html. for example

Jon

"Jon" <fornews (AT) hotmail (DOT) com> wrote

Does anyone know of a robots.txt checker/validator
that will show exactly what is actually being blocked ?
I know about and have run a couple of the many checkers
but those are really just for syntax and they don't
really say.... hey look dummy... you're blocking your
main index.html. for example

Jon
No idea about checker/validators,

but do check that you have a carriage return / line feed at the end of the
last line.
Eric.

"Eric Johnston" <eric.johnston (AT) blueyonder (DOT) co.uk> wrote

"Jon" <fornews (AT) hotmail (DOT) com> wrote in message
news:1f79ccf3.0401090824.3620e3ad (AT) posting (DOT) google.com...
Does anyone know of a robots.txt checker/validator
that will show exactly what is actually being blocked ?
I know about and have run a couple of the many checkers
but those are really just for syntax and they don't
really say.... hey look dummy... you're blocking your
main index.html. for example

Jon
No idea about checker/validators,
but do check that you have a carriage return / line feed at the end of the
last line.
Eric.

I use vi to edit all my files on *NIX servers.
Are you saying make sure there is a carriage return and a line feed
on the last line or literally a carriage return / line feed on
the last line. I've read some robots tutorials and I've never
seen a reference to this.

I think this would be a useful tool as part of a spider checker
suite. The results would be.. These files and/or directories
have been blocked by your robots.txt file... is this what
you intended ???

Jon

"Jon" <fornews (AT) hotmail (DOT) com> wrote

"Eric Johnston" <eric.johnston (AT) blueyonder (DOT) co.uk> wrote

"Jon" <fornews (AT) hotmail (DOT) com> wrote in message
news:1f79ccf3.0401090824.3620e3ad (AT) posting (DOT) google.com...
Does anyone know of a robots.txt checker/validator
that will show exactly what is actually being blocked ?
I know about and have run a couple of the many checkers
but those are really just for syntax and they don't
really say.... hey look dummy... you're blocking your
main index.html. for example

Jon
No idea about checker/validators,
but do check that you have a carriage return / line feed at the end of
the
last line.
Eric.

I use vi to edit all my files on *NIX servers.
Are you saying make sure there is a carriage return and a line feed
on the last line or literally a carriage return / line feed on
the last line. I've read some robots tutorials and I've never
seen a reference to this.

I think this would be a useful tool as part of a spider checker
suite. The results would be.. These files and/or directories
have been blocked by your robots.txt file... is this what
you intended ???

Jon

By "last line" I meant the last line with visible text, i.e. the final
instruction in the list..
When editing the robots.txt file just make sure to terminate the end of the
last line containing visible text by pressing the return key so that the
cursor ends up in the left position of the empty line below. When
something interprets the robots.txt file there is a possibility that a line
of text which is not terminated may not the interpreted.

Best regards, Eric.