strange situation

Hi,
something is wrong...
i published my website few weeks ago.. and.. i'm tracking some search engine
bots.
they come to my site, get robots.txt and they leave the website without
requesting internal pages or links.
anyone could please explain me why.

please take a look at the website

http://www.lemiecime.it

many thanks

--
Francesco

Togli l'antispam per rispondere in privato

On Thu, 03 Jun 2004 22:35:12 GMT, "Francesco"
<viali (AT) TOGLIQUESTO (DOT) tiscali.it> wrote:

Hi,
something is wrong...
i published my website few weeks ago.. and.. i'm tracking some search engine
bots.
they come to my site, get robots.txt and they leave the website without
requesting internal pages or links.
anyone could please explain me why.

please take a look at the website

http://www.lemiecime.it

many thanks

What's in the robots text?

BB

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> ha scritto nel messaggio
news:59nvb099d3gictaurf9tu8s116f50oj7a7 (AT) 4ax (DOT) com

On Thu, 03 Jun 2004 22:35:12 GMT, "Francesco"
viali (AT) TOGLIQUESTO (DOT) tiscali.it> wrote:
i published my website few weeks ago.. and.. i'm tracking some
search engine bots.
they come to my site, get robots.txt and they leave the website
without requesting internal pages or links.
anyone could please explain me why.

please take a look at the website

http://www.lemiecime.it

many thanks

What's in the robots text?

http://www.lemiecime.it/robots.txt:

User-Agent: *
Disallow: /stats
Disallow: /cgi-bin
Disallow: /guestbookthanks-- FrancescoTogli l'antispam per rispondere in
privato

"Francesco" <viali (AT) TOGLIQUESTO (DOT) tiscali.it> wrote

"Big Bill" <kruse (AT) cityscape (DOT) co.uk> ha scritto nel messaggio
news:59nvb099d3gictaurf9tu8s116f50oj7a7 (AT) 4ax (DOT) com
On Thu, 03 Jun 2004 22:35:12 GMT, "Francesco"
viali (AT) TOGLIQUESTO (DOT) tiscali.it> wrote:
i published my website few weeks ago.. and.. i'm tracking some
search engine bots.
they come to my site, get robots.txt and they leave the website
without requesting internal pages or links.
anyone could please explain me why.

please take a look at the website

http://www.lemiecime.it

many thanks

What's in the robots text?

http://www.lemiecime.it/robots.txt:

User-Agent: *
Disallow: /stats
Disallow: /cgi-bin
Disallow: /guestbookthanks-- FrancescoTogli l'antispam per rispondere in
privato

I now see this..
User-Agent: *
Disallow: /stats
Disallow: /cgi-bin
Disallow: /guestbookThere should be a carriage return after the k, otherwise
the last line might not work.
Best regards, Eric.

"Eric Johnston" <eric.johnston (AT) blueyonder (DOT) co.uk> ha scritto nel
messaggio news:UC5wc.139$Rj6.69 (AT) pathologist (DOT) blueyonder.net

"Francesco" <viali (AT) TOGLIQUESTO (DOT) tiscali.it> wrote in message
news:QHTvc.282075$hc5.12278479 (AT) news3 (DOT) tin.it...
"Big Bill" <kruse (AT) cityscape (DOT) co.uk> ha scritto nel messaggio
news:59nvb099d3gictaurf9tu8s116f50oj7a7 (AT) 4ax (DOT) com
On Thu, 03 Jun 2004 22:35:12 GMT, "Francesco"
viali (AT) TOGLIQUESTO (DOT) tiscali.it> wrote:
i published my website few weeks ago.. and.. i'm tracking some
search engine bots.
they come to my site, get robots.txt and they leave the website
without requesting internal pages or links.
anyone could please explain me why.
I now see this..
User-Agent: *
Disallow: /stats
Disallow: /cgi-bin
Disallow: /guestbookThere should be a carriage return after the k,
otherwise the last line might not work.
Best regards, Eric.

ok, thanks for your suggestion.
but I don't think this is why search engines are not spidering my
websites...
isn't it ?

--
Francesco
http://www.lemiecime.it
Togli l'antispam per rispondere in privato

"Francesco" <viali (AT) TOGLIQUESTO (DOT) tiscali.it> wrote

"Eric Johnston" <eric.johnston (AT) blueyonder (DOT) co.uk> ha scritto nel
messaggio news:UC5wc.139$Rj6.69 (AT) pathologist (DOT) blueyonder.net
"Francesco" <viali (AT) TOGLIQUESTO (DOT) tiscali.it> wrote in message
news:QHTvc.282075$hc5.12278479 (AT) news3 (DOT) tin.it...
"Big Bill" <kruse (AT) cityscape (DOT) co.uk> ha scritto nel messaggio
news:59nvb099d3gictaurf9tu8s116f50oj7a7 (AT) 4ax (DOT) com
On Thu, 03 Jun 2004 22:35:12 GMT, "Francesco"
viali (AT) TOGLIQUESTO (DOT) tiscali.it> wrote:
i published my website few weeks ago.. and.. i'm tracking some
search engine bots.
they come to my site, get robots.txt and they leave the website
without requesting internal pages or links.
anyone could please explain me why.
I now see this..
User-Agent: *
Disallow: /stats
Disallow: /cgi-bin
Disallow: /guestbookThere should be a carriage return after the k,
otherwise the last line might not work.
Best regards, Eric.

ok, thanks for your suggestion.
but I don't think this is why search engines are not spidering my
websites...
isn't it ?

--
Francesco
http://www.lemiecime.it
Togli l'antispam per rispondere in privato

1. It was an idea based on previous experience with unterminated lines.
Since there can be various parameters the software waits for and end of line
code before it can be sure it has all the parameters. If in doubt, type in
the whole robots.txt again, avoiding any mistakes in the typing and pressing
return after the last k You don't want hidden invalid non-displaying bytes
embedded in the file. If the software that reads the robots.txt thinks it
has a line like Disallow: / then all robots will be stopped.
Read this: http://www.searchengineworld.com/mis..._txt_crawl.htm

2. I have just tested your robots.txt with a validator at
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
It points out you have a capital letter A for agent. This is not a fatal
error for all search engines, but a "warning" and therefore may or may not
be acceptable to all search engine robots. Change this to lower case a to
be on the safe side.

Best regards, Eric.