Re: I am giving up wikipedia mirror

Ignoramus23035 <ignoramus23035 (AT) NOSPAM (DOT) 23035.invalid> wrote:

Last night, I made changes for robots.txt, so far no effect.

Takes at least a day. Did you check with Google Sitemaps if Google
understands your new version? It's easy to make a tiny mistake.


--
John Experienced (web) developer: http://castleamber.com/

Perl RSS Builder: http://johnbokma.com/perl/rss-web-feed-builder.html

Ignoramus23035 <ignoramus23035 (AT) NOSPAM (DOT) 23035.invalid> wrote:

On 6 Mar 2006 20:14:06 GMT, John Bokma <john (AT) castleamber (DOT) com> wrote:
Ignoramus23035 <ignoramus23035 (AT) NOSPAM (DOT) 23035.invalid> wrote:

Last night, I made changes for robots.txt, so far no effect.

Takes at least a day. Did you check with Google Sitemaps if Google
understands your new version? It's easy to make a tiny mistake.

Well, I think that robots.txt overrides sitemaps.

No, I mean, Site maps has an option to check robots.txt. Apologies for
being a bit vague.


http://www.google.com/webmasters/sit...stats?siteUrl=

and click on robots.txt tab.

Nifty eh?

--
John Freelance Perl programmer: http://castleamber.com/

Quick Bookmarks:http://johnbokma.com/firefox/quick-l...bookmarks.html

Fleeing from the madness of the Castle Amber - software development jungle
John Bokma <john (AT) castleamber (DOT) com> stumbled into
news:alt.internet.search-engines,alt.www.webmaster
and said:

...
and click on robots.txt tab.

talking of robots.txt - is it possible to stick wildcards in the exclusion
list?

--
William Tasso

whither a trophy?

"William Tasso" <SpamBlocked (AT) tbdata (DOT) com> wrote:

Fleeing from the madness of the Castle Amber - software development
jungle John Bokma <john (AT) castleamber (DOT) com> stumbled into
news:alt.internet.search-engines,alt.www.webmaster
and said:

...
and click on robots.txt tab.

talking of robots.txt - is it possible to stick wildcards in the
exclusion list?

<http://www.robotstxt.org/wc/norobots.html>

From what I read, no. Problem is, the robots.txt "standard" (ha ha) is
badly written. Maybe because back in those days you could count the bots
on one finger.

--
John
Sig-o-matic says:
OMG, lolzzz (c)... brb

On Mon, 06 Mar 2006 22:05:21 +0100, William Tasso <SpamBlocked (AT) tbdata (DOT) com>
wrote:

and click on robots.txt tab.

talking of robots.txt - is it possible to stick wildcards in the
exclusion list?

In general - no (check www.robotstxt.org for details). Some bots may know
better.

Best,
Borek
--
http://www.chembuddy.com
http://www.bpp.com.pl

On Mon, 06 Mar 2006 22:45:22 +0100, Borek
<m.borkowski (AT) delete (DOT) chembuddy.these.com.parts> wrote:

and click on robots.txt tab.

talking of robots.txt - is it possible to stick wildcards in the
exclusion list?

In general - no (check www.robotstxt.org for details). Some bots may
know better.

Sorry all - no idea why dupes are posted I am sending only once, looks
there is some bug either in news server or in Opera. It happens only when
the thread is shared between newsgroups (alt.internet.search-engines and
alt.www.webmaster in this case). In one group (aise - where I have
originally posted) there are single posts, in the second - aww - there are
dupes.

I am posting in aww this time, perhaps there will be dupes in aise?

Best,
Borek
--
http://www.chembuddy.com
http://www.ph-meter.info

Deciding to do something for the good of humanity, Borek
<m.borkowski (AT) delete (DOT) chembuddy.these.com.parts> declared in
alt.internet.search-engines,alt.www.webmaster:

In one group (aise - where I have
originally posted) there are single posts, in the second - aww - there are
dupes.

Not here. Only saw your post once.

--
Mark Parnell

Now implementing http://blinkynet.net/comp/uip5.html

On Tue, 07 Mar 2006 01:12:22 +0100, Mark Parnell
<webmaster (AT) clarkecomputers (DOT) com.au> wrote:

In one group (aise - where I have
originally posted) there are single posts, in the second - aww - there
are
dupes.

Not here. Only saw your post once.

Doesn't matter where I post - it lands twice in aww and once in aise. But
if it is only my problem I am not going to investigate

Best,
Borek
--
http://www.chembuddy.com
http://www.bpp.com.pl

talking of robots.txt - is it possible to stick wildcards in the
exclusion
list?

There's "User-Agent: *", but apart from that, not according to the
standard. (Though certain robots support certain extensions to the
standard.)

Google has some support for wildcards:
(http://www.google.com/webmasters/remove.html)
To remove all files of a specific file type (for example, .gif), you'd use
the following robots.txt entry:
User-agent: Googlebot
Disallow: /*.gif$

To remove dynamically generated pages, you'd use this robots.txt entry:
User-agent: Googlebot
Disallow: /*?