blocking robots.txt from non-robots

Perhaps I didn't say it right. *I'm wanting to block the robots.txt
that
I'm feeding search engines from being given to anybody else.

If Google catch you they will exclude you from the index.

'Don't deceive your users or present different content to search
engines than you display to users, which is commonly referred to as
"cloaking." '

I can't believe this.

Then you need to make a greater effort.

"Joe Fox" <ny152 (AT) none (DOT) invalid> wrote:

I don't want certain humans (only a few hundred in number but all on
dynamic IPs in several countries) to be able to read the robots.txt that
I'm giving search engines because I don't want them to know what pages I
am telling SE's "disallow"

What's so wrong with this?

You don't own Google, and they choose not to be used in this way.
It's their index, and they make the rules.
If you choose to ignore them ... they may choose to ignore your site.

Nothing underhand there, either.
--

Andrew
http://www.seo2seo.com/
http://www.sick-site-syndrome.com/
First things first - but not necessarily in that order.

http://www.whalesrevenge.com/

Joe Fox <ny152 (AT) none (DOT) invalid> wrote:

Not really, or is it possible that they could also get my .htaccess?
I didn't think that was possible. If they ask for a robots.txt and
get one that's got nothing more than a pointer to a sitemap that will
satisfy 'em.

Let's assume for arguments sake that those people *want* to see your
robots.txt. If you feed Google something different than them, they will
notice as soon as they check Google, because if you disallow Google some
directories, while your robots.txt says allow, they will wonder why all
pages in some directory don't show up in Google, but are available on your
site.

I really don't get why you want to hide your robots.txt.

--
John Bokma http://johnbokma.com/

Joe Fox <ny152 (AT) none (DOT) invalid> wrote:

I don't want certain humans (only a few hundred in number but all on
dynamic IPs in several countries) to be able to read the robots.txt that
I'm giving search engines because I don't want them to know what pages I
am telling SE's "disallow"

Maybe it helps if you explain the why. Which is not: I don't want them to
read. What do you want to achieve? Why should those people not be able to
read your robots.txt

What's so wrong with this?

To me it sounds pointless. I see no gain in it, but maybe you can explain
better the *why*?

BTW: I disagree with others that Google et al should have a problem with
this. Although it's cloaking, it's not something (in this case) Google
should care about. It's like getting upset about a site that shows a flag
on its page based on the country you connect from, and feeds Google a flag
of the USA :-)

--
John Bokma http://johnbokma.com/

John Bokma schreef:

Joe Fox <ny152 (AT) none (DOT) invalid> wrote:

I don't want certain humans (only a few hundred in number but all on
dynamic IPs in several countries) to be able to read the robots.txt that
I'm giving search engines because I don't want them to know what pages I
am telling SE's "disallow"

Maybe it helps if you explain the why. Which is not: I don't want them to
read. What do you want to achieve? Why should those people not be able to
read your robots.txt

What's so wrong with this?

To me it sounds pointless. I see no gain in it, but maybe you can explain
better the *why*?

He doesn't want real people to know which files he is hiding and were to
find them.

I agree John, it is pointless.

BTW: I disagree with others that Google et al should have a problem with
this. Although it's cloaking, it's not something (in this case) Google
should care about. It's like getting upset about a site that shows a flag
on its page based on the country you connect from, and feeds Google a flag
of the USA :-)

If i were a bot, i would get very upset. Eating flags all day long seems
to me not the best meal one can get.



--
Webdesign: http://vision2form.nl/webontwerp/
Korte handleiding zoekmachine optimalisatie / gevonden worden:
http://vision2form.nl/webontwerp/gevonden-worden.html
Lifestyle - wonen reizen en genieten : http://vision4living.com

John Bokma <john (AT) castleamber (DOT) com> wrote in
news:Xns9A4B64E22EB17castleamber (AT) 130 (DOT) 133.1.4:

Joe Fox <ny152 (AT) none (DOT) invalid> wrote:

Not really, or is it possible that they could also get my .htaccess?
I didn't think that was possible. If they ask for a robots.txt and
get one that's got nothing more than a pointer to a sitemap that will
satisfy 'em.

Let's assume for arguments sake that those people *want* to see your
robots.txt. If you feed Google something different than them, they
will notice as soon as they check Google, because if you disallow
Google some directories, while your robots.txt says allow, they will
wonder why all pages in some directory don't show up in Google, but
are available on your site.

They may wonder but that's it.

I really don't get why you want to hide your robots.txt.

How about this idea?

Given: I want to hide robots.txt from the public in general and a certain
group in particular.

Given: You and others here don't see the need or point in my doing this.

Given: Ultimately, if there's a potential penalty for doing this then
that's mine to risk.

Can we simply agree to disagree and save discussion of *why* for another
time and go into some details about *how*?

Thanks

Big Bill <bill (AT) kruse (DOT) co.uk> wrote in
news:njpqr39aqlb7g6p9efcfqk67ash18hbh0o (AT) 4ax (DOT) com:

On Thu, 21 Feb 2008 00:10:25 -0600, Joe Fox <ny152 (AT) none (DOT) invalid
wrote:

Phil Payne <phil (AT) isham-research (DOT) co.uk> wrote in
news:adf1b99f-5c4d-48a8-b348-49a9cacc2453 (AT) n77g2000hse (DOT) googlegroups.com:

Perhaps I didn't say it right. *I'm wanting to block the robots.txt
that

I'm feeding search engines from being given to anybody else.

If Google catch you they will exclude you from the index.

'Don't deceive your users or present different content to search
engines than you display to users, which is commonly referred to as
"cloaking." '



I can't believe this.

I'm not trying to cloak my content or pull anything underhanded.

You are, though, you're trying to cloak your robots.txt.

Which I cannot for the life of me see what is wrong with that.

I consider it to be a proprietary document that gets fed to search
engines as needed. Humans have zero business even being able to read it.

John Bokma <john (AT) castleamber (DOT) com> wrote in
news:Xns9A4B65A59A1Fcastleamber (AT) 130 (DOT) 133.1.4:

Joe Fox <ny152 (AT) none (DOT) invalid> wrote:

I don't want certain humans (only a few hundred in number but all on
dynamic IPs in several countries) to be able to read the robots.txt
that I'm giving search engines because I don't want them to know what
pages I am telling SE's "disallow"

Maybe it helps if you explain the why. Which is not: I don't want them
to read. What do you want to achieve? Why should those people not be
able to read your robots.txt

Because I consider it to be a proprietary document and I do not want
these people to be able to determine if I am allowing or disallowing
certain pages to be indexed.

What's so wrong with this?

To me it sounds pointless. I see no gain in it, but maybe you can
explain better the *why*?

As I said in another post, Can we save discussion of *why* for another
time and talk about *how*?

Thanks

Tonnie Lubbers <t.prasing (AT) chello (DOT) nl> wrote in
news:decfe$47bdba7a$d52e9221$16202 (AT) news (DOT) chello.nl:

John Bokma schreef:
Joe Fox <ny152 (AT) none (DOT) invalid> wrote:

I don't want certain humans (only a few hundred in number but all on
dynamic IPs in several countries) to be able to read the robots.txt
that I'm giving search engines because I don't want them to know
what pages I am telling SE's "disallow"

Maybe it helps if you explain the why. Which is not: I don't want
them to read. What do you want to achieve? Why should those people
not be able to read your robots.txt

What's so wrong with this?

To me it sounds pointless. I see no gain in it, but maybe you can
explain better the *why*?

He doesn't want real people to know which files he is hiding and were
to find them.

Exactly.

I agree John, it is pointless.

Perhaps, but if pointless it is also harmless. I'm sure that
google & others don't give a fork whether human users are able to
determine what I allow or disallow to be indexed. I'm not hiding
content, I'm wanting to keep to myself what is or is not indexed.

On 21 Feb 2008 15:59:34 GMT, John Bokma <john (AT) castleamber (DOT) com> wrote:

Joe Fox <ny152 (AT) none (DOT) invalid> wrote:

I don't want certain humans (only a few hundred in number but all on
dynamic IPs in several countries) to be able to read the robots.txt that
I'm giving search engines because I don't want them to know what pages I
am telling SE's "disallow"

Maybe it helps if you explain the why. Which is not: I don't want them to
read. What do you want to achieve? Why should those people not be able to
read your robots.txt

What's so wrong with this?

To me it sounds pointless. I see no gain in it, but maybe you can explain
better the *why*?

BTW: I disagree with others that Google et al should have a problem with
this. Although it's cloaking, it's not something (in this case) Google
should care about. It's like getting upset about a site that shows a flag
on its page based on the country you connect from, and feeds Google a flag
of the USA :-)

I never said they'd care about it, just that it was cloaking.

BB
--

http://www.kruse.co.uk/
http://www.fat-odin.com/
http://www.here-be-posters.co.uk/