Lots of noise about user agent strings

Seems developers of mobile applications are pretty much devoted to UA
sniffing:

<URL: http://wurfl.sourceforge.net/vodafonerant/index.htm >


--
Rob

RobG wrote:

Seems developers of mobile applications are pretty much devoted
to UA sniffing:

URL: http://wurfl.sourceforge.net/vodafonerant/index.htm

Yes, it appears to be someone who has made a serious mistake complaining
about the reality that made it a mistake. Quoting a bit of that page
shows that quite a few false beliefs were behind the mistake:-

All the way since the inception of the web, HTTP clients have
had unique User-Agent (UA) strings which let the server know
who they were. This mechanism was taken over as-is by mobile
browser manufacturers. While there have been a few exceptions
due to device manufacturer's sloppiness, it is accurate to say
that 99.99% of the devices out there have unique UA strings
which can be associated to brand, model and a bunch of other
info about the device properties.

HTTP clients have not had unique UA strings for well over a decade now,
and even UA strings that were distinct from others were explicitly
designed to prevent the server from knowing which client was being used
(hence the "Mozilla" bit at the front of IE's UA header, it was put
there only to mislead servers).

And it is absolutely not the case the 99.99% of devices have unique UA
strings. I may not get to look at mobile device browsers that often but
to date the vast majority of the scriptable browsers I have examined
have had UA strings that were more or less indistinguishable from those
of IE 6.

The bottom line remains that the HTTP specification defines the
User-Agent header as an arbitrary sequence of characters and does not
even require that the sequence of characters be the same for two
consecutive requests, let alone being in any sense unique. Any attempt
to treat something that is specified in that way as a source of
information is going to be a very obvious mistake.

Richard.

On May 27, 2:23 pm, Gregor Kofler <use... (AT) gregorkofler (DOT) at> wrote:

RobG meinte:

Seems developers of mobile applications are pretty much devoted to UA
sniffing:

URL:http://wurfl.sourceforge.net/vodafonerant/index.htm

A real expert:

"All the way since the inception of the web, HTTP clients have had
unique User-Agent (UA) strings which let the server know who they were."

What offense to you have to that?
User-Agent string is intended to let the server know who is requesting
the page, this is why they were created and how they are used since
Mosaic. Is your version being that it is a NCSA invented beatifying
ornament of HTTP request? Sorry to say then that it is utterly wrong
though semi-poetic version.

User-Agent are unique to each browser and each browser version unless
the browser code is manually reverse-engineered and altered by the end
user in violence of EULA. Such situation is definitely possible but
serious solutions do not normally account users surfing the Web with
hacked software: unless it is a special statistics collector.

The Vodafon problem is a problem causing money loss to the involved
MMS / media push providers. The problem is not deadly because the
agent info is not removed but only reformatted - yet such things are
not allowed anyway, the intermediary servers must not be refactoring
HTTP requests. Otherwise one day by typing example.com one will end up
in example.net because some intermediary server's admin will decide
that example.net is better as the target address. It is easy to start
and much harder to stop so, yes, such things should be squeezed out
right at the beginning.

VK schrieb am 27.05.2008 15:50:

On May 27, 2:23 pm, Gregor Kofler <use... (AT) gregorkofler (DOT) at> wrote:
RobG meinte:

Seems developers of mobile applications are pretty much devoted to UA
sniffing:
URL:http://wurfl.sourceforge.net/vodafonerant/index.htm
A real expert:

"All the way since the inception of the web, HTTP clients have had
unique User-Agent (UA) strings which let the server know who they were."

What offense to you have to that?
User-Agent string is intended to let the server know who is requesting
the page, this is why they were created and how they are used since
Mosaic. Is your version being that it is a NCSA invented beatifying
ornament of HTTP request? Sorry to say then that it is utterly wrong
though semi-poetic version.

User-Agent are unique to each browser and each browser version unless
the browser code is manually reverse-engineered and altered by the end
user in violence of EULA. Such situation is definitely possible but

Opera was shipped a while in the default config of cloning IE6 to avoid
user problems.

serious solutions do not normally account users surfing the Web with
hacked software: unless it is a special statistics collector.

i dont think all those opera users hacked their software .-)

--
Mit freundlichen Grüßen
Holger Jeromin

On May 27, 6:48 pm, Holger Jeromin <news03_2... (AT) katur (DOT) de> wrote:

Opera was shipped a while in the default config of cloning IE6 to avoid
user problems.

Opera never was shipped User-Agent string _cloning_ any of existing IE
User-String. In some older releases Opera's User-Agent string was
partially altered to contain some keywords most oftenly used for
server-side or client-side IE detection so it had "MSIE" and
"Microsoft Internet Explorer" in it. It still was vendor-specific
enough to detect Opera if one was targeted to detect exactly Opera and
not IE.

Opera is not alone nor it is the champion in this producer-humiliating
activity. All time the best in my collection are remaining some
releases of Safari with User-Agent string - and I'm not kidding -
"Mozilla/5.0 MSIE Microsoft Internet Explorer like Gecko; KHTML..."
with the part after KHTML giving the actual Safari-specific info.
Sometimes when I am depressed this "Microsoft Internet Explorer like
Gecko" cheers me up. :-)

In any case the User-Agent spoofing business faded out way of time ago
as the "brand putting down" impact proved to be much higher than any
possible immediate benefits.

For the client-side there is also navigator.vendor string which is
more easy to parse for the producer name. It is not relevant for the
original Vodafon topic.

On May 27, 8:55 pm, Gregor Kofler <use... (AT) gregorkofler (DOT) at> wrote:

A-ha. How come, that Firefox explicitly allows me to set my UA
identification string to whatever I want?

You are sounding like a child, really: "Look ma', I have just pulled
out a wheel out of my new toy car!. Am I cool or what?" :-)

I have no intention to stop you from your hacking exercises. After
Firefox is done
http://www.beatnikpad.com/archives/2...ent-in-firefox
please feel free to start pulling out another wheel :-)
http://www.pctools.com/guides/registry/detail/799/

As I said before it is perfectly possible in this or another way for
absolutely any browser on the market. Another question that it has no
relation to the real Web development. The amount of users going
through the User-Agent string change doesn't exceed the amount of
other inadequately acting groups of Web surfers: Lynx users, self-made
browser users, officially registered psycho cases with Internet
access: and other inevitable small shrink one has to expect in any
business involved with a big amount of people.

There is one puzzling point that bothers while reading similar to that
discussions. It is a bit OT to OP but overall fits well into User-
Agent string questions.
Let's us imagine that the cell phone radio pollution indeed did get
some unexpected brain cells damage effect - so every single user in
the world first thing of all finds the appropriate hack for her
proffered browser and changing the current User-Agent string to "There
is no Web, there is only Xul". Therefore server-side detection and
User-Agent detection as such become totally unreliable. The only hope
is the client-side feature detection (and a few remaining sane doctors
searching a remedy from the brain disease). So the dream of some came
true. Cool. I just have one small question:
Look at the User-Agent hacks for Firefox or at IE. Now look at this
chunk of code:
document.getFoobar = new Function;
window.alert(typeof document.getFoobar);
or
document.getFoobar = new Object;
window.alert(typeof document.getFoobar);
So now the question: who and why had decided that much more labor and
skill intensive procedure will be most probably used - but an easy
like a moo-cow runtime feature spoofing will never be? Was it some
common "internal feeling" or at least once there was a reasonable
discussion on the subject? What were the arguments? Thank you in
advance.

On May 27, 8:23 pm, Gregor Kofler <use... (AT) gregorkofler (DOT) at> wrote:

RobG meinte:

Seems developers of mobile applications are pretty much devoted to UA
sniffing:

URL:http://wurfl.sourceforge.net/vodafonerant/index.htm

A real expert:

"All the way since the inception of the web, HTTP clients have had
unique User-Agent (UA) strings which let the server know who they were."

He should probably try to understand web technologies and not write 43kB
of rant.

The complaint is based on the belief that the UA string is the only
viable way to reliably deliver web content and downloads to mobile
devices. The owners of these sites want users to be able to download
games, software, ringtones, etc. to their mobile devices and have
confidence that they should work.

As a result, they have a database of thousands of UA strings that are
used to identify the browser and device and attempt to deliver
appropriate content.

It seems to me that they've ignored the simplest of solutions, which
might include delivering small test downloads so the user can discover
what works on their device (or not), or to just ask the user what
device they are using.

I don't have any experience in developing or using such sites, I was
wondering if anyone here has and can comment on the situation.



--
Rob

RobG wrote:

On May 27, 8:23 pm, Gregor Kofler wrote:
RobG meinte:

Seems developers of mobile applications are pretty much devoted
to UA sniffing:

URL:http://wurfl.sourceforge.net/vodafonerant/index.htm

A real expert:

"All the way since the inception of the web, HTTP clients have
had unique User-Agent (UA) strings which let the server know
who they were."

If that statement were true there would be precisely zero examples of
web browsers with User Agent strings that were not unique (i.e. that
could not be discriminated from all other user agent's UA strings by
examining the sequence of characters that they contain). Such a
statement is proved false by any single example of a browser with a
(default) UA string that is indistinguishable from that of another
browser. So when a 2003 release of IceBrowsr 5 defaults to a UA string
of "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98)" (observing that IE
5.0 installed on Windows 98 would use precisely the same sequence of
characters in its default UA string) we know the assertion to be
false.

The second browser with a non-unique UA string makes the point stronger,
and the third stronger still, and so on, but the fact that the belief is
false is revealed by just the first.

It is virtually inevitable (at least in a technical context) that acting
on a false belief will have more or less undesirable consequences.

He should probably try to understand web technologies and not
write 43kB of rant.

The complaint is based on the belief that the UA string is the
only viable way to reliably deliver web content and downloads
to mobile devices.

That would be a false belief, because even if UA string based browser
sniffing were the best alternative available that still would not make
it "reliable". And the article is about the author's realisation of one
of the many reasons why it is not reliable.

The owners of these sites want users to be able to download
games, software, ringtones, etc. to their mobile devices
and have confidence that they should work.

So they must have information relating make/model of these devices
in order to know what 'should work' on them.

As a result, they have a database of thousands of UA strings

Literally thousands? That is realistic if you are talking about all
models of device having an embedded web browser, but with thousands of
possible permutations the sanity of any test based on searching for any
short character sequence in a UA string looks extremely doubtful.

that are used to identify the browser and device and attempt
to deliver appropriate content.

But that also implies a database of makes/models relating to
capabilities.

It seems to me that they've ignored the simplest of solutions,
which might include delivering small test downloads so the user
can discover what works on their device (or not), or to just ask
the user what device they are using.

I would have thought a very simple solution would be to have the user
identify the make and model of the device they were using (as they are
probably in a very good position to know as it is usually written on the
casing) and then filter the available content based on that information.

I.E. First page; pick a manufacture: Nokia, Sonny, Motorola, etc. ->
Second page; pick a model range - > 0 -1000, 1001 - 2000 ... (or
whatever) -> third page; pick the specific model -> forth page; pick a
category: ring tones, games, etc. All either dynamically generated or
pre-processed from the database of model/capability/resource
relationships that must exist in order for the UA string nonsense to be
as functional as it is.

That would also deal with one of my pet complaints about this "too
clever for its own good" content delivery, which is that I (or any user)
might not want to download something for the device that I am using at
the time, but instead I might want to download something for a different
device. Apple have done a good job of illustrating this issue when you
attempt to download Safari updates. I have a number of PCs running
different operating systems and I have a Mac mini (that I use for
testing Mac browsers). The Mac mini does not have an Internet
connection, and in any event I much prefer downloading software at work
(where the bandwidth is extremely good and so the process very fast) and
putting in on a flash disc for transfer to the Mac. But Apple's web site
has started insisting on only presetting options to download Safari
versions for the OS it deduces you are using from the UA string of the
client visiting their site. Left to its own devices it will only give me
options to download Windows Safari. Obviously I get round Apple's
attempt to be 'clever' by changing IE's UA string, but I should not have
to, and I certainly don't like the implication that I am not capable of
deciding what I should be downloading for myself.

That illustrates why UA strings are not a viable means of identifying
web browsers; as soon as web site developers started using them to
discriminate about what they would serve to the client, while making
poor choices about what they were serving, it became necessary
circumvent the actions of those developers and the available approach
was to change the UA string. And that all started over a decade ago, and
was formalised in 1999 in RFC 2616, so nobody has a good excuse for not
understanding the true nature of the UA string after all this time.

I don't have any experience in developing or using such
sites, I was wondering if anyone here has and can comment
on the situation.

I don't have any experience in that particular area either, but I bet
things would go more smoothly for them if their developers were not
labouring under false beliefs, technically ignorant (about HTTP in this
case) and a little more willing to get on with correcting their mistakes
instead of blaming the external factors that expose them.

Richard.

On May 28, 3:37 pm, "Richard Cornford" <Rich... (AT) litotes (DOT) demon.co.uk>
wrote:

[snip]

That illustrates why UA strings are not a viable means of identifying
web browsers

If you were a system administrator and you wanted to send gzipped
JavaScript files to save bandwidth, how would you determine which
browsers could accept gzipped files and which could not? I have only
read explanations how to do this with the user agent string. I have
some ideas but have never tried any of them. For example, send a
gzipped file and then a non-gzipped file to see if the first file
worked. I'm curious what you would do or if you have any experience
with this area where user agent string is used.

[snip]

Peter

On May 29, 12:22 pm, Peter Michaux <petermich... (AT) gmail (DOT) com> wrote:

On May 28, 3:37 pm, "Richard Cornford" <Rich... (AT) litotes (DOT) demon.co.uk
wrote:

[snip]

That illustrates why UA strings are not a viable means of identifying
web browsers

If you were a system administrator and you wanted to send gzipped
JavaScript files to save bandwidth,

Many people consider bandwidth to be bytes sent by the server, whereas
it should be measured as bits transmitted by the modem after the
addition of network stuff and compression.

I asked about this on the Apache news group (figuring they'd be a
suitably open-minded and knowledgable lot) and got two responses, the
one I listened to suggested it is pointless zipping files because:

1. modems are optimised to compress text for transmission and so
likely compress files better (or at least no worse) than general
purpose compression programs

2. letting the modem do the work saves CPU effort at both ends

3. if the file is zipped before transmission, subsequent modem
compression may actually result in more data to transmit (though
likely not much more)

Anyhow, here's a link:

<URL:
http://groups.google.com.au/group/al...55117c72c4b5e0

how would you determine which
browsers could accept gzipped files and which could not? I have only
read explanations how to do this with the user agent string. I have
some ideas but have never tried any of them. For example, send a
gzipped file and then a non-gzipped file to see if the first file
worked. I'm curious what you would do or if you have any experience
with this area where user agent string is used.

There is an Open Mobile Alliance (OMA) user agent profile
specification that UAs can use to transmit capability and preference
information:

<URL: http://www.openmobilealliance.org/te...20011020-a.pdf

It is supposed to be used by developers of WAP and other mobile-
specific sites to identify device characteristics and then serve
appropriate content. The fact that projects like WURFL seem to be
more popular for such things shows that developers don't trust the
profile information to do the job.

Whether UA string or agent profiles are used, it seems that feature
detection has been abaondoned (perhaps it was never seen as a serious
contenter) as a strategy for determining mobile UA capability.

Maybe feature detection is employed as a second strategy for minor
irregularities and after device capability has already been defined
(or assumed) fairly precisely (where "precisely" is used in its
mathematic sense, which is quite different to "accurately").


--
Rob