Lots of noise about user agent strings

On May 27, 7:11 pm, VK <schools_r... (AT) yahoo (DOT) com> wrote:

On May 27, 8:55 pm, Gregor Kofler <use... (AT) gregorkofler (DOT) at> wrote:

Lots of rantage demonstrating a total lack of understanding of how browsers work snipped

You know, it's better to remain silent and be thought a fool than to
open your mouth and remove all doubt.

On May 29, 9:40 am, Gordon wrote:

On May 27, 7:11 pm, VK wrote:
Lots of rantage demonstrating a total lack of understanding
of how browsers work snipped

You know, it's better to remain silent and be thought a fool
than to open your mouth and remove all doubt.

But would a fool recognise that even when told?

On May 29, 12:40 pm, Gordon <gordon.mc... (AT) ntlworld (DOT) com> wrote:

Lots of rantage demonstrating a total lack of understanding of how browsers work snipped

A strange snip note as I did not talk about of "how browsers work"
whatever that would mean. I explained how the server-side detection
works in commercial solutions and I hinted why client-side browser
identification methods are more reliable than the features detection.
It is for sure that both can be spoofed but the feature detection is
much more easy to spoof, plus ot has two other major faults:
1) it is client-side scripting dependent
2) it often causes round trip for the content (first load feature
detecting page, see what can the agent handle, get it from the
server).
Neither of both is acceptable for mobile device solutions where still
a noticeable share of devices is still asking for WML and WMLScript
respectively, not Javascript; and where a noticeable share of devices
is still working under the legacy GPRS networks with the max speed
9,200 bod with the actual speed rarely going above 7,600 bod because
of implied error correction mechanics.

You know, it's better to remain silent and be thought a fool than to
open your mouth and remove all doubt.

Well, you can open your own - pretty big as I see - mouth to explain
the reliability advantages of feature detection over User-Agent string
study for the prominent desktop UAs.
And I have one humble demand to everyone: anyone willing to post in
the relation of WAP devices _please_ have at least one commercial WAP
project participation in your portfolio; otherwise so far this thread
gets more and more looking like a bunch of virgins discussing secrets
of sex: funny to read but practically pointless. :-)

On May 28, 9:27 pm, RobG <rg... (AT) iinet (DOT) net.au> wrote:

On May 29, 12:22 pm, Peter Michaux <petermich... (AT) gmail (DOT) com> wrote:

On May 28, 3:37 pm, "Richard Cornford" <Rich... (AT) litotes (DOT) demon.co.uk
wrote:

[snip]

That illustrates why UA strings are not a viable means of identifying
web browsers

If you were a system administrator and you wanted to send gzipped
JavaScript files to save bandwidth,

Many people consider bandwidth to be bytes sent by the server, whereas
it should be measured as bits transmitted by the modem after the
addition of network stuff and compression.

I asked about this on the Apache news group (figuring they'd be a
suitably open-minded and knowledgable lot) and got two responses, the
one I listened to suggested it is pointless zipping files because:

1. modems are optimised to compress text for transmission and so
likely compress files better (or at least no worse) than general
purpose compression programs

2. letting the modem do the work saves CPU effort at both ends

I didn't know modems do this. There must be a standard compression
algorithm to ensure the receiver knows how to decompress.

3. if the file is zipped before transmission, subsequent modem
compression may actually result in more data to transmit (though
likely not much more)

Anyhow, here's a link:

URL:http://groups.google.com.au/group/al...ion/browse_frm...

Hmm. This is quite contrary to the current popular thought about
gzipping JavaScript before sending it over the wire.

Steve Souders works for Yahoo!'s performance team and has made many
experiments. I believe he watches total page load time in Firebug and
so that would include modem decompression time. You can see in the
Editorial Review section of the following page there are 14 rules to
speed a page.

<URL: http://www.amazon.com/dp/0596529309>

One of the rule is gzip components.

More confusion added to the pile.

Another issue is that files do not need to be compressed "on the fly"
by the server. They can be pre compressed and if the client can handle
the compressed version then that is the one sent.

[snip]

Peter

Peter Michaux wrote:

On May 28, 3:37 pm, Richard Cornford wrote:
snip
That illustrates why UA strings are not a viable means of
identifying web browsers

If you were a system administrator and you wanted to send
gzipped JavaScript files to save bandwidth, how would you
determine which browsers could accept gzipped files and
which could not?

The HTTP Accept-Encoding header sent with the request would seem
like the obvious place to start (as that is precisely what it is
for).

I have only read explanations how to do this with the user
agent string.

Incredible, and incredibly foolish as HTTP very explicitly allows
proxies to change the encoding. That is, if a client cannot handle
gzip but the proxy can it can ask the server for gzip, decompress
it and send the identity encoded result to the client. It could
also do this the other way around, but it would be unlikely
that doing so would be seen as a good idea. And it could also
disregard any client preference for a compressed encoding and only
make identity requests to servers itself.

So a proxy may or may not send on the client's UA string or
substitute an alternative (which does not matter as the UA string
is arbitrary) and it may or may not impose the same encoding
limitations as the client. That would make looking at the UA
string at all in this context extremely foolish. Indeed more
foolish that ignoring q values in the Accept header when content
negotiating HTML/XHTML.

I have some ideas but have never tried any of them. For
example, send a gzipped file and then a non-gzipped file
to see if the first file worked. I'm curious what you
would do or if you have any experience with this area where
user agent string is used.

I have probably just answered both of those questions(?).

Richard.

Peter Michaux wrote:

On May 28, 3:37 pm, "Richard Cornford" <Rich... (AT) litotes (DOT) demon.co.uk
wrote:
That illustrates why UA strings are not a viable means of identifying
web browsers

If you were a system administrator and you wanted to send gzipped
JavaScript files to save bandwidth, how would you determine which
browsers could accept gzipped files and which could not?

One would scan the Accept-Encoding request header value for "gzip", and then
use gzip(1) or a gzip implementation to compress the message body. There
are libraries like cgi_buffer which are capable of that. Apache 2.0+ has
mod_deflate.

I have only read explanations how to do this with the user agent string.

That is a pity. Had you read RFCs 1945 and 2616 more thoroughly as I think
I recommended to you before, you would also have found the specification of
this header.


PointedEars
--
realism: HTML 4.01 Strict
evangelism: XHTML 1.0 Strict
madness: XHTML 1.1 as application/xhtml+xml
-- Bjoern Hoehrmann

In comp.lang.javascript message <4f28e817-0351-4268-928e-73f7590011ee@j3
3g2000pri.googlegroups.com>, Wed, 28 May 2008 21:27:09, RobG
<rgqld (AT) iinet (DOT) net.au> posted:

Many people consider bandwidth to be bytes sent by the server, whereas
it should be measured as bits transmitted by the modem after the
addition of network stuff and compression.

Not necessarily. My pages are (almost all) small enough that, with any
reasonably recent modem, the download time itself will not upset any
possibly-significant readers. I would support efficiency of transfer,
on general grounds. But what is most important to me is the transfer
count maintained, erratically, by the server system, because there is a
monthly limit.

For any in a similar situation : I put in a ROBOTS.TXT file, denying
access to all but the Home Page INDEX.HTM. Over the course of a month
(during which I re-enabled robot access to some categories, and restored
some hidden topics), access was down by three quarters and still
dropping. I disabled ROBOTS.TXT a fortnight ago, and access had doubled
(to half the original) and is still rising.

--
(c) John Stockton, nr London UK. ?@merlyn.demon.co.uk IE7 FF2 Op9 Sf3
news:comp.lang.javascript FAQ <URL:http://www.jibbering.com/faq/index.html>.
<URL:http://www.merlyn.demon.co.uk/js-index.htm> jscr maths, dates, sources.
<URL:http://www.merlyn.demon.co.uk/> TP/BP/Delphi/jscr/&c, FAQ items, links.

On May 29, 4:19 pm, "Richard Cornford" <Rich... (AT) litotes (DOT) demon.co.uk>
wrote:

Peter Michaux wrote:
On May 28, 3:37 pm, Richard Cornford wrote:
snip
That illustrates why UA strings are not a viable means of
identifying web browsers

If you were a system administrator and you wanted to send
gzipped JavaScript files to save bandwidth, how would you
determine which browsers could accept gzipped files and
which could not?

The HTTP Accept-Encoding header sent with the request would seem
like the obvious place to start (as that is precisely what it is
for).

I believe that the issue is that IE6 claims it can accept gzip but in
actual fact it cannot due to a decompression bug. This bug may only
apply to files over a certain size. This leads to the use of the user
agent string.

I have only read explanations how to do this with the user
agent string.

Incredible, and incredibly foolish as HTTP very explicitly allows
proxies to change the encoding. That is, if a client cannot handle
gzip but the proxy can it can ask the server for gzip, decompress
it and send the identity encoded result to the client. It could
also do this the other way around, but it would be unlikely
that doing so would be seen as a good idea. And it could also
disregard any client preference for a compressed encoding and only
make identity requests to servers itself.

So a proxy may or may not send on the client's UA string or
substitute an alternative (which does not matter as the UA string
is arbitrary) and it may or may not impose the same encoding
limitations as the client. That would make looking at the UA
string at all in this context extremely foolish. Indeed more
foolish that ignoring q values in the Accept header when content
negotiating HTML/XHTML.

Interesting. I do need to read these documents more.

[snip]

Thanks,
Peter

Peter Michaux wrote:

On May 28, 9:27 pm, RobG <rg... (AT) iinet (DOT) net.au> wrote:
On May 29, 12:22 pm, Peter Michaux <petermich... (AT) gmail (DOT) com> wrote:
On May 28, 3:37 pm, "Richard Cornford" <Rich... (AT) litotes (DOT) demon.co.uk
wrote:

1. modems are optimised to compress text for transmission and so
likely compress files better (or at least no worse) than general
purpose compression programs

That's dubious. Modem compression has to be real-time, whereas
general-purpose compression is often run out-of-band. When you gzip,
you usually don't do it while sending the data.

Consequently, modem compression (LAP-M with BTLZ, V.44, MNP-5, or
whatever) has to make different trade-offs than general-purpose
compression. The modem compression standards use smaller dictionaries
and windows than the most aggressive general-purpose LZ compressors
(eg level-9 gzip).

For that matter, the modem compression standards I'm familiar with are
not optimized for text; they're general-purpose adaptive entropy encoders.

2. letting the modem do the work saves CPU effort at both ends

True, unless you're using a software modem (like the so-called
"Winmodems").

I didn't know modems do this. There must be a standard compression
algorithm to ensure the receiver knows how to decompress.

Yes, since the late 1980s or early 1990s. (I don't recall the exact
history and a trivial Google search didn't turn one up in the first
few hits.) Look up MNP-5 (an early, widely-supported proprietary
protocol), V.42bis (the first ITU standard that included compression),
BTLZ (the version of Lempel-Ziv used in V.42bis), and V.44 (a later
and more aggressive compressor).

Compression for modems came along shortly after decent synchronous
protocols (most notably LAPM, an asymmetric HDLC protocol) were
introduced, getting rid of the async framing overhead and allowing for
decent blocking of data.

3. if the file is zipped before transmission, subsequent modem
compression may actually result in more data to transmit (though
likely not much more)

Any growth should be negligible. Modem compression protocols have
uncompressed modes.

Hmm. This is quite contrary to the current popular thought about
gzipping JavaScript before sending it over the wire.

Actually, it isn't, if you study the subject in a bit more depth.

First, as I explained above, out-of-band compression with
general-purpose compressors typically will yield better compression
than what a POTS modem will achieve.

Second, many people are not using POTS modems for their connections.
Sometimes they're on uncompressed LANs. Sometimes they're using
high-speed (so-called "broadband", though that's a misnomer)
connections, like cable or DSL or FiOS. I'll admit that I haven't
looked into what kinds of compression are typically done on those
networks, but simply taking a bunch of received wisdom about POTS
modems and assuming it applies everywhere would be foolish.

Finally, precompressing the payload may have other performance
effects, because you produce smaller TCP segments. Besides saving
somewhat on TCP and IP overhead (probably negligible), you may improve
pacing (particularly if the client or server has poorly-written code
that is vulnerable to things like Nagle/Delayed-ACK Interaction),
reduce stack overhead on both ends, etc.

Steve Souders works for Yahoo!'s performance team and has made many
experiments.

There's a huge body of literature on TCP/IP performance. A handful of
experiments by "Yahoo!'s performance team" might give some decent
general guidelines, but they're not much better ground for
generalization than the "modems compress" folklore is.

The real rule is that there is no set of rules that adequately covers
all situations. If you find there's a performance problem for a
particular case, you can investigate that and often improve it; and
your improvements may result in better performance for most or all of
your users. But blanket recommendations like "compress Javascript" (or
don't) are the litanies of the cargo cults.

--
Michael Wojcik
Micro Focus
Rhetoric & Writing, Michigan State University

On May 29, 4:22*am, Peter Michaux <petermich... (AT) gmail (DOT) com> wrote:

If you were a system administrator and you wanted to send gzipped
JavaScript files to save bandwidth, how would you determine which
browsers could accept gzipped files and which could not?

Looking at the Accept-Encoding header.