Re: caching

*Brian* wrote:
<snip />

Lots of advice on how to defeat caches, but I found nothing on how to
calculate a last-modified date or process an if-modified-since header.
It's been a very revealing search.

I recently used:
fishbowl.pastiche.org/2002/10/21/http_conditional_get_for_rss_hackers

To implement: www.andrewu.co.uk/clj/cachecontrol/, which seems to be
doing the job for me but in JScript ASP.
--
Andrew Urquhart
- My reply address is invalid, use: www.andrewu.co.uk/contact/

"Brian" <usenet3 (AT) julietremblay (DOT) com.invalid> wrote


: feeds. But first I need figure out how to get the last-modified date, in
: one case from a cgi search engine, and another from a MySQL database.
:
Not sure if you can determine the last-modified date from search results.
It seems the result set is dynamic (always changing), so there is no point.

I am curious why you can't get it from MySQL. If content is stored
in the database and updates also happen there, it shouldn't be difficult
to update the LAST_MOD_TS field (provided you have one).

This is what I do in serving pages through our CMS. There are "indicators"
that can help in determining whether content are really dynamic (vs. generated).
For generated (content equivalent to being static) pages, I would add the
last-modified date to the header. Otherwise it is left out (i.e. always modified).

--
Long
www.webcharm.ca - Integrated content management web hosting

"Brian" <usenet3 (AT) julietremblay (DOT) com.invalid> wrote

: Long - CM web hosting wrote:
:
: > Brian wrote
: >
: > : But first I need figure out how to get the last-modified date, in
: > : one case from a cgi search engine, and another from a MySQL database.
: >
: > Not sure if you can determine the last-modified date from search results.
: > It seems the result set is dynamic (always changing), so there is no point.
:
: When the site is crawled, the search index has been modified. Until the
: next crawl, the data is not changing. That, however, is not MySQL but
: Perl. I haven't figured out to access the date in the search script, but
: it may not matter. I'd also need to process an "if-modified-since"
: header and return a 304 or 200 depending on the dates, and that's a bit
: beyond my abilities.
:
Now I understand. It seems this should be the responsibility of the webcrawler.
For example, Google had cached my index page but only recached it recently
(and the cached version is out of date again).

: > I am curious why you can't get it from MySQL.
:
: Ah, but that's what I'm trying to figure out. I can query the table for
: general info; the table row that's returned includes a Last_update. Now
: how to get just that one field. If I get it figured out, I'll try to
: post the solution on the web so others won't go banging their head
: against the wall.
:
I assume you know the row_id or file_name value, then perhaps you
can use

SELECT LAST_UPDATE FROM ARTICLES WHERE FILE_NAME='index.html'

Perhaps it is more complicated and needing table joins...

Just a thought.

--
Long
www.webcharm.ca - Integrated content management web hosting

"Brian" <usenet3 (AT) julietremblay (DOT) com.invalid> wrote

: Long - CM web hosting wrote:
:
: > Brian wrote...
:
: >> When the site is crawled, the search index has been modified. Until
: >> the next crawl, the data is not changing.
: >>
: >> I'd also need to process an "if-modified-since" header and return a
: >> 304 or 200 depending on the dates
:
: > It seems this should be the responsibility of the webcrawler.
:
: Uh, first: I'm running the crawler for the site search. (How else could
: I do it?) Second: The point is that clients that repeat a search should
: receive a 304 not modified. How can it be the responsibility of anyone
: but me to check the GET headers and send a 304 if it's appropriate?
:
I didn't get that impression from your OP...so excuse me for going off
on a tangent.

: > For example, Google had cached my index page but only recached it
: > recently (and the cached version is out of date again).
:
: I'm afraid I don't see how that's relevant. But for static pages, it's
: still my responsibility to make sure that Googlebot -- and all other
: conforming http clients -- get a 304 when the situation calls for it.
:
: >>> I am curious why you can't get it from MySQL.
:
: > I assume you know the row_id or file_name value
: >
: > Perhaps it is more complicated and needing table joins...
:
: I'm not querying the table data; I'm querying meta data about the table.
:
Same as above...best of luck then...


--
Long
www.webcharm.ca - Integrated content management web hosting