Re: Apache 2.0.52 file cache / refresh rules (168 Views)
Reply
Honored Contributor
Hein van den Heuvel
Posts: 6,588
Registered: ‎05-19-2003
Message 1 of 6 (168 Views)

Apache 2.0.52 file cache / refresh rules

I'm trying to make a constantly being updated OpenVMS (log) file available as an intranet web location using an Apache server which happens to be there on the system.

I _think_ the most optimal way to do this is to create an (stream-lf) file with "SHR=UPI+GET" and fsync every now and then.

That sharing setting itself works.
From the box I can $TYPE the file and see all fsynced/flushed data. And F$FILE can see EOF +FFB move upon each $FLUSH as expected.

But from the outside, I only get to see the first update and nothing else until the file revision count or revision time is changed.
Dunno which of the two is the actual trigger as they typically change hand in hand.

So when the process CLOSES the file all information becomes visible upon 'refresh'.
And when an outside command like $SET FILE/STAT/SHARE changes RDT/RVN, then a refresh will give all data flushed out at time, but otherwise a refresh, or a fresh window will present the stale first-write date.

It this normal/acceptable?
Is there an Apache Config to influence this behavior? Is this Apache version specific?

I suppose that my workaround would be to close and re-open the file for append instead of fflush/fsync, but I'm reluctant to bow to that.

Comments?

Hein

Server: OpenVMS V8.3-H1, Itanium,
$ mcr APACHE$COMMON:[000000]APACHE$HTTPD -v
Server version: Apache/2.0.52
Server built: Apr 16 2008 07:40:33


Honored Contributor
Craig A Berry
Posts: 495
Registered: ‎05-06-2003
Message 2 of 6 (168 Views)

Re: Apache 2.0.52 file cache / refresh rules

Does your HTML header include

<meta http-equiv="Cache-Control" content="nocache" />

or an "Expires" clause to allow caching only for a specific period of time?

Are there any proxies in between Apache and the client that think they know better than you do how often content can be updated?
Honored Contributor
Hein van den Heuvel
Posts: 6,588
Registered: ‎05-19-2003
Message 3 of 6 (168 Views)

Re: Apache 2.0.52 file cache / refresh rules

Thanks Craig!

I'm actually not writting any HTML.

Just plunking a text file in an accessible directory with a program similar to the one below.

Using "$ TYPE" you can see all record all the time, in groups of 3 records as designed.

Using an exploder directed to http::/x.x.x.x/hein/test.txt you can see all records $ TYPE shows at the activation time, but you will not see any new ones upon refresh nor re-access.

The web-page will show all records again after changing the RDT/RVN, for example after issueing the command:
$ set file/sta/share/log test_file_name

It will be stuck at that sample until the next file attribute update or close (of course).

Hein

#include stdio
#include unistd
main()
{
int i;
FILE *test_file;
test_file = fopen( "test_file_name", "w", "shr=upi,get");
for (i = 1; i < 100; i++) {
fprintf (test_file, "Hello%4d\n", i);
fflush (test_file); // hand over to RMS, to write on rundown.
if (!(i % 3)) fsync(fileno(test_file)); // Update EOF on disk.
sleep (10);
}
}
Honored Contributor
Craig A Berry
Posts: 495
Registered: ‎05-06-2003
Message 4 of 6 (168 Views)

Re: Apache 2.0.52 file cache / refresh rules

hein> I'm actually not writting any HTML.

In that case the header is probably being written for you by a default handler for text/plain documents. You might look at the full data stream from the client (maybe with Lynx or curl) and see what cache control, if any, you're getting.

I've never done this but a quick Google indicates that you can get very fine-grained control over caching by editing your httpd.conf to enable the non-default mod_expires module (which does exist in APACHE$COMMON:[MODULES]). In your case it might make sense to apply an ExpiresDefault directory with a very short expiration time to the directory in which your log files live.

Docs here:

<>



Honored Contributor
Hoff
Posts: 4,962
Registered: ‎01-29-2006
Message 5 of 6 (168 Views)

Re: Apache 2.0.52 file cache / refresh rules

Or write a little DCL or some printf wrappers and roll your own HTML output:

http://labs.hoffmanlabs.com/node/277

There's a port of libwww available via Mr App's site, too.

http://www.johndapps.com
Honored Contributor
John Gillings
Posts: 2,995
Registered: ‎07-31-2003
Message 6 of 6 (168 Views)

Re: Apache 2.0.52 file cache / refresh rules

Hein,

I've banged my head against this issue many times. From a reader process you won't see an EOF update unless you close and reopen the file - I've never found a way around that. Have a look at the source of TYPE/TAIL/CONTINUOUS (make sure you have a bucket handy, it's really gross ;-)

Since you probably don't want to see the whole file on every update, why not put a CGI wrapper around TYPE/TAIL with an adjustable window?

> but I'm reluctant to bow to that.

You're still thinking back in the 70's when opening a file was a big deal. These days, if it avoids hackarounds and complexity, just do the extra opens. You won't be able to measure any real performance difference!

Another option... I have a log file scanner which tracks several hundred log files, scanning the output for interesting strings and sending them to various reporting mechanisms. For some files, it also ships off all records to a PC using a very simple "shadowing" protocol, to reproduce a copy of the file in almost real time.

Web based stuff then hits the (updating) copies on the PC. Production scans the file once, but the logs can be scanned as many times as you like off the PC, with minimal impact on production.

>I _think_ the most optimal way to do this
>is to create an (stream-lf) file
>with "SHR=UPI+GET" and fsync every now and
>then.

I disagree! You're incurring multiple fsyncs regardless of demand. Reduce the frequency and you increase the latency of the log information. By no means an optimal tradeoff.

Rather than burden the writer with accommodating potential readers, have the writer open the file "SHR=UPI+GET+PUT" and just write. Don't bother with fsync.

Now when your reader wants an update they open the file for shared APPEND access, then close it. That way you get an update NOW, rather than at the last fsync, and you only incur the cost of updating the EOF when and if you need it.

If you can't change the source of the image writing the log file to enable shared write, put it in a DCL wrapper which opens a PPF SHARE=(READ,WRITE) and direct log file output to the PPF.
A crucible of informative mistakes
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.