Re: [PLUG] log rotation with compression Re: logtail2 Re: 'logtail' Re: 'logcheck'

[JP Vossen <jp@jpsdomain.org>]

> Date: Sat, 17 Oct 2009 20:18:02 -0700 (PDT)
> From: Edmond Rodriguez <erodrig_97@yahoo.com>
> 
> Now we introduce log compression after rotation. 
> 
> and more code is needed once again, as the inode is lost.  I saw one site that talked about saving lines from the last tail, then on the next tail, and after discovering the inode changed, pulling up the last compressed log, and trying to match the saved lines to compressed file is indeed a rotated log.  You cannot make the assumption about the last compressed log, because it could be the current log mysteriously changed inodes, or that more than one rotation took place.   
> 
> This seems like a challenging issue.  How to know a previous log, if it is compressed. 

Now you know partly why somelog and somelog.0 are not compressed, and 
somelog.1.gz .. somelog.N.gz are...  :-)


> Date: Sun, 18 Oct 2009 01:46:51 -0400
> From: Dan Widyono <dan@widyono.net>
> 
> Would it be possible to circumvent this issue entirely by using a compressed
> filesystem (e.g. fusecompress), perhaps solely for the logs?

Interesting thought.  I don't know all that much about compression, but 
I'm pretty sure that doing it all at once is faster, more effective and 
more efficient than doing it in tiny dribs and drabs as messages come 
in.  I think that's another part of reason for the scheme above.

The last part being that recent logs are probably more often read and 
used than older ones.  If you compress on write or too soon, you have 
less effective and efficient compression and a greater chance for a need 
to decompress than if you wait and only compress "older" stuff.

With today's disk sizes and CPUs speeds and the fact that most machines 
are doing nothing 90%+ of the time, this doesn't matter.  But remember 
that some of these protocols and processes were given a lot of thought 
and worked out when that kind of thing did have more impact and cost.

My $0.02,
JP
----------------------------|:::======|-------------------------------
JP Vossen, CISSP            |:::======|      http://bashcookbook.com/
My Account, My Opinions     |=========|      http://www.jpsdomain.org/
----------------------------|=========|-------------------------------
"Microsoft Tax" = the additional hardware & yearly fees for the add-on
software required to protect Windows from its own poorly designed and
implemented self, while the overhead incidentally flattens Moore's Law.
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug