Help understanding some Lire conceptual issues

[Jim Lancaster]

Thanks for the very detailed response.  It is extremely helpful.

[snip]
> 	You can transmit most logs directly via the network through 
> "syslog".  The initial implementation was UDP-based, which can lose a 
> hell of a lot of data on a busy network.  However, modern 
> implementations of syslog (e.g., syslog-ng, nsyslog, ssyslog, etc...) 
> can be done over TCP, which won't lose traffic.
> 
> 	Failing that, you can use network-accessible 
> filesystems such as 
> NFS, AFS/Coda, SMB/Samba, etc....
> 
> 	If that doesn't work, then you can move the files via 
> the network 
> with tools like ftp, rcp, scp, etc....
> 
> 	Pretty much all of these methods have variants that will work 
> across a wide variety of OSes, including virtually all *nix 
> implementations but also including non Unix-like OSes such as those 
> from Microsoft, MacOS 9 and earlier, etc....
> 

As I re-read the User's Manual, I see there are two basic
configurations: client-server, and standalone. It appears that
client-server relies on mail, while one can use command line tools to
process local logs in the standalone configuration.  There is a
section-heading in Chapter 3 called 'syslog' (p. 19 of PDF), but no
explanation follows.  The manual appears incomplete, as it does in the
sections for 'Automatically Processing Log Files In A Server Farm' in
Chapter 4, so your response is very helpful in filling in the gaps.

Am I to assume that Lire, itself, provides no mechanism for
collecting/accumulating/storing the log files to be processed in the
standalone configuration?  Does Lire leave it up to you to get the log
files to the server by whatever method?

[snip]
> 	Of course, at the time I was working at the largest ISP in 
> Belgium, so it's not really surprising that we'd have really massive 
> web proxy logs.

Completely off-topic:  My operations manager is from Brussels. He came
to the states as high school exchange student and never left.  One of
our best clients is a South African.  When he met Serge for the first
time, he noticed a trace of an accent--actually he noticed the *lack* of
an accent--and asked Serge where he was from.  Serge says, "Would you
believe East Texas?" (a region of almost unintelligibly-thick accents,
and obviously the wrong answer).  Julian didn't fall for it, so Serge
tells him the real story.  Next thing I know, the two of them are
jabbering away in Afrikaans, Julian's native language.  I had no idea
that Afrikaans and Flemish were so closely related.  It was like
old-home week.  Your name surname appears to be neither Flemish or
French--Are you English? (Please ignore the question if I am being too
personal.)

> >  Do you mean that you would not use 'Lire' in a production 
> > environment?  Or that you would not use e-mail to transmit 
> the logs in 
> > a production  environment?
> 
> 	I wouldn't use e-mail as a transmission method in a 
> production environment.
> 
> 	I would definitely use lire as a log processing system in a 
> production environment, and I look forward to the day when I can get 
> rid of all those damn shell scripts for which I have accumulated 
> maintenance responsibility, and point people at lire instead.

If I might ask (especially after the last question <g>), what is your
general configuration?  Do you have Lire running on the same box as the
services generating the logs?  Or are you using some method to move the
logs to the Lire server?  Do you have multiple Lire servers?

[snip]
> >  4.  Rigid storage structure - The database structure where the log 
> > data  is stored forces logs of all different types to conform to a 
> > single  table definition.
> 
> 	Well, lire has this same problem, to a degree.

I can believe it, but isn't the DFL concept a step/leap in the right
direction?

[snip]
> >  8.  Poor alerting/notification mechanisms - Most create a 
> > notification  for every event that matches a trigger.  This is not 
> > satisfactory in a  world where a small failure may result 
> in the same 
> > error being generated
> >  hundreds-- if not thousands (or in your case, 
> millions)--of times within
> >  a short period.  (e.g., tape drive hardware or SCSI-bus failure.)
> 
> 	That's a real-time network/system monitoring tool, which is 
> totally different.  Lire doesn't have anything in this area -- it is 
> reporting only.
> 

This last issue I added mostly for completeness--It is one that most
logging systems have.  Most logging systems tack on an
alerting/notification mechanism, but the Lire manuals are very explicit
on this topic:  "Lire is a batch processor, it isn't a real-time log
analyzer" (p. 5).

It is my contention that log analysis and alerting cannot, as of now, be
completely automated. Good analysis is the result of good
interpretation, which still requires human intervention.  For the
forseeable future, someone will have to read and interpret the logs.
Can this task be made easier?  Of course.  For example, in my in-house
log processing system, I first summarize events by customer, server,
then frequency of occurence.  (How many times has Event 'x' happened
during the reporting period?)  Using this method, my staff can
immediately focus on the big (recurring) problems first.  Our system is
setup so that the reviewer can check a box next to an event that needs
further follow-up, then send a notification to the network administrator
assigned to that account.  Without this intermediate review step, an
automated alerting system will send an alert for every error message
generated.  Not very useful some of the time.

> 	Try nagios, bigbrother, bigsister, pong, spong, or any of those 
> sorts of programs.  If you want network/system trending, look at 
> tools like mrtg, rrdtool (plus optional front-ends such as cricket 
> and orca), etc....  Moreover, none of these systems really have 
> anything to do with log processing -- by the time the data has hit 
> the logs, it's not real-time any more.
> 

[snip]
I have looked at most of these, paying particular interest to MRTG,
RRDTool.  I completely agree with you.

Thanks again for your detailed respone.

Jim


***************** Announcement *****************
We are pleased to announce that we have officially changed our name to Sagiss, LLC. Please update my e-mail address and contact information in your records to reflect our new domain name, "sagiss.com."
Our mailing address and phone numbers will remain the same.

To find out more, please visit our website: http://www.sagiss.com, or call us at 214-989-0440

-- 
To UNSUBSCRIBE, email to questions-request at logreport.org with a subject of 
"unsubscribe". Trouble? Send an email with subject "help" to 
questions-request at logreport.org