Help understanding some Lire conceptual issues

[Jim Lancaster]

> >  2. The idea of mailing raw log files to an 
> analyzing/reporting engine 
> > is  also a stroke of genius. It completely by-passes the 
> messy issues 
> > of log  collection and storage.
> 
> 	E-mail works as a testing mechanism, but not much else.  It 
> absolutely does not scale...

Point taken, but in the documentation I see no other transmission
mechanism. Did I overlook something?

> ...It is not at all unusual to see log files 
> that are tens or hundreds of megabytes in size, or even a gigabyte or 
> more.  At one previous employer, we were generating web proxy logs 
> that were over a gigabyte per hour.

My clients' firewall logs accumulate at a rate of about 10MB per day,
per firewall.  But they are, by far, the biggest logs we generally deal
with.  I would take it for granted that logs of the size you are talking
about would need specialized software to process them.  What do you use
to process a log that accumulates at the rate of 1GB/hr?  Surely that
tool would be overkill for processing tape backup logs, for example.

> 	Still, it's a nice testing mechanism, and can usually be abused 
> to handle small logs, but since e-mail isn't guaranteed reliable, I 
> wouldn't use it in a production environment.

Do you mean that you would not use 'Lire' in a production environment?
Or that you would not use e-mail to transmit the logs in a production
environment?

I have spent several months immersed in trying to solve the problem of
log collection, consolidation and storage. As long as you ignore the log
monsters, like firewall or web proxy logs, there are some commercial
products that do a reasonably good job for one log type or another.  But
they suffer from a number of problems:
1.  Limited scope - They can only manage a limited number of log types.
(e.g., Syslog or Windows event logs only.)
2.  Lack of extensibility - It is difficult (if not impossible) to add
monitoring for new or different log types.
3.  Closed solutions - Except for standard Syslog, they generally
require the use of matching client agent and server.  The client for one
will not work with the server of another solution.
4.  Rigid storage structure - The database structure where the log data
is stored forces logs of all different types to conform to a single
table definition.
5.  Single-enterprise view of the network - There is no provision for
multiple customers and/or multiple locations, greatly limiting their use
to MSPs like me.
6.  They require local network connectivity or the use of VPNs. (i.e.,
Client agents cannot reside on remote subnets or across the Internet
from the server.)
7.  Lack of a web interface
8.  Poor alerting/notification mechanisms - Most create a notification
for every event that matches a trigger.  This is not satisfactory in a
world where a small failure may result in the same error being generated
hundreds-- if not thousands (or in your case, millions)--of times within
a short period.  (e.g., tape drive hardware or SCSI-bus failure.)

Thanks for your response.

Jim


***************** Announcement *****************
We are pleased to announce that we have officially changed our name to Sagiss, LLC. Please update my e-mail address and contact information in your records to reflect our new domain name, "sagiss.com."
Our mailing address and phone numbers will remain the same.

To find out more, please visit our website: http://www.sagiss.com, or call us at 214-989-0440

-- 
To UNSUBSCRIBE, email to questions-request at logreport.org with a subject of 
"unsubscribe". Trouble? Send an email with subject "help" to 
questions-request at logreport.org