Need to know how large the logs can be

[Francis J. Lacoste]

On ven, 2003-05-23 at 16:03, Travis Theune wrote:
> I'm looking around for an analyzer that can handle large (>400MB) log files.
> One of the main requirements is a "Duration" analisys, which Lire does (and few others), but is also capable of handling such large log files.

If you aren't in a hurry, Lire will do the job. But it will requires
significant amount of disk space (5-6 times the size of the
log file). 

Depending on the CPU you have, this could take several hours to days.

> 
> If you could let me know ~ how long it would take to analyze a log file of that size with Lire on a reasonably robust dedicated machine, I'd appreciate it.
> 

The amount of time it takes depends on the amount of data you have, but
also on the complexity of the analysis you do. If you build a report 
which only contains the tables you need, it should take significantly
less time then generating the report that come by default for web log
file.

What is best would be to try Lire on a small (~40MB) log file and use
that as a basis. With such a log file, the processing time should
scale about linearly from there (i.e. 400MB should take about 10x time
than a 40MB, add a little more for the extra overhead).

P.S. I'm not aware of any of our users which process log files of that
size, but again, nobody really told us what is the size of the logs they
are processing anyway.

Kind regards,

Francis J. Lacoste

-- 
Francis J. Lacoste              . .           http://www.logreport.org
/^LogReport$/               . .               flacoste at logreport.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.logreport.org/pipermail/questions/attachments/20030708/0a433164/attachment.bin