Need to know how large the logs can be

[Arnaud Taddei]

We treated much bigger files than 400 MB so it is certainly feasible, I 
remember that we had to process files bigger than 2 GB, there were 
system and perl limits here but we went around.

However it was a long time ago for me and I don't remember the time it 
took. Of course this depends on the criticity for you to process these 
logs and the money you have in hardware. Log processing is more and more 
critical for a variety of reasons. So tell us more what kind of hardware 
you are going to use here ...

A++

Francis J. Lacoste wrote:

>On ven, 2003-05-23 at 16:03, Travis Theune wrote:
>  
>
>>I'm looking around for an analyzer that can handle large (>400MB) log files.
>>One of the main requirements is a "Duration" analisys, which Lire does (and few others), but is also capable of handling such large log files.
>>    
>>
>
>If you aren't in a hurry, Lire will do the job. But it will requires
>significant amount of disk space (5-6 times the size of the
>log file). 
>
>Depending on the CPU you have, this could take several hours to days.
>
>  
>
>>If you could let me know ~ how long it would take to analyze a log file of that size with Lire on a reasonably robust dedicated machine, I'd appreciate it.
>>
>>    
>>
>
>The amount of time it takes depends on the amount of data you have, but
>also on the complexity of the analysis you do. If you build a report 
>which only contains the tables you need, it should take significantly
>less time then generating the report that come by default for web log
>file.
>
>What is best would be to try Lire on a small (~40MB) log file and use
>that as a basis. With such a log file, the processing time should
>scale about linearly from there (i.e. 400MB should take about 10x time
>than a 40MB, add a little more for the extra overhead).
>
>P.S. I'm not aware of any of our users which process log files of that
>size, but again, nobody really told us what is the size of the logs they
>are processing anyway.
>
>Kind regards,
>
>Francis J. Lacoste
>
>  
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.logreport.org/pipermail/questions/attachments/20030709/af5a7c38/attachment.html