Information about optimisation of lire for bigger log file

[Francis J. Lacoste]

On Thu, 2003-10-23 at 15:19, Mohamed Menouar BATTATA wrote:
> Hi,
> I'am trying to configure LIRE to generate weekly report on Provider-1 
> Checkpoint
> I realise that the volume of log retrieved by LEA client (OPSEC) was for my 
> first process about 40 Mo / week
> Lire generate to me the report in 1hour 25 min. great report for QOS.
> But my problem is how to process a log file of 850 Mo / week for my client. 
> under half day
> 
> So i hope there is a solution by using Mysql  can you help me about any 
> valuable information to configure
> this solution on linux .

The current Lire CVS, which should come out as Lire 1.4 beta in the
coming days, uses a SQLite database for the storage of the DLF data and
generates the report using SQL statements on that storage.

Comparisons with Lire 1.3 shows a 57% speed improvement for generating
a firewall report from a 240K Cisco IOS log file (40Megs). The report
was generated in less than 30 minutes (User+System time). 

I do not know how many lines represents a 850Mo LEA log file, but you
can expect to reduce the current processing time by half.

You can either try current CVS or wait a little for the beta to come
out.

Kind regards,

Francis J. Lacoste

-- 
Francis J. Lacoste              . .           http://www.logreport.org
/^LogReport$/               . .               flacoste at logreport.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.logreport.org/pipermail/questions/attachments/20031025/38121fdf/attachment.bin