Performance and DLF questions

[Wytze van der Raay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Yoanis,

On 16.01.2007 23:16, Yoanis Gil Delgado wrote:
> My server is PowerEdge 1800 with 1GB RAM DDR2 and 2 CPU at 3.0GZ. I
> would like to know why lire needs 15 minutes
> to parse and generate the report for a single postfix log file ? Is that
> normal ? I've notice that memory usage it's pretty low and hard disk
> access isn't too intensive.

I'v observed somewhat similar disappointing performance with processing
sendmail log files, which is very similar to postfix except for the
DLF converter. But it turns out that most of the processing time is spent
in the analysis phase (dlf2xml), which is common between postfix and
sendmail. In my case the elapsed time increased more or linearly with the
size of the logfile, until the logfile exceeded a size somewhere around
100 - 150 MB. Above that size the elapsed time increased dramatically
(non-linearly!),and the processor utilization dropped from 90% to 4%
- -- calculated as (user time + system time) / elapsed time -- while the
system started paging heavily.

You indicate that you don't see intensive hard disk access, which seems
to suggest that your logfile isn't exceeding the "critical boundary" I
mentioned above, but you might want to double-check by looking at the
processor utilization for running your lire analysis process (assuming
that there is not a lot of other activity on your system of course).

> Is there anything I can do?

If your system has started paging, you could split up your logfile in
smaller chunks, and process them separately, then recombine the results.
But if the lire process is still more or less cpu-bound for your size
of logs, there isn't much you can do -- except perhaps looking critically
at the amount of reporting generated by lire: by configuring it to produce
only the subreports that are really interesting to you (instead of the
fairly extensive default set), you may be able to reduce the analysis time.

> On the other hand, i would like to know if there is way to save DLF
> Stores into a MySQL database instead of Sqlite one?

In theory it is possible, but not without a significant programming effort.
Read these pages for some more details:
  http://logreport.org/contact/lists/development/msg00932.php
  http://logreport.org/contact/lists/development/msg00933.php
  http://logreport.org/contact/lists/development/msg00935.php
As far as I know the work alluded to here was never completed, but
you might want to check with the author.

Regards,
- -- wytze
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFFtdg2qs+zhiEbbu8RAkO8AJ4nCJ8Pc/FJwAx9ZppCKzkr+9VBoQCbBmK/
i9sDEseLPsXXGSlfAIaccB8=
=S9b0
-----END PGP SIGNATURE-----

-- 
To UNSUBSCRIBE, email to questions-request at logreport.org with a subject of 
"unsubscribe". Trouble? Send an email with subject "help" to 
questions-request at logreport.org