Memory consumption problem

[Wytze van der Raay]

On Wed, May 04, 2005 at 11:06:30AM +0200, Joost van Baal wrote:
> On Wed, May 04, 2005 at 11:24:45AM +0300, Arunas Pranckevicius wrote:
> > 
> > I am using lire 2.0.1 for our servers log analysis
> 
> On GNU/Linux, you wrote on IRC.
> 
> > and encountered one
> > problem:
> > 
> > When using it on out central syslog server,  lr_log2report parses 151 mb
> > sized syslog file and
> > takes all server memory (500Mb RAM plus huge swap) when using syslog
> > parser.
> > The server freeze when go out of free RAM and it work all time with swap
> > then.
> > 
> > Is any way to limit used memory for lire?
> 
> There has been a private discussion among Lire developers, ( Message-ID:
> <4136DEB9.3000709 at nlnet.nl>, Date: Thu, 02 Sep 2004 10:50:01 +0200 )
> about Lire performance.  It's believed the bottleneck is in the dfl2xml
> phase, and performance might be strongly related to the specific
> subreports used.  So, you might gain a lot by disabling some subreports
> you're not interested in anyway.
> 
> I remember there is some setting, to toggle between disk- and
> memory-consumption.  Wytze, Francis: is this toggle removed in some
> previous release?
> 
> Furthermore, it _might_ be 150MB log needs 500MB memory.  I hope others
> will join this discussion...

I am not aware anymore of a software setting to influence Lire's
memory consumption (perhaps Francis is?).

The discussion in Sep 2004 referenced above concerned a sendmail log
of about 350 MB (analyzed on a server with 512 MB + 1 GB swap). In
that case the server did not freeze, but performance was very poor
due to heavy paging of the dlf2xml process (the total run took around
30 hours elapsed time ...). I have not found the time yet to complete
the analysis of this anomaly, but I did some runs with a subset of
that logfile. With 20% (70 MB) of the log data, performance was still
"normal" (i.e. bounded by CPU speed, not by paging/swapping), but with
50% (175 MB) of the log data, significant slowdown occurred due to paging.
Looking at the CPU utilization for the complete job in each case:

	 35 MB data	89%
	 70 MB data	85%
	175 MB data	25%
	350 MB data	 4%

which illustrates the problem nicely.

Turning off one specific report which was suspected to cause a lot
of memory consumption helped a little bit, but not significantly.
Cutting the logfile in smaller parts to prevent heavy paging from
occurring is a much more effective strategy at this moment.

What remains to be done is profiling of the dlf2xml process to
figure out *why* it needs progressively more memory for larger
logfiles, and find a way to fix that if possible ...

Regards,
-- wytze
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.logreport.org/pipermail/questions/attachments/20050506/1bf24646/attachment.bin