dbmail support for msgstore superservice

[Edwin Groothuis]

Greetings,

I've created a log-processor for DBmail (www.dbmail.org). Attached are:
- The convertor dbmail2dlf.in (Note: before going into CVS, you
  have to tweak for the reasons it should be called .in)
- Two more reports.
- msgstore.cfg diff

Right now I only support the POP3 logging of DBmail since its IMAP
logging is sh*t^2.

Edwin

-- 
Edwin Groothuis      |            Personal website: http://www.MavEtJu.org
edwin at mavetju.org    |    Weblog: http://www.mavetju.org/weblog/weblog.php 
bash$ :(){ :|:&};:   | Interested in MUDs? http://www.FatalDimensions.org/
-------------- next part --------------
#! /usr/bin/perl -w

use strict;
use lib '/usr/local/share/perl5';

use Lire::DlfSchema;
use Lire::Syslog;
use Lire::Program qw/:msg :dlf/;

use vars qw/ $dlf_maker $dlflines $debug /;

sub print_dlf {
    my $dlf=$dlf_maker->($_[0]);
    print join( " ", @$dlf ), "\n";
    $dlflines++;
}

my $schema = eval { Lire::DlfSchema::load_schema( "msgstore" ) };
lr_err( "failed to load msgstore schema: $@" ) if $@;

$dlf_maker =
  $schema->make_hashref2asciidlf_func( qw/time localserver client_ip user protocol prot_cmd messages_downloaded bytes_downloaded stored_messages stored_size session_duration status/);

my $lines	= 0;
$dlflines	= 0;
my $errorlines  = 0;
my @server_msg  = ();
$debug          = 0;
my $syslog_parser = new Lire::Syslog;

my $parser = new Lire::Syslog;
init_dlf_converter( "msgstore" );
my $failed_line = undef;
my %data = ();
while ( <> ) {
    chomp;
    $lines++;

    my $rec = eval { $syslog_parser->parse( $_ ) };
    if ($@) {
	lr_warn( "line $. is an invalid syslog message: $@" );
	$errorlines++;
    }

    next unless defined $rec->{process} && $rec->{process} =~/^dbmail\//;

    my $pid	= $rec->{pid};
    my $line	= $rec->{content};

    my %dlf=();
    $dlf{time}		= $rec->{timestamp};
    $dlf{localserver}	= $rec->{hostname};
    $dlf{protocol}	= "pop" if ($rec->{process} eq "dbmail/pop3d");
    $dlf{protocol}	= "imap" if ($rec->{process} eq "dbmail/imap4");
    # yes, they might be uninitialized but will not anymore when they're
    # used later on:
    $dlf{client_ip}	= $data{$pid}{client_ip};
    $dlf{user}		= $data{$pid}{user};;

    if ($line=~/^handle_client..: incoming connection from \[([\d\.]+)\]$/) {
	$data{$pid}{client_ip}	= $1;
	$data{$pid}{time_start}	= $rec->{timestamp};

	$dlf{client_ip}	= $data{$pid}{client_ip};
	$dlf{user}	= "[unauthenticated]";
	$dlf{prot_cmd}	= "connect";
	print_dlf(\%dlf);
	next;
    }

    if ($line=~/^handle_client..: user [\w\d]+ logging out \[message=(\d+), octets=(\d+)\]/) {

	# ignore leftovers from previous logfiles
	next if (!defined $data{$pid}{client_ip});

	$data{$pid}{messages_stop}	= $1;
	$data{$pid}{bytes_stop}	= $2;
	$data{$pid}{time_stop}	= $rec->{timestamp};

	$dlf{prot_cmd}		= "close";
	$dlf{messages_downloaded} = $data{$pid}{messages_start} - $data{$pid}{messages_stop};
	$dlf{bytes_downloaded}	= $data{$pid}{bytes_start} - $data{$pid}{bytes_stop};
	$dlf{session_duration}	= $data{$pid}{time_stop} - $data{$pid}{time_start};

	print_dlf(\%dlf);
	$data{$pid}=();
	next;
    }

    if ($line=~
	/^pop3..: user ([\w\d]+) logged in \[messages=(\d+), octets=(\d+)\]$/) {

	# ignore leftovers from previous logfiles
	next if (!defined $data{$pid}{client_ip});

	$data{$pid}{user}		= $1;
	$data{$pid}{messages_start}	= $2;
	$data{$pid}{bytes_start}	= $3;
	$data{$pid}{time_start}		= $rec->{timestamp};;

	$dlf{user}	= $data{$pid}{user};
	$dlf{prot_cmd}	= "login";
	$dlf{stored_messages}	= $data{$pid}{messages_start};
	$dlf{stored_size}	= $data{$pid}{bytes_start};
	print_dlf(\%dlf);

	next;
    }

    #
    # problemos et al
    #
    if ($line=~
	/^pop3..: user \[([\w\d]+)\] tried to login with wrong password/) {

	# ignore leftovers from previous logfiles
	next if (!defined $data{$pid}{client_ip});

	$dlf{user}	= $1;
	$dlf{prot_cmd}	= "badlogin";

	print_dlf(\%dlf);
	%data=();
	next;
    }

}

end_dlf_converter( $lines, $dlflines, $errorlines );

__END__


=pod

=head1 NAME

dbmail2dlf - convert DBMail IMAP or POP log files to the msgstore DLF

=head1 SYNOPSIS

B<dbmail2dlf> STDIN STDOUT

=head1 DESCRIPTION

This program converts DBMail syslog files generated by the IMAP or
POP services to the msgstore DLF.

=head1 LIMITATIONS

The IMAP logging is hopeless and not supported yet.

=head1 EXAMPLES

To process a log as produced by DBMail:

 $ dbmail2dlf < mail-log

dbmail2dlf will be rarely used on its own, but is more likely
called by lr_log2report:

 $ cat /var/log/mail-log | lr_run lr_log2report dbmail

=head1 SEE ALSO

nmsmmp2dlf(1), nmsstore2dlf(1)

=head1 VERSION

$Id$

=head1 AUTHORS

Edwin Groothuis <edwin at mavetju.org>

=head1 COPYRIGHT

Copyright (C) 2002 Edwin Groothuis <edwin at mavetju.org>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

3. The name of the copyright holder may not be used to endorse or promote
products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO
EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY
OF SUCH DAMAGE.

=cut
-------------- next part --------------
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE lire:report-spec PUBLIC
  "-//LogReport.ORG//DTD Lire Report Specification Markup Language V1.0//EN"
  "http://www.logreport.org/LRSML/1.0/lrsml.dtd">
<lire:report-spec xmlns:lire="http://www.logreport.org/LRSML/"
 superservice="msgstore" id="top-user-most-messages">

 <lire:title>Top User Most Message Store Report</lire:title>
 <lire:description>
  <para>This report specification will generate a report showing the
   the users that transported the most message from the messsage
   store. The number of users to include in the report is configurable.
  </para>
 </lire:description>

 <lire:param-spec>
  <lire:param name="users_to_show" type="int" default="10">
   <lire:description>
    <para>This parameter controls the number of users to include in
     the report.</para>
   </lire:description>
  </lire:param>
 </lire:param-spec>

 <lire:display-spec>
  <lire:title>Users With Most Messages Downloaded, Top $users_to_show</lire:title>
 </lire:display-spec>

 <lire:filter-spec>
  <lire:eq arg1="$prot_cmd" arg2="close"/>
 </lire:filter-spec>

 <lire:report-calc-spec>
  <lire:group sort="-messages_downloaded_total" limit="$users_to_show">
   <lire:field name="user"/>
   <lire:sum name="messages_downloaded_total" field="messages_downloaded"/>
  </lire:group>
 </lire:report-calc-spec>

</lire:report-spec>

<!-- 
Keep this comment at the end of the file
Local variables:
mode: xml
-->
-------------- next part --------------
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE lire:report-spec PUBLIC
  "-//LogReport.ORG//DTD Lire Report Specification Markup Language V1.0//EN"
  "http://www.logreport.org/LRSML/1.0/lrsml.dtd">
<lire:report-spec xmlns:lire="http://www.logreport.org/LRSML/"
 superservice="msgstore" id="top-user-most-bytes">

 <lire:title>Top User Most Message Store Report</lire:title>
 <lire:description>
  <para>This report specification will generate a report showing the
   the users that transported the most bytes from the messsage
   store. The number of users to include in the report is configurable.
  </para>
 </lire:description>

 <lire:param-spec>
  <lire:param name="users_to_show" type="int" default="10">
   <lire:description>
    <para>This parameter controls the number of users to include in
     the report.</para>
   </lire:description>
  </lire:param>
 </lire:param-spec>

 <lire:display-spec>
  <lire:title>Users With Most Bytes Downloaded, Top $users_to_show</lire:title>
 </lire:display-spec>

 <lire:filter-spec>
  <lire:eq arg1="$prot_cmd" arg2="close"/>
 </lire:filter-spec>

 <lire:report-calc-spec>
  <lire:group sort="-bytes_downloaded_total" limit="$users_to_show">
   <lire:field name="user"/>
   <lire:sum name="bytes_downloaded_total" field="bytes_downloaded"/>
  </lire:group>
 </lire:report-calc-spec>

</lire:report-spec>

<!-- 
Keep this comment at the end of the file
Local variables:
mode: xml
-->
-------------- next part --------------
Index: msgstore.cfg
===================================================================
RCS file: /cvsroot/logreport/service/msgstore/msgstore.cfg,v
retrieving revision 1.1
diff -u -r1.1 msgstore.cfg
--- msgstore.cfg	5 Aug 2002 02:43:56 -0000	1.1
+++ msgstore.cfg	13 Sep 2002 13:57:07 -0000
@@ -13,3 +13,5 @@
 =section Top user
 top-user-login			users_to_show=20
 top-user-select			users_to_show=20
+top-user-most-messages		users_to_show=20
+top-user-most-bytes		users_to_show=20