syncing problem on Solaris and *BSD systems, 0 lines in dlf (was: Re: sendmail syslog files ?)

[Joost van Baal]

On Wed, Nov 07, 2001 at 04:55:13AM -0800, Faruk Celik wrote:
> 
> --- Joost van Baal <joostvb at logreport.org> wrote:
> > On Tue, Nov 06, 2001 at 10:54:46PM -0800, Faruk
> > Celik wrote:
> > > 
> > > I'm trying to analyze our Sendmail SMTP logs.
> > > I'm using Sendmail on Solaris 2.7 and 
> > > lire-full-20011017.tar.gz.
<snip>
> > > and LIRE created an empty /tmp/report.txt and
<snip>
> "0 lines of information from the 37458 lines in the
> log......"
<snip>
> >  grep 'sendmail\[' /var/log/syslog | lr_run
> > lr_log2report /tmp/error email sendmail \
> >    > /tmp/report.txt
<snip>
> 
> After I've read your mail I've changed my
> $(prefix)/etc/lire/defaults.local like that:
> 
> -------------My new defaults.local STARTED-----------
> # $(prefix)/etc/lire/defaults.local - local settings
> for Lire 
> #   system on mymachine ;)
> # to be sourced by $(prefix)/etc/lire/defaults
> # automatically generated by lr_config(1) on Tue Nov 
> 6 16:48:45 EET 2001
> # --------------------------------
> # I take out the my machine related informations. 
> # --------------------------------
> 
> KEEP=1
> DEBUG=1
> ARCHUVE=1

Oops, that should be 'ARCHIVE'
                          ^

> LOGGING=stderr
> DEFAULT_OUTPUT_FORMAT=txt
> LR_TARGET_USER=sysadmin
> LR_USERLEVEL=advanced
> LR_MAX_MEMORY=40Megs
> jobfiles_weekly=""
> jobfiles_daily=""
> 
> -------------My new defaults.local FINISHED-----------
> 
> And I got a lot of debug_output to console. I attached
> a file named "lire_shell_output.txt" to this mail.

Thanks!  This helps.

<snip>
> I'm not going to give up Joost ;-]. 
> Because I liked your project. 

Well, thanks!

> And I'm sure you know the absence of a
> real/nice/detailed "sendmail log analyser" tool. 
> 
> (I checked Sendmail FAQ
> http://www.sendmail.org/faq/section4.html#4.7 .
> Most of them didn't give me number of rejected ,
> successfull mails and they're producing so simple
> reports.)
<snip>

Just sent the maintainer an email, thanks for the pointer.



OK, the debug output features:

email sendmail lr_tag-20011107134547-27784 lr_log2xml info gonna run 
  sendmail2dlf  > 
  /$(prefix)/tmp/lr_log2xml.sendmail.lr_tag-20011107134547-27784.27797.dlf

email all lr_tag-20011107134547-27784 sendmail2dlf info read 39194 lines, 
  output 62197 lines

email sendmail lr_tag-20011107134547-27784 lr_log2xml info lr_db_store 
  lr_tag-20011107134547-27784 dlflines        0 succeeded

email all lr_tag-20011107134547-27784 lr_dlf2xml info 'DLF contains 0 
  records; start and end time are unavailable

I guess we've hit a bug.

The problem seems to be the generated dlf file,
(/$(prefix)/tmp/lr_log2xml.sendmail.lr_tag-20011107134547-27784.27797.dlf
in your case ), is not synced to disk, before it's being opened again,
to run

 LINES=`wc -l < $DLFFILE`

(that's in the lr_dlf2xml code.)

When I tested the script, I had to add some sync's.   I made a mistake,
however.  The sync call is not portable.

SunOS 5.7 fsync(3C)

     The fsync()  function  is  different  from  sync(),
     which  schedules  disk I/O for all files  but returns before
     the I/O completes. The fsync() function forces all outstand-
     ing  data  operations to synchronized file integrity comple-
     tion

The GNU file utilities `sync' "ensures everything in memory is written
to disk."

The linux sync systemcall:

       According to  the  standard  specification  (e.g.,  SVID),
       sync()  schedules  the  writes,  but may return before the
       actual writing is done.   However,  since  version  1.3.20
       Linux  does actually wait.

BSD-ish systems need fsync(2), like Solaris, to get guaranteed flush to
disk.  Unfortunately, there's no commandline interface on Solaris to
call fsync.  FreeBSD has got /usr/bin/fsync . OpenBSD has no commandline
fsync.  Perl doesn't offer an fsync interface.  We really _need_ the
sync, since what's done is:

 $TODLF $FLAGS > $DLFFILE

 LINES=`wc -l < $DLFFILE`

 lr_dlf2xml $SUPERSERVICE $REPORT_CFG $DLFFILE

and lr_dlf2xml really needs a file to operate on.

I don't think cludging with tee(1) will increase robustness.

I need some thinking on this.  I'll get back to it tomorrow.  It surely
is an interesting bug :)


OK, there is a cludgy workaround:

I used ${prefix} /usr/local/bin

cochon at abramowitz:~% export
  PATH=/usr/local/bin:/usr/local/libexec/lire:/usr/local/libexec/lire/email:/usr/bin:/bin

cochon at abramowitz:~% grep 'sendmail\[' /var/log/syslog | sendmail2dlf > \
  /tmp/sendmail.dlf

cochon at abramowitz:~% grep 'sendmail\[' /var/log/syslog | \
 lr_run sendmail2dlf > /tmp/sendmail.dlf

cochon at abramowitz:~% lr_run lr_dlf2xml email /usr/local/etc/lire/email.cfg \
 /tmp/sendmail.dlf > /tmp/sendmail.xml

cochon at abramowitz:~% lr_run lr_xml2ascii < /tmp/sendmail.xml > \
 /tmp/sendmail.report

This will give you an ascii report.

You'll hear about the sync problems.  I'd very much like to get this bug
fixed.

Bye,

Joost

-- 
Joost van Baal              . .           http://www.logreport.org/
                           .   .
/^LogReport$/               . .               joostvb at logreport.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.logreport.org/pipermail/questions/attachments/20011107/95c90b51/attachment.bin