The problem of opening large files

[Arnaud Taddei]

thanks for the clarification on this one ...
   - I will check that on my SuSE machine and my Solaris machines

... but as I wrote in my previous message this triggered OTHER toughts
on the issue. So, still from a logical point of view:

   - you cannot assume that people will be in a position of changing
this (think about complex production centers)

   - we are still facing that log files will be split overtime and we
will need to find a way to ensure that we can treat more than one with a
continuation mechanism to decrease the number of orphan and unfinished
events.


Let me know your position.

A++

Wessel Dankers wrote:
> 
> On 2002-03-24 22:01:55+0100, Arnaud Taddei wrote:
> >
> > Recently I showed some customers on what kind of things they could get from an
> > LDAP log using my own scripts (not yet in Lire) in order to make sure that my
> > ideas are ok before we can introduce an LDAP superservice.
> >
> > I ran into the problem that my program had to read a 2GB log file for 3 days of
> > LDAP log and it failed with
> >
> >       Cannot open file: Value too large for defined data type at
> > /homedir/a/admin/project/bin/testopen line 6, <IN> chunk 1.
> >
> > The code was simply:
> >
> >       open(F, $file) || die "Cannot read $file: $!";
> >
> > so I was stuck because I have no infrastructure to split the input file and I
> > was squeezed by time as I had to provide results. After some investigation I
> > discovered that the sysopen call requires to copy the data in the system which
> > means that no way this approach will NEVER work in the general term. So how some
> > programs like less, or cat can open a file that large? Because they use MMAP. As
> 
> Well open(2) certainly doesn't copy a file into memory.
> The problem you're experiencing is what happens when perl doesn't have
> large file support (perhaps because your libc doesn't support it). Once
> perl is compiled with large file support it will pass the O_LARGEFILE flag
> to open(2) and everything will go ok.
> 
> bzzrt:/tmp% ll biglog
> -rw-r-----    1 wsl      wsl          8.0T Mar 24 23:37 biglog
> bzzrt:/tmp% perl -e 'open(FH, "biglog"); $a = <FH>; print $a;'
> Mar 24 23:32:16 bzzrt syslogd 1.4.1#10: restart.
> 
> This is on a Debian GNU/Linux ia32 system (running the development version
> of Debian).
> 
> --
> Wessel Dankers <wsl at logreport.org>
> 
>   ------------------------------------------------------------------------
>    Part 1.2Type: application/pgp-signature

-- 
To UNSUBSCRIBE, email to development-request at logreport.org with a subject of
"unsubscribe". Trouble? Send an email with subject "help" to
development-request at logreport.org