The problem of opening large files

[Wessel Dankers]

On 2002-03-24 22:01:55+0100, Arnaud Taddei wrote:
> 
> Recently I showed some customers on what kind of things they could get from an
> LDAP log using my own scripts (not yet in Lire) in order to make sure that my
> ideas are ok before we can introduce an LDAP superservice. 
> 
> I ran into the problem that my program had to read a 2GB log file for 3 days of
> LDAP log and it failed with 
> 
> 	Cannot open file: Value too large for defined data type at
> /homedir/a/admin/project/bin/testopen line 6, <IN> chunk 1.
> 
> The code was simply:
> 
> 	open(F, $file) || die "Cannot read $file: $!";
> 
> so I was stuck because I have no infrastructure to split the input file and I
> was squeezed by time as I had to provide results. After some investigation I
> discovered that the sysopen call requires to copy the data in the system which
> means that no way this approach will NEVER work in the general term. So how some
> programs like less, or cat can open a file that large? Because they use MMAP. As

Well open(2) certainly doesn't copy a file into memory.
The problem you're experiencing is what happens when perl doesn't have
large file support (perhaps because your libc doesn't support it). Once
perl is compiled with large file support it will pass the O_LARGEFILE flag
to open(2) and everything will go ok.

bzzrt:/tmp% ll biglog
-rw-r-----    1 wsl      wsl          8.0T Mar 24 23:37 biglog
bzzrt:/tmp% perl -e 'open(FH, "biglog"); $a = <FH>; print $a;'
Mar 24 23:32:16 bzzrt syslogd 1.4.1#10: restart.

This is on a Debian GNU/Linux ia32 system (running the development version
of Debian).

--
Wessel Dankers <wsl at logreport.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 230 bytes
Desc: not available
Url : http://lists.logreport.org/pipermail/development/attachments/20020324/e85e56f3/attachment.bin