opening huge files, a little slow

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

opening huge files, a little slow

misi e

Hello,

would like to use vim to edit large files (600mb). Unfortunatelly, it
looks like opening such large files even on a Pentium 3.4Ghz with PATA
disk takes quite some time (ubuntu, kernel 2.6.22-14). I know that vim
creates some internal tree representation of the file, but I wonder if
something could be tuned, so that such loading times decrease, even if
memory usage would increase.

I tried to open a 767.187.800 bytes file (I created this file by cat-
ing all vim src files 100 times ) :

time vim  -u ./.vimrc --noplugin  -c":q" ../all3.c

the result is:

real    0m17.670s
user    0m15.457s
sys     0m1.428s

the .vimrc contains only

set noswapfile

I profiled vim, and opening the same file I got the file below....

So the question would be, is it possible to tune sthg. to open faster
these files??

Regards
Misi

below the gprof head

XXXXX$ head -40 no_swap

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls   s/call   s/call  name
 46.12      1.96     1.96        1     1.96     4.22  readfile
 23.53      2.96     1.00 31025600     0.00     0.00  ml_append_int
 12.94      3.51     0.55 31025601     0.00     0.00  ml_updatechunk
  8.24      3.86     0.35 31361959     0.00     0.00  ml_find_line
  4.94      4.07     0.21 31025600     0.00     0.00  ml_append
  0.47      4.09     0.02  1876672     0.00     0.00  mf_put
  0.47      4.11     0.02  1655954     0.00     0.00  mf_find_hash
  0.47      4.13     0.02  1655954     0.00     0.00  mf_get
  0.47      4.15     0.02        1     0.02     0.02  ml_delete
  0.47      4.17     0.02                             ml_setmarked
  0.24      4.18     0.01  1876675     0.00     0.00  mf_ins_hash
  0.24      4.19     0.01  1655955     0.00     0.00  mf_rem_hash
  0.24      4.20     0.01  1655954     0.00     0.00  mf_rem_used
  0.24      4.21     0.01   698266     0.00     0.00  ml_add_stack
  0.24      4.22     0.01   441920     0.00     0.00  alloc
  0.24      4.23     0.01   258689     0.00     0.00  mf_trans_del
  0.24      4.24     0.01   258600     0.00     0.00  ml_lineadd
  0.24      4.25     0.01                             vim_chdirfile
  0.00      4.25     0.00  1876674     0.00     0.00  mf_ins_used
  0.00      4.25     0.00   453656     0.00     0.00  lalloc
  0.00      4.25     0.00   453591     0.00     0.00  vim_free
  0.00      4.25     0.00   220720     0.00     0.00  mf_alloc_bhdr
  0.00      4.25     0.00   220720     0.00     0.00  mf_free_bhdr
  0.00      4.25     0.00   220720     0.00     0.00  mf_new
  0.00      4.25     0.00   220720     0.00     0.00  mf_release
  0.00      4.25     0.00   219851     0.00     0.00  ml_new_data
  0.00      4.25     0.00    11709     0.00     0.00  RealWaitForChar
  0.00      4.25     0.00    11709     0.00     0.00  mch_breakcheck



--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

RE: opening huge files, a little slow

JohnBeckett

misi wrote:
> would like to use vim to edit large files (600mb).

The best way to tune Vim for large files is Dr Chip's script:
http://www.vim.org/scripts/script.php?script_id=1506

Due to a limitation in how Vim uses 32-bit integers (on a 32-bit system), files over
2GB may not be detected as large.

There are two tips on this (which need to be merged and cleaned), but they are made
obsolete by the above:
http://vim.wikia.com/wiki/VimTip343
http://vim.wikia.com/wiki/VimTip611

John


--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

iler.ml
In reply to this post by misi e
On Sun, Jun 15, 2008 at 3:36 PM, misi <[hidden email]> wrote:

Hello,

would like to use vim to edit large files (600mb). Unfortunatelly, it
looks like opening such large files even on a Pentium 3.4Ghz with PATA
disk takes quite some time (ubuntu, kernel 2.6.22-14). I know that vim
creates some internal tree representation of the file, but I wonder if
something could be tuned, so that such loading times decrease, even if
memory usage would increase.

I tried to open a 767.187.800 bytes file (I created this file by cat-
ing all vim src files 100 times ) :

time vim  -u ./.vimrc --noplugin  -c":q" ../all3.c

the result is:

real    0m17.670s
user    0m15.457s
sys     0m1.428s

Hmm you are lucky. Same command takes me 43 sec
on 700mb file, E2180 processor with 2GB RAM,
ubuntu feisty, kernel 2.6.26-rc5, and ul=-1 added to ./vimrc.



the .vimrc contains only

set noswapfile

I profiled vim, and opening the same file I got the file below....

So the question would be, is it possible to tune sthg. to open faster
these files??

Regards
Misi

below the gprof head

XXXXX$ head -40 no_swap

Flat profile:

Each sample counts as 0.01 seconds.
 %   cumulative   self              self     total
 time   seconds   seconds    calls   s/call   s/call  name
 46.12      1.96     1.96        1     1.96     4.22  readfile
 23.53      2.96     1.00 31025600     0.00     0.00  ml_append_int
 12.94      3.51     0.55 31025601     0.00     0.00  ml_updatechunk
 8.24      3.86     0.35 31361959     0.00     0.00  ml_find_line
 4.94      4.07     0.21 31025600     0.00     0.00  ml_append
 0.47      4.09     0.02  1876672     0.00     0.00  mf_put
 0.47      4.11     0.02  1655954     0.00     0.00  mf_find_hash
 0.47      4.13     0.02  1655954     0.00     0.00  mf_get
 0.47      4.15     0.02        1     0.02     0.02  ml_delete
 0.47      4.17     0.02                             ml_setmarked
 0.24      4.18     0.01  1876675     0.00     0.00  mf_ins_hash
 0.24      4.19     0.01  1655955     0.00     0.00  mf_rem_hash
 0.24      4.20     0.01  1655954     0.00     0.00  mf_rem_used
 0.24      4.21     0.01   698266     0.00     0.00  ml_add_stack
 0.24      4.22     0.01   441920     0.00     0.00  alloc
 0.24      4.23     0.01   258689     0.00     0.00  mf_trans_del
 0.24      4.24     0.01   258600     0.00     0.00  ml_lineadd
 0.24      4.25     0.01                             vim_chdirfile
 0.00      4.25     0.00  1876674     0.00     0.00  mf_ins_used
 0.00      4.25     0.00   453656     0.00     0.00  lalloc
 0.00      4.25     0.00   453591     0.00     0.00  vim_free
 0.00      4.25     0.00   220720     0.00     0.00  mf_alloc_bhdr
 0.00      4.25     0.00   220720     0.00     0.00  mf_free_bhdr
 0.00      4.25     0.00   220720     0.00     0.00  mf_new
 0.00      4.25     0.00   220720     0.00     0.00  mf_release
 0.00      4.25     0.00   219851     0.00     0.00  ml_new_data
 0.00      4.25     0.00    11709     0.00     0.00  RealWaitForChar
 0.00      4.25     0.00    11709     0.00     0.00  mch_breakcheck

 

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

Ben Schmidt
In reply to this post by misi e

misi wrote:
> Hello,
>
> would like to use vim to edit large files (600mb). Unfortunatelly, it
> looks like opening such large files even on a Pentium 3.4Ghz with PATA
> disk takes quite some time (ubuntu, kernel 2.6.22-14). I know that vim
> creates some internal tree representation of the file, but I wonder if
> something could be tuned, so that such loading times decrease, even if
> memory usage would increase?
[...]
> time vim  -u ./.vimrc --noplugin  -c":q" ../all3.c
>
> the result is:
>
> real    0m17.670s
> user    0m15.457s
> sys     0m1.428s

I think what is taking the time is actually reading the file from disk.
Vim reads the entire file into memory, and getting 600 MB off disk takes
time.

[...]
> I profiled vim, and opening the same file I got the file below....
[...]

> Flat profile:
>
> Each sample counts as 0.01 seconds.
>   %   cumulative   self              self     total
>  time   seconds   seconds    calls   s/call   s/call  name
>  46.12      1.96     1.96        1     1.96     4.22  readfile
>  23.53      2.96     1.00 31025600     0.00     0.00  ml_append_int
>  12.94      3.51     0.55 31025601     0.00     0.00  ml_updatechunk
>   8.24      3.86     0.35 31361959     0.00     0.00  ml_find_line
>   4.94      4.07     0.21 31025600     0.00     0.00  ml_append
[...]
>   0.00      4.25     0.00    11709     0.00     0.00  mch_breakcheck

The profile only shows an execution time of four and a quarter seconds.
Is this really how long it took? If so, it's probably because most of
the file was cached in memory by the OS from your previous reading of
it. I don't think that kind of time is unreasonable by any means, and
although you could probably scrape a bit off it, I doubt it would be
worth it, as it would be likely to impair code readability somewhat,
and/or cause errors, and these are worse than slowness! If it didn't
take that long, I wonder what doesn't show up in the profile. Perhaps
the overhead of the function calls themselves. If that's the case,
optimisation would again be possible, but harder than optimising the
time that *does* show up, so again probably not worth it or unwise for
the same reasons as above.

I may look into it a little more, particularly if you send back some
more details, but I doubt much can be done without a pretty big
overhaul--i.e. making Vim read in a background thread or lazily or such.

Ben.




--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

misi e
In reply to this post by JohnBeckett

Hello..

I know the trick with the swapfile, I created a small .vimrc that sets
noswapfile. This was used in my example...



On Jun 16, 3:33 am, "John Beckett" <[hidden email]> wrote:

> misi wrote:
> > would like to use vim to edit large files (600mb).
>
> The best way to tune Vim for large files is Dr Chip's script:http://www.vim.org/scripts/script.php?script_id=1506
>
> Due to a limitation in how Vim uses 32-bit integers (on a 32-bit system), files over
> 2GB may not be detected as large.
>
> There are two tips on this (which need to be merged and cleaned), but they are made
> obsolete by the above:http://vim.wikia.com/wiki/VimTip343http://vim.wikia.com/wiki/VimTip611
>
> John
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

misi e
In reply to this post by Ben Schmidt

I like the idea of reading in the background... As I know there is no
threading so far in vim, isn,t it? But few things probably could be
done in threads like syntax parsing for highlighting. Sometimes this
is also an issue, for eg. when opening a huge xml file etc.


yes....  thanks for the "clear eyes"... I was hurrying to post the
results, as I spent quite some time with measurement. The cumulative
seconds are indeed 4.25 seconds... I did not check those, was looking
in the time spent in %.   However now I tested the following:


src$ VIM=/usr/share/vim VIMRUNTIME=/usr/share/vim/vim71/ time  ./vim -
u ./.vimrc --noplugins  -c":q" ../all3.c

(I mean, I measured with time..)

the time shown is


10.18user 1.52system 0:13.20elapsed

so 13 seconds.... no idea why is so big difference between the time
measured by time and gprof... What is important is that at the end the
time spent to open the 600MB file and quit vim is 13 seconds..

I would like to tell you however, that this is not to blame vim... vim
is the best editor!!





On Jun 16, 7:52 am, Ben Schmidt <[hidden email]> wrote:

> misi wrote:
> > Hello,
>
> > would like to use vim to edit large files (600mb). Unfortunatelly, it
> > looks like opening such large files even on a Pentium 3.4Ghz with PATA
> > disk takes quite some time (ubuntu, kernel 2.6.22-14). I know that vim
> > creates some internal tree representation of the file, but I wonder if
> > something could be tuned, so that such loading times decrease, even if
> > memory usage would increase?
> [...]
> > time vim  -u ./.vimrc --noplugin  -c":q" ../all3.c
>
> > the result is:
>
> > real    0m17.670s
> > user    0m15.457s
> > sys     0m1.428s
>
> I think what is taking the time is actually reading the file from disk.
> Vim reads the entire file into memory, and getting 600 MB off disk takes
> time.
>
> [...]
>
> > I profiled vim, and opening the same file I got the file below....
> [...]
> > Flat profile:
>
> > Each sample counts as 0.01 seconds.
> >   %   cumulative   self              self     total
> >  time   seconds   seconds    calls   s/call   s/call  name
> >  46.12      1.96     1.96        1     1.96     4.22  readfile
> >  23.53      2.96     1.00 31025600     0.00     0.00  ml_append_int
> >  12.94      3.51     0.55 31025601     0.00     0.00  ml_updatechunk
> >   8.24      3.86     0.35 31361959     0.00     0.00  ml_find_line
> >   4.94      4.07     0.21 31025600     0.00     0.00  ml_append
> [...]
> >   0.00      4.25     0.00    11709     0.00     0.00  mch_breakcheck
>
> The profile only shows an execution time of four and a quarter seconds.
> Is this really how long it took? If so, it's probably because most of
> the file was cached in memory by the OS from your previous reading of
> it. I don't think that kind of time is unreasonable by any means, and
> although you could probably scrape a bit off it, I doubt it would be
> worth it, as it would be likely to impair code readability somewhat,
> and/or cause errors, and these are worse than slowness! If it didn't
> take that long, I wonder what doesn't show up in the profile. Perhaps
> the overhead of the function calls themselves. If that's the case,
> optimisation would again be possible, but harder than optimising the
> time that *does* show up, so again probably not worth it or unwise for
> the same reasons as above.
>
> I may look into it a little more, particularly if you send back some
> more details, but I doubt much can be done without a pretty big
> overhaul--i.e. making Vim read in a background thread or lazily or such.
>
> Ben.
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

misi e
In reply to this post by Ben Schmidt

hello,

you almost convinced me with the disk time...

but that is not true. Reading the same file with a command like grep,
takes 567msecs.
try

time grep XXXXXXXXXXX  ./all3.c #this will not find anything, but will
process all the file.. and no mmap is used!!!!!
the result is:

real    0m0.466s
user    0m0.232s
sys     0m0.236s

so grep spends some more time in userspace as well but the kernel time
is the same as of my small test program I wrote to read the same file
into memory..


The result is

real    0m0.253s
user    0m0.000s
sys     0m0.252s

This means that reading 600mb of memory takes 0.252 secs in kernel
(that is the concrete IO...).

So reading the 600mb file takes 252 time, and processing 13 seconds...
bellow is the little c program...


so.. again I would make positive critic. Just wondering if sthg. could
be tuned.




#include <unistd.h>
#include <fcntl.h>


int main(int argc, char ** argv)
{

   if(argc < 1)
   {
      printf("file name not provided..\n");
      return -1;
   }

   int fd = open(argv[1], O_RDONLY);

   if(fd < 0)
   {
      printf("cannot open file %s", argv[1]);
      return -1;
   }
   int chunk_size = 100000;
   if(argc > 2)
   {
      chunk_size = atoi(argv[2]);
   }

   char *buffer = (char * )malloc(chunk_size);
   int res;
   int count = 0;
   while( (res = read(fd, buffer, chunk_size) ) > 0)
   {
      count ++;
   }

   printf("total read %d*%d=%d", count, chunk_size, count*chunk_size);
//sorrry but deallocation in next version...
}







On Jun 16, 7:52 am, Ben Schmidt <[hidden email]> wrote:

> misi wrote:
> > Hello,
>
> > would like to use vim to edit large files (600mb). Unfortunatelly, it
> > looks like opening such large files even on a Pentium 3.4Ghz with PATA
> > disk takes quite some time (ubuntu, kernel 2.6.22-14). I know that vim
> > creates some internal tree representation of the file, but I wonder if
> > something could be tuned, so that such loading times decrease, even if
> > memory usage would increase?
> [...]
> > time vim  -u ./.vimrc --noplugin  -c":q" ../all3.c
>
> > the result is:
>
> > real    0m17.670s
> > user    0m15.457s
> > sys     0m1.428s
>
> I think what is taking the time is actually reading the file from disk.
> Vim reads the entire file into memory, and getting 600 MB off disk takes
> time.
>
> [...]
>
> > I profiled vim, and opening the same file I got the file below....
> [...]
> > Flat profile:
>
> > Each sample counts as 0.01 seconds.
> >   %   cumulative   self              self     total
> >  time   seconds   seconds    calls   s/call   s/call  name
> >  46.12      1.96     1.96        1     1.96     4.22  readfile
> >  23.53      2.96     1.00 31025600     0.00     0.00  ml_append_int
> >  12.94      3.51     0.55 31025601     0.00     0.00  ml_updatechunk
> >   8.24      3.86     0.35 31361959     0.00     0.00  ml_find_line
> >   4.94      4.07     0.21 31025600     0.00     0.00  ml_append
> [...]
> >   0.00      4.25     0.00    11709     0.00     0.00  mch_breakcheck
>
> The profile only shows an execution time of four and a quarter seconds.
> Is this really how long it took? If so, it's probably because most of
> the file was cached in memory by the OS from your previous reading of
> it. I don't think that kind of time is unreasonable by any means, and
> although you could probably scrape a bit off it, I doubt it would be
> worth it, as it would be likely to impair code readability somewhat,
> and/or cause errors, and these are worse than slowness! If it didn't
> take that long, I wonder what doesn't show up in the profile. Perhaps
> the overhead of the function calls themselves. If that's the case,
> optimisation would again be possible, but harder than optimising the
> time that *does* show up, so again probably not worth it or unwise for
> the same reasons as above.
>
> I may look into it a little more, particularly if you send back some
> more details, but I doubt much can be done without a pretty big
> overhaul--i.e. making Vim read in a background thread or lazily or such.
>
> Ben.
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

Nico Weber-3

Hi,

> but that is not true. Reading the same file with a command like grep,
> takes 567msecs.

is it faster if you do `set swapsync=` and `set nofsync` before  
reading the file? If so, ext3 is to blame: http://taint.org/2008/03/12/122601a.html

Nico


--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

Antonio Colombo
Hi misi,

> you almost convinced me with the disk time...

> but that is not true. Reading the same file with a command like grep,
> takes 567msecs.
> try

> time grep XXXXXXXXXXX ./all3.c #this will not find anything, but will
> process all the file.. and no mmap is used!!!!!
> the result is:

The problem with "the time it takes to read" in Unix is elusive.
It depends on the number of real physical reads Unix has to do.
If the data is already in the buffers, the performance is much better.
Example (with a 27MB file I had not been using in advance):

(aca) root /e/ita > time grep zxdfgghjjkk voci.txt

real 0m1.342s
user 0m0.028s
sys 0m0.048s
(aca) root /e/ita > time grep zxdfgghjjkk voci.txt

real 0m0.093s
user 0m0.040s
sys 0m0.024s
(aca) root /e/ita > time grep zxdfgghjjkk voci.txt

real 0m0.088s
user 0m0.028s
sys 0m0.036s
(aca) root /e/ita > time grep zxdfgghjjkk voci.txt

real 0m0.082s
user 0m0.032s
sys 0m0.028s
(aca) root /e/ita > time grep zxdfgghjjkk voci.txt

real 0m0.062s
user 0m0.016s
sys 0m0.028s
(aca) root /e/ita > time grep zxdfgghjjkk voci.txt

real 0m0.055s
user 0m0.024s
sys 0m0.020s
(aca) root /e/ita > time grep zxdfgghjjkk voci.txt

real 0m0.077s
user 0m0.024s
sys 0m0.036s
(aca) root /e/ita > time grep zxdfgghjjkk voci.txt

real 0m0.049s
user 0m0.020s
sys 0m0.024s
(aca) root /e/ita > ll voci.txt 
-rwxrwxr-x 1 root users 27338279 15 mag 2007 voci.txt

I don't know in which conditions you performed your tests,
but ideally you should reboot once per each test, and
perform the test without having used the file in advance,
in order to be sure that the Unix buffers are empty (of
the tested file).

Then a comparison can be safely used.

Cheers, Antonio
--
   /||\    | Antonio Colombo
  / || \   | [hidden email]
/  ()  \  |  [hidden email]
(___||___) |   [hidden email]

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

misi e

Ola!,

You are rigth with the buffering effect, however...

I made the following experiments on a HPUX machine this time. (Do not
have acces to the other one at the momont)

1. made a copy of the 600MB file, and indeed first reading takes
longer with vim:
first reading:
XXXX>time vim --noplugin -u ./.vimrc XXXXXXX/tmp/cucu.bebe -c":q"
39.07s real    26.18s user    10.72s system
second reading:
XXXX>time vim --noplugin -u ./.vimrc XXXXXXX/tmp/cucu.bebe -c":q"
33.13s real    25.27s user     7.29s system

2. made again another copy, and this time reading with my smal
program:
first reading:
time ./load  XXXXX/tmp/cucu.bebe 10000
total read 67706*10000=677060000    1.28s real     0.03s user
1.23s system

second reading
XXXX>time ./load  XXXX/tmp/cucu.bebe 10000
total read 67706*10000=677060000    1.25s real     0.04s user
1.21s system


with grep
first:
XXXX>time gr XXXXXXXXXXXXXXXXXXX   XXXXX/tmp/cucu.bebe
    1.44s real     0.28s user     1.10s system
second:
   1.41s real     0.29s user     1.09s system

and again vim:

XXXX>time vim --noplugin -u ./.vimrc XXXXXXX/tmp/cucu.bebe -c":q"
   41.77s real    26.25s user    10.81s system


So, in conclusion, the buffer effect is there, but is not significant.
Vim spends for some reason 10 times more in kernel for simple reading.
And a lot in userspace. I understand that it takes time to build the
data structure for the internal representation of the text, but I have
the feeling that there is space for optimization. I would be happy to
help, if the owner of the project gives some signes...

VIM IS THE BEST EDITOR

Misi











On Jun 17, 9:32 am, "Antonio Colombo" <[hidden email]> wrote:

> Hi misi,
>
> > you almost convinced me with the disk time...
>
> > but that is not true. Reading the same file with a command like grep,
> > takes 567msecs.
> > try
>
> > time grep XXXXXXXXXXX ./all3.c #this will not find anything, but will
> > process all the file.. and no mmap is used!!!!!
> > the result is:
>
> The problem with "the time it takes to read" in Unix is elusive.
> It depends on the number of real physical reads Unix has to do.
> If the data is already in the buffers, the performance is much better.
> Example (with a 27MB file I had not been using in advance):
>
> (aca) root /e/ita > time grep zxdfgghjjkk voci.txt
>
> real 0m1.342s
> user 0m0.028s
> sys 0m0.048s
> (aca) root /e/ita > time grep zxdfgghjjkk voci.txt
>
> real 0m0.093s
> user 0m0.040s
> sys 0m0.024s
> (aca) root /e/ita > time grep zxdfgghjjkk voci.txt
>
> real 0m0.088s
> user 0m0.028s
> sys 0m0.036s
> (aca) root /e/ita > time grep zxdfgghjjkk voci.txt
>
> real 0m0.082s
> user 0m0.032s
> sys 0m0.028s
> (aca) root /e/ita > time grep zxdfgghjjkk voci.txt
>
> real 0m0.062s
> user 0m0.016s
> sys 0m0.028s
> (aca) root /e/ita > time grep zxdfgghjjkk voci.txt
>
> real 0m0.055s
> user 0m0.024s
> sys 0m0.020s
> (aca) root /e/ita > time grep zxdfgghjjkk voci.txt
>
> real 0m0.077s
> user 0m0.024s
> sys 0m0.036s
> (aca) root /e/ita > time grep zxdfgghjjkk voci.txt
>
> real 0m0.049s
> user 0m0.020s
> sys 0m0.024s
> (aca) root /e/ita > ll voci.txt
> -rwxrwxr-x 1 root users 27338279 15 mag 2007 voci.txt
>
> I don't know in which conditions you performed your tests,
> but ideally you should reboot once per each test, and
> perform the test without having used the file in advance,
> in order to be sure that the Unix buffers are empty (of
> the tested file).
>
> Then a comparison can be safely used.
>
> Cheers, Antonio
> --
>    /||\    | Antonio Colombo
>   / || \   | [hidden email]
> /  ()  \  |  [hidden email]
> (___||___) |   [hidden email]
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

Antonio Colombo

Hi Misi,

your tests still are not in the "ideal" conditions, unless there

are reboots after the creation of each new test file. The creation

of the file of course leaves a lot of buffers in memory.

In my tests I used existing files I had not been using at all

from boot time.

.

Apart from that, your program and "grep" do not really need

to have in their private memory (not in the Unix buffer) more

than one "little piece" of the file. Of course vim needs and uses

its private memory to hold the whole file. To make things even,

you should modify your program in order to read each new

chunk of the file in a different piece of memory, thus having

them all in memory at the same time at the end of the run.

I expect the difference to be noticeable (unless the memory

of the HP-UX server is gigantic).

Cheers, Antonio

 PS Off Topic: I like more "ciao" as a throw away file name.  ;-)

--

   /||\    | Antonio Colombo
  / || \   | [hidden email]
/  ()  \  |  [hidden email]
(___||___) |   [hidden email]

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: opening huge files, a little slow

misi e

I will make the measurement on the initial machine, will work with
different files and post the results.

The code I used is little different, the while loop is allocating new
memory for each chunk:


   while( (res = read(fd, buffer, chunk_size) ) > 0)
   {
      buffer = (char *)malloc(chunk_size);
      count ++;
   }

I removed the line, but memory allocation is not influencing that
much... if I work with rel. huge chunks> 5000


Misi




On Jun 17, 12:47 pm, "Antonio Colombo" <[hidden email]> wrote:

> Hi Misi,
>
> your tests still are not in the "ideal" conditions, unless there
>
> are reboots after the creation of each new test file. The creation
>
> of the file of course leaves a lot of buffers in memory.
>
> In my tests I used existing files I had not been using at all
>
> from boot time.
>
> .
>
> Apart from that, your program and "grep" do not really need
>
> to have in their private memory (not in the Unix buffer) more
>
> than one "little piece" of the file. Of course vim needs and uses
>
> its private memory to hold the whole file. To make things even,
>
> you should modify your program in order to read each new
>
> chunk of the file in a different piece of memory, thus having
>
> them all in memory at the same time at the end of the run.
>
> I expect the difference to be noticeable (unless the memory
>
> of the HP-UX server is gigantic).
>
> Cheers, Antonio
>
>  PS Off Topic: I like more "ciao" as a throw away file name.  ;-)
>
> --
>    /||\    | Antonio Colombo
>   / || \   | [hidden email]
> /  ()  \  |  [hidden email]
> (___||___) |   [hidden email]
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---