Will vim copy all file content to memory when i open a file?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Will vim copy all file content to memory when i open a file?

Zhigang Song
Will vim copy all file content to memory when I open a file?

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Shlomi Fish-2
On Sun, 11 Feb 2018 18:09:02 +0800
Zhigang Song <[hidden email]> wrote:

> Will vim copy all file content to memory when I open a file?
>

opening this file in vim:

-rw-r--r-- 1 shlomif shlomif 608170649 Feb 12 12:19 Total-Imp3.txt
shlomif[3fc]:~/Download/unpack/games/freecell/freecell-pro-3fc-deals--split$ wc Total-Imp3.txt
 55946300  55946300 608170649 Total-Imp3.txt
shlomif[3fc]:~/Download/unpack/games/freecell/freecell-pro-3fc-deals--split$
────────────────────────────────────────────────────────────────────────────────

takes 10% of my 8 GB of RAM, so I guess vim loads the file into RAM.

Regards,

        Shlomi

--
-----------------------------------------------------------------
Shlomi Fish       http://www.shlomifish.org/
https://is.gd/MQHVF3 - The Atom Text Editor edits a 2,000,001B file

There is no IGLU Cabal. The problem of founding an IGLU Cabal has been proven,
in a surprise move, to be equivalent to the question of the existence of God,
fully‐tolerant religions and NP‐complete oracles.   — Omer Zak

Please reply to list if it's a mailing list post - http://shlom.in/reply .

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Bram Moolenaar
In reply to this post by Zhigang Song

Zhigang Song wrote:

> Will vim copy all file content to memory when I open a file?

Vim will read the file, mainly to count the number of lines, so you can
do "1234G".

Not all of the file is kept in memory, especially if it's a big file.
Check out the 'maxmem' and 'maxmemtot' options.

--
I'm sure that I asked CBuilder to do a "full" install.  Looks like I got
a "fool" install, instead. Charles E Campbell, Jr, PhD


 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\  an exciting new programming language -- http://www.Zimbu.org        ///
 \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Matt Ackeret
In reply to this post by Zhigang Song
On Sun, 11 Feb 2018, Zhigang Song wrote:
>Will vim copy all file content to memory when I open a file?

No, that's why vim is fast opening gigantic files.

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Tony Mechelynck
On Tue, Feb 13, 2018 at 2:41 AM, Matt Ackeret <[hidden email]> wrote:
> On Sun, 11 Feb 2018, Zhigang Song wrote:
>>Will vim copy all file content to memory when I open a file?
>
> No, that's why vim is fast opening gigantic files.

However, unlike sed, Vim is not a stream editor (i.e. an editor which
processes the file in sequence from beginning to end, writing the
output at the same time). Vim edits its file(s) in whatever sequence
the user moves the cursor, writes it only for :w :x or similar, and
for that it "prefers" to hold the whole file in memory, and it will do
so as long as each buffer is small in comparison to 'maxmem' and all
buffers combined in comparison to 'maxmemtot'. On my 64-bit system,
Vim sets both these options by default to 4021798 (a little under 4
GiB which would be 4194304 KiB, and exactly half the value on the
MemTotal line of /proc/meminfo). It doesn't use all that in practice
though: my big memory-gobbler is the browser, which I have to restart
every few days to reclaim some RAM.

Best regards,
Tony.

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Lifepillar
On 13/02/2018 03:15, Tony Mechelynck wrote:

> On Tue, Feb 13, 2018 at 2:41 AM, Matt Ackeret <[hidden email]> wrote:
>> On Sun, 11 Feb 2018, Zhigang Song wrote:
>>> Will vim copy all file content to memory when I open a file?
>>
>> No, that's why vim is fast opening gigantic files.
>
> However, unlike sed, Vim is not a stream editor (i.e. an editor which
> processes the file in sequence from beginning to end, writing the
> output at the same time). Vim edits its file(s) in whatever sequence
> the user moves the cursor, writes it only for :w :x or similar, and
> for that it "prefers" to hold the whole file in memory, and it will do
> so as long as each buffer is small in comparison to 'maxmem' and all
> buffers combined in comparison to 'maxmemtot'. On my 64-bit system,
> Vim sets both these options by default to 4021798 (a little under 4
> GiB which would be 4194304 KiB, and exactly half the value on the
> MemTotal line of /proc/meminfo).
In my system (macOS and Vim 8.0.1500), if I run `vim --clean`
repeatedly, both maxmem and maxmemtot are set to the same huge value,
which changes at every run, such as 9007199253926612 or
9007199253919906. I wish that were approx. half the KB of my RAM
but alas it is not. Are those normal values to see?

Life.



--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Lifepillar
In reply to this post by Bram Moolenaar
On 12/02/2018 20:58, Bram Moolenaar wrote:

>
> Zhigang Song wrote:
>
>> Will vim copy all file content to memory when I open a file?
>
> Vim will read the file, mainly to count the number of lines, so you can
> do "1234G".
>
> Not all of the file is kept in memory, especially if it's a big file.
> Check out the 'maxmem' and 'maxmemtot' options.

With noswapfile, though, those parameters are ignored, aren't they? The
documentation doesn't mention that.

Life.


--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Tony Mechelynck
In reply to this post by Lifepillar
On Tue, Feb 13, 2018 at 9:49 PM, Lifepillar <[hidden email]> wrote:
[...]
> In my system (macOS and Vim 8.0.1500), if I run `vim --clean`
> repeatedly, both maxmem and maxmemtot are set to the same huge value, which
> changes at every run, such as 9007199253926612 or
> 9007199253919906. I wish that were approx. half the KB of my RAM
> but alas it is not. Are those normal values to see?
>
> Life.

I don't think so. These settings are supposed to be expressed in
kilobytes, so the values you give above would be on the close order of
9 petabytes, which seems a little too big compared to the memory size
usually available on today's computers.

Try adding the following to your vimrc:

    " make sure the memory sizes are set to a "reasonable" value
    " but allow an explicit setting up to 2½ times our default
    if &maxmem > 5000000
        set maxmem=2000000
    endif
    if &maxmemtot > 5000000
        set maxmem=2000000
    endif

Best regards,
Tony.

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Tony Mechelynck
On Thu, Feb 15, 2018 at 12:28 AM, Tony Mechelynck
<[hidden email]> wrote:

> On Tue, Feb 13, 2018 at 9:49 PM, Lifepillar <[hidden email]> wrote:
> [...]
>> In my system (macOS and Vim 8.0.1500), if I run `vim --clean`
>> repeatedly, both maxmem and maxmemtot are set to the same huge value, which
>> changes at every run, such as 9007199253926612 or
>> 9007199253919906. I wish that were approx. half the KB of my RAM
>> but alas it is not. Are those normal values to see?
>>
>> Life.
>
> I don't think so. These settings are supposed to be expressed in
> kilobytes, so the values you give above would be on the close order of
> 9 petabytes, which seems a little too big compared to the memory size
> usually available on today's computers.
>
> Try adding the following to your vimrc:
>
>     " make sure the memory sizes are set to a "reasonable" value
>     " but allow an explicit setting up to 2½ times our default
>     if &maxmem > 5000000
>         set maxmem=2000000
>     endif
>     if &maxmemtot > 5000000
>         set maxmem=2000000

Oops! This should be "set maxmemtot=2000000"

>     endif
>
> Best regards,
> Tony.

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Christian Brabandt
In reply to this post by Tony Mechelynck

On Do, 15 Feb 2018, Tony Mechelynck wrote:

> On Tue, Feb 13, 2018 at 9:49 PM, Lifepillar <[hidden email]> wrote:
> [...]
> > In my system (macOS and Vim 8.0.1500), if I run `vim --clean`
> > repeatedly, both maxmem and maxmemtot are set to the same huge value, which
> > changes at every run, such as 9007199253926612 or
> > 9007199253919906. I wish that were approx. half the KB of my RAM
> > but alas it is not. Are those normal values to see?
> >
> > Life.
>
> I don't think so. These settings are supposed to be expressed in
> kilobytes, so the values you give above would be on the close order of
> 9 petabytes, which seems a little too big compared to the memory size
> usually available on today's computers.

Then perhaps we have a bug on how Vim initially determines the maxmem
and maxmemtot option.

Best,
Christian
--
Melkt die Bäuerin die Kühe, hat der Bauer keine Mühe.

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Will vim copy all file content to memory when i open a file?

Lifepillar
In reply to this post by Tony Mechelynck
On 15/02/2018 00:31, Tony Mechelynck wrote:

> On Thu, Feb 15, 2018 at 12:28 AM, Tony Mechelynck
> <[hidden email]> wrote:
>> On Tue, Feb 13, 2018 at 9:49 PM, Lifepillar <[hidden email]> wrote:
>> [...]
>>> In my system (macOS and Vim 8.0.1500), if I run `vim --clean`
>>> repeatedly, both maxmem and maxmemtot are set to the same huge value, which
>>> changes at every run, such as 9007199253926612 or
>>> 9007199253919906. I wish that were approx. half the KB of my RAM
>>> but alas it is not. Are those normal values to see?
>>>
>>> Life.
>>
>> I don't think so. These settings are supposed to be expressed in
>> kilobytes, so the values you give above would be on the close order of
>> 9 petabytes, which seems a little too big compared to the memory size
>> usually available on today's computers.
>>
>> Try adding the following to your vimrc:
>>
>>      " make sure the memory sizes are set to a "reasonable" value
>>      " but allow an explicit setting up to 2½ times our default
>>      if &maxmem > 5000000
>>          set maxmem=2000000
>>      endif
>>      if &maxmemtot > 5000000
>>          set maxmem=2000000
>
> Oops! This should be "set maxmemtot=2000000"

Yes, if I set maxmem and maxmemtot in my vimrc, those values stick and,
as far as I can tell, the limits are respected when I open some large
buffers. So, as already pointed out, this may be a bug in the way Vim
detects the amount of memory in macOS.

I have reported the issue: https://github.com/vim/vim/issues/2646.

Life.

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.