Memory is used up when I replace "\t0\n" to "\n"

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory is used up when I replace "\t0\n" to "\n"

Aleafs

Dear all,

A bug seems appeared when I tried to delete the last column of a text
file (700K bytes, 20K lines ).

Memory is used up and the computer need to be reboot  when I oprate as
below:

:1,$  s/\t0\n/\n/g

In the other hand, command like this works normal:

:1,$ s/\t0$//g

The version of my vim is 7.2. It can be reappeared on both Linux and
Windows.

Best regards

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

RE: Memory is used up when I replace "\t0\n" to "\n"

JohnBeckett

Aleafs wrote:
> :1,$  s/\t0\n/\n/g

In a substitute, \n means two different things:
- In the pattern, it refers to a newline.
- In the replacement, it refers to a null byte (8 zero bits).

You can see this at ':help :s' by following the first two links.

In a replacement, '\r' inserts a newline. Yes, it's strange, and
no, I didn't try to work out why this issue caused the problem
you report.

John


--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Memory is used up when I replace "\t0\n" to "\n"

Aleafs

Thanks a lot. Expecting your conclusion


On 9月10日, 下午4时13分, "John Beckett" <[hidden email]> wrote:

> Aleafs wrote:
> > :1,$  s/\t0\n/\n/g
>
> In a substitute, \n means two different things:
> - In the pattern, it refers to a newline.
> - In the replacement, it refers to a null byte (8 zero bits).
>
> You can see this at ':help :s' by following the first two links.
>
> In a replacement, '\r' inserts a newline. Yes, it's strange, and
> no, I didn't try to work out why this issue caused the problem
> you report.
>
> John
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Memory is used up when I replace "\t0\n" to "\n"

Matt Wozniski-2
In reply to this post by JohnBeckett

On Thu, Sep 10, 2009 at 4:13 AM, John Beckett wrote:
>
> Aleafs wrote:
>> :1,$  s/\t0\n/\n/g

Note that :% is a shortcut for :1,$

> In a substitute, \n means two different things:
> - In the pattern, it refers to a newline.
> - In the replacement, it refers to a null byte (8 zero bits).
>
> You can see this at ':help :s' by following the first two links.
>
> In a replacement, '\r' inserts a newline. Yes, it's strange, and
> no, I didn't try to work out why this issue caused the problem
> you report.

My guess: Vim is really, really bad at handling extremely long lines,
and if the goal was to delete "the last column of a text file" (I'm
assuming every line ended with "\t0") then this command told vim to
turn the file into one 700kb long line.

~Matt

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Memory is used up when I replace "\t0\n" to "\n"

iler.ml
On Fri, Sep 11, 2009 at 06:57, Matt Wozniski <[hidden email]> wrote:

On Thu, Sep 10, 2009 at 4:13 AM, John Beckett wrote:
>
> Aleafs wrote:
>> :1,$  s/\t0\n/\n/g

Note that :% is a shortcut for :1,$

> In a substitute, \n means two different things:
> - In the pattern, it refers to a newline.
> - In the replacement, it refers to a null byte (8 zero bits).
>
> You can see this at ':help :s' by following the first two links.
>
> In a replacement, '\r' inserts a newline. Yes, it's strange, and
> no, I didn't try to work out why this issue caused the problem
> you report.

My guess: Vim is really, really bad at handling extremely long lines,
and if the goal was to delete "the last column of a text file" (I'm
assuming every line ended with "\t0") then this command told vim to
turn the file into one 700kb long line.

1. I don't understand why vim grows here (800MB and up)
without giving memory overallocation error based on 'maxmem' and 'maxmemtot'. 
'maxmem' and 'maxmemtot' have values of of 150524 (that's KB, or 150MB). 
It's undo memory causes memory overuse here. 

2. This problem of memory use here (and speed)is is with 'noswapfile' and 'undolevels', and it's solved by:

    :set noswapfile undolevel=-1,

Then vim finishes the replacement (wait 1/2 minutes).
And vim opens 700k-long file consisting of single line and many 0-bytes, in the first place, without problem.

Testcase:

1) 700k file consisting of single line and many 0-bytes
perl -e 'printf "%8d %8d %8d %8d\n", $_,$_,$_,$_ for(1..20000)' >/tmp/1
tr '\n' '\000' </tmp/1 >/tmp/2
vim /tmp/2                 # no problem

2) vim repalces newlines to 0-bytes without growing in memory, 700k file 20k lines:
perl -e 'printf "%8d %8d %8d %8d\n", $_,$_,$_,$_ for(1..20000)' >/tmp/1
vim /tmp/1
:set noswapfile undolevel=-1
:%s/\n/\n/               " finished in 30 sec, resul tis single 700k-long line

3) without 'et noswapfile undolevel=-1', blows up vim to pieces:
perl -e 'printf "%8d %8d %8d %8d\n", $_,$_,$_,$_ for(1..20000)' >/tmp/1
vim /tmp/1
:%s/\n/\n/ " grows beyond 1G in size and killed by OS, if you have patience
Killed

In scenario #3, I thought vim would report error based on 'maxmem' and 'maxmemtot', but
it did not. My values of 'maxmem' and 'maxmemtot' are 150254 (=150MB), but why
vim grew over 800MB in vm size without giving memory size error ?

Yakov
 

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---