Lost file using Vim (first time ever in 15 years)

[Ben Fritz]

On Tuesday, May 15, 2012 12:24:21 PM UTC-5, Chris Jones wrote:

> On Tue, May 15, 2012 at 11:23:57AM EDT, Ben Fritz wrote:
>
> > Since the error only occurs if 'backup' and 'writebackup' are both
> > off, maybe we could only do the "dry run" conversion in this
> > situation? I.e., if either 'backup' or 'writebackup' are set, do what
> > Vim has always done; but if neither are set, do the extra check first.
> > We could even put a nice notice in :help 'writebackup' saying "if this
> > option is off, writing the file will take longer due to extra safety
> > checks" to discourage turning it off in the first place.
>
> So you would penalize those of us who turned off this backup/writebackup
> bloat..?
>

From Christian's test data, it looked like the slow-down was on the order of a few seconds for a write of a big file. I think that would be a small price to pay to know that the entire content of your file isn't going to be lost because Vim decided it could not modify them. I don't consider it a penalty. I consider it giving users some of the security of 'writebackup' even if they don't have it turned on.

> Or would you force us to reinstate it so as not to be penalized..?
>
> Aha... you turned off backup and writebackup.. you wicked little
> so-and-so.. You seriously thought you were going get away with it..?
>

It's risky, but it's also a valid use scenario, so that's why I want to see the file contents protected against being cleared for no good reason, even if the option is turned off.

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Bram Moolenaar]

Ben Fritz wrote:

> On Monday, May 14, 2012 2:39:54 PM UTC-5, Gary Johnson wrote:
> >
> > As I understand the problem, they also had to have 'nobackup' and
> > 'nowritebackup' set, the latter which is not the default.  So the
> > user threw away the safety net, disregarded the warning sign, and
> > because of this I have to wait longer each time I write a file?
> >
> > Of course, if the slowdown is negligible, then there's probably no
> > reason not to go with Christian's change.
> >
>
> Since the error only occurs if 'backup' and 'writebackup' are both
> off, maybe we could only do the "dry run" conversion in this
> situation? I.e., if either 'backup' or 'writebackup' are set, do what
> Vim has always done; but if neither are set, do the extra check first.
> We could even put a nice notice in :help 'writebackup' saying "if this
> option is off, writing the file will take longer due to extra safety
> checks" to discourage turning it off in the first place.

The 'writebackup' option is to catch cases where writing fails.  It
means that Vim can restore the original file.  There is no point in
additional work.

If a user switches off 'writebackup' he must be prepared for losing
work.  Perhaps this is not sufficiently clear to the user?

Considering how very few times this problem occurs I don't think it is
justified to do any work on this.  Except perhaps a better explanation
in the docs in some place.

--
SOLDIER: Where did you get the coconuts?
ARTHUR:  Through ... We found them.
SOLDIER: Found them?  In Mercea.  The coconut's tropical!
                 "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\  an exciting new programming language -- http://www.Zimbu.org        ///
 \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Christian Brabandt]

Hi Ben!

On Di, 15 Mai 2012, Ben Fritz wrote:
> Since the error only occurs if 'backup' and 'writebackup' are both
> off, maybe we could only do the "dry run" conversion in this
> situation? I.e., if either 'backup' or 'writebackup' are set, do what
> Vim has always done; but if neither are set, do the extra check first.
> We could even put a nice notice in :help 'writebackup' saying "if this
> option is off, writing the file will take longer due to extra safety
> checks" to discourage turning it off in the first place.

I think this is a good compromise¹. First off all, a successful write
should have priority, so one needs to be very cautious, in case an
encoding from 'enc' to 'fenc' needs to be done.

That small write penalty should be ok, since it is only noticeable when
writing larger files (e.g. files with a size of several megabytes) and
although I usually use Vim with large csv files of several hundred
megabytes, I only seldom save files with that size, so I think editing
so large files is not done by the majority of vim users.

I have done some more statistics and in the standard case (when
writebackup is set), writing is as fast as usually, while only when
'writebackup' is turned off, write performance is impacted.


¹) updated patch at https://gist.github.com/2704019

regards,
Christian
--
Glück ist ein Stuhl, der plötzlich dasteht, wenn man sich zwischen
zwei andere setzen will.
                -- George Bernard Shaw

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Tim Chase]

On 05/15/12 13:46, Christian Brabandt wrote:
> That small write penalty should be ok, since it is only
> noticeable when writing larger files (e.g. files with a size of
> several megabytes) and although I usually use Vim with large csv
> files of several hundred megabytes,

I also use Vim on CSV files of 10-100 megs on occasion.  I'd also
like to make sure the tests are also performed on slow network
connections.  Occasionally those CSV files come from a SMB share
over wireless which is *ehem* less than performant.  I don't know
how netrw would tie into this testing as well.

-tim


--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Ben Fritz]

On Tuesday, May 15, 2012 1:33:39 PM UTC-5, Bram Moolenaar wrote:

> Ben Fritz wrote:
>
> > On Monday, May 14, 2012 2:39:54 PM UTC-5, Gary Johnson wrote:
> > >
> > > As I understand the problem, they also had to have 'nobackup' and
> > > 'nowritebackup' set, the latter which is not the default.  So the
> > > user threw away the safety net, disregarded the warning sign, and
> > > because of this I have to wait longer each time I write a file?
> > >
> > > Of course, if the slowdown is negligible, then there's probably no
> > > reason not to go with Christian's change.
> > >
> >
> > Since the error only occurs if 'backup' and 'writebackup' are both
> > off, maybe we could only do the "dry run" conversion in this
> > situation? I.e., if either 'backup' or 'writebackup' are set, do what
> > Vim has always done; but if neither are set, do the extra check first.
> > We could even put a nice notice in :help 'writebackup' saying "if this
> > option is off, writing the file will take longer due to extra safety
> > checks" to discourage turning it off in the first place.
>
> The 'writebackup' option is to catch cases where writing fails.  It
> means that Vim can restore the original file.  There is no point in
> additional work.
>
> If a user switches off 'writebackup' he must be prepared for losing
> work.  Perhaps this is not sufficiently clear to the user?
>
> Considering how very few times this problem occurs I don't think it is
> justified to do any work on this.  Except perhaps a better explanation
> in the docs in some place.
>

I don't think it's clear at all. The help for 'writebackup' gives no indication that this option is important to leave on to prevent losing not just your edits in progress, but the entire file you're editing. This is especially surprising when the write error has nothing to do with the ability to write to the filesystem, only with Vim's ability to convert characters between encodings. Additionally, it's not just a matter of 'writebackup'. The 'backupskip' option prevents 'writebackup' from taking effect if it matches. So if a user is editing a file in the 'backupskip' pattern, potentially in a directory covered by this pattern BY DEFAULT, like /tmp, they risk losing data.

I don't think that default Vim settings should ever let Vim delete an entire file without being asked to do so.

At least this thread prompted me to double-check and make sure 'writebackup' is safely set on my system :-)

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[John Beckett-2]

Bram Moolenaar wrote:
> If a user switches off 'writebackup' he must be prepared for
> losing work.  Perhaps this is not sufficiently clear to the
> user?
>
> Considering how very few times this problem occurs I don't
> think it is justified to do any work on this.  Except perhaps
> a better explanation in the docs in some place.

How about twweaking the error message?

Vim currently shows:
    "bad.tmp" E513: write error, conversion failed
        (make 'fenc' empty to override)
    WARNING: Original file may be lost or damaged don't
    quit the editor until the file is successfully written!

I suggest:
    "bad.tmp" E513: write error, conversion failed
    WARNING: The file on disk has probably been corrupted.
    Do not quit until the file is successfully written!
    Try saving it again after entering (nothing after "="):
    :set fenc=

Or, when writebackup is switched off, issue a clear warning?

John

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Gary Johnson-4]

On 2012-05-16, John Beckett wrote:

> Bram Moolenaar wrote:
> > If a user switches off 'writebackup' he must be prepared for
> > losing work.  Perhaps this is not sufficiently clear to the
> > user?
> >
> > Considering how very few times this problem occurs I don't
> > think it is justified to do any work on this.  Except perhaps
> > a better explanation in the docs in some place.
>
> How about twweaking the error message?
>
> Vim currently shows:
>     "bad.tmp" E513: write error, conversion failed
> (make 'fenc' empty to override)
>     WARNING: Original file may be lost or damaged don't
>     quit the editor until the file is successfully written!
>
> I suggest:
>     "bad.tmp" E513: write error, conversion failed
>     WARNING: The file on disk has probably been corrupted.
>     Do not quit until the file is successfully written!
>     Try saving it again after entering (nothing after "="):
>     :set fenc=

That is better.

In this scenario, the user has edited the contents of the file to
his liking and wants to save those contents to disk.  He has also
chosen the file-encoding that he wants.  He doesn't know that the
two are incompatible.  He may not even know that it's possible for
the two to be incompatible.

The error messages above tell the user how to safely save the file
contents.  Neither tell him what the problem is, where the problem
is, or how to get to the desired state of those contents saved with
the desired file-encoding.  That is, after a successful sequence of
":set fenc=", ":w", the user is still left without the desired
file-encoding and no clue as how to change to it.

It would be helpful to the user if the error message included a
statement like, "Character at line 77, column 14 cannot be encoded
with the specified 'fileencoding'."

It that is too much work to implement, then at least ":help E513"
should be more explicit about that particular error, the cause, and
possible solutions.  The error message might even suggest reading
":help E513" since that doesn't seem to occur to some users.

If I have time this evening and no one beats me to it, I'll try to
write up something.

> Or, when writebackup is switched off, issue a clear warning?

That would be really annoying, especially if the user knew what he
was doing and did it often.

Regards,
Gary

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Michael Henry-5]

On 05/15/2012 08:42 PM, Gary Johnson wrote:
> In this scenario, the user has edited the contents of the file to
> his liking and wants to save those contents to disk. He has also
> chosen the file-encoding that he wants. He doesn't know that the
> two are incompatible. He may not even know that it's possible for
> the two to be incompatible.

How does the following sound for damage control?  If Vim
encounters an encoding error during a write, it could
automatically re-try the write with utf-8 (or whatever fail-safe
encoding makes sense) to prevent leaving the user with an empty
file.  This shouldn't cost extra time for users with valid
settings.  Since the clobbered file was of zero length anyway,
it seems that nothing further would be lost by overwriting the
empty file with data in some default encoding.  Then the error
message could be changed to mean "your encoding failed; I've
replaced your file with an encoding that will work; change your
encoding and try again if you don't like this choice".  Then, a
user with invalid settings who issues :q! will not find his file
clobbered, and users with valid settings won't be penalized with
slower writes.

Michael Henry

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Marko Mahnič]

On May 16, 4:59 am, Michael Henry wrote:
>
> How does the following sound for damage control?  If Vim
> encounters an encoding error during a write, it could
> automatically re-try the write with utf-8 (or whatever fail-safe
> encoding makes sense) to prevent leaving the user with an empty
> file.

Wouldn't it be better if Vim would ask the user what to do when
encoding
with 'fenc' fails?  Some of the options could be:
   - save with 'enc' encoding
   - save with 'fenc' encoding and replace invalid characters with '?'
   - don't save anything, I'll fix it

This options should be give no matter what the value of 'writebackup'
is.
Until the user selects an option, nothing should be written to disk.
Only
then the user can be blamed if he looses data.

On May 15, 1:33pm, Bram Moolenaar wrote:
>
> If a user switches off 'writebackup' he must be prepared for losing
> work.  Perhaps this is not sufficiently clear to the user?

The file should never be truncated because of some 'error' in the
settings.
The user that removed the 'writebackup' may not be the same user that
set
'fenc'. It is not fair for the second user to pay the price of
someone
else's 'mistake'. The software is there to handle such cases without
loosing any data.

Marko

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Ben Fritz]

On Wed, May 16, 2012 at 2:29 AM, Marko Mahnič <[hidden email]> wrote:
> On May 16, 4:59 am, Michael Henry wrote:
>>
>> How does the following sound for damage control?  If Vim
>> encounters an encoding error during a write, it could
>> automatically re-try the write with utf-8 (or whatever fail-safe
>> encoding makes sense) to prevent leaving the user with an empty
>> file.

I don't like it, because a failed write will still modify the file. I
think nobody would expect that a failed write would modify the file.
I'd rather check beforehand whether the write can succeed before even
trying. I do like it better than destroying the file. I'd rather have
the file encoding change than the file contents be deleted. A warning
message should be displayed, that says, "Encoding conversion error
corrupted file contents during write. File encoding on disk is now
xxxxx."

>
> Wouldn't it be better if Vim would ask the user what to do when
> encoding
> with 'fenc' fails?  Some of the options could be:
>   - save with 'enc' encoding
>   - save with 'fenc' encoding and replace invalid characters with '?'
>   - don't save anything, I'll fix it
>
> This options should be give no matter what the value of 'writebackup'
> is.

This would be nice.

> Until the user selects an option, nothing should be written to disk.

The problem is, something IS written to disk before the conversion
error occurs. The file is truncated in preparation for writing, but
then the write fails, so we abandon the write. But the damage has
already been done.

Is there a reason we couldn't do the encoding conversion in memory,
and if it succeeds, only then open the file to write the data? That
seems like the sanest way to do things. Only open the file when you
know the write will succeed. IIUC, that's the gist of Christian's
patch, except the patch does the conversion twice to determine whether
the write will succeed. Why can't we just store off and re-use the
already converted text instead of converting again?

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Christian Brabandt]

Hi Benjamin!

On Mi, 16 Mai 2012, Benjamin Fritz wrote:

> Is there a reason we couldn't do the encoding conversion in memory,
> and if it succeeds, only then open the file to write the data? That
> seems like the sanest way to do things. Only open the file when you

Well, I wouldn't rather not have 2 copies of the buffer in memory.

regards,
Christian
--

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Benjamin R. Haskell-8]

On Thu, 17 May 2012, Christian Brabandt wrote:

> Hi Benjamin!
>
> On Mi, 16 Mai 2012, Benjamin Fritz wrote:
>
>> Is there a reason we couldn't do the encoding conversion in memory,
>> and if it succeeds, only then open the file to write the data? That
>> seems like the sanest way to do things. Only open the file when you
>
> Well, I wouldn't rather not have 2 copies of the buffer in memory.

Surely there's a way to "do the encoding conversion in memory" without
creating a full copy of the buffer.  Just running through the buffer
character-by-character, or line-wise, and ensuring that all characters
can be encoded using &fenc should work.

--
Best,
Ben H

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Alessandro Antonello]

But, this way, how can Vim rollback if the conversion fails? Undoing everything in reverse order? IMHO, the 'writebackup' option should be ignored when a character encoding conversion is needed. Warning the user when that fails.

Em 17/05/2012 08:09, "Benjamin R. Haskell" <[hidden email]> escreveu:

On Thu, 17 May 2012, Christian Brabandt wrote:

Hi Benjamin!

On Mi, 16 Mai 2012, Benjamin Fritz wrote:

Is there a reason we couldn't do the encoding conversion in memory, and if it succeeds, only then open the file to write the data? That seems like the sanest way to do things. Only open the file when you

Well, I wouldn't rather not have 2 copies of the buffer in memory.

Surely there's a way to "do the encoding conversion in memory" without creating a full copy of the buffer.  Just running through the buffer character-by-character, or line-wise, and ensuring that all characters can be encoded using &fenc should work.

--
Best,
Ben H

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Benjamin R. Haskell-8]

[Reversing the top-posting]

On Thu, 17 May 2012, Alessandro Antonello wrote:

> Em 17/05/2012 08:09, "Benjamin R. Haskell" escreveu:
>
>> On Thu, 17 May 2012, Christian Brabandt wrote:
>>
>>  Hi Benjamin!
>>>
>>> On Mi, 16 Mai 2012, Benjamin Fritz wrote:
>>>
>>>> Is there a reason we couldn't do the encoding conversion in memory,
>>>> and if it succeeds, only then open the file to write the data? That
>>>> seems like the sanest way to do things.
>>>>
>>>
>>> Well, I wouldn't rather not have 2 copies of the buffer in memory.
>>>
>>
>> Surely there's a way to "do the encoding conversion in memory"
>> without creating a full copy of the buffer.  Just running through the
>> buffer character-by-character, or line-wise, and ensuring that all
>> characters can be encoded using &fenc should work.
>
> But, this way, how can Vim rollback if the conversion fails?  Undoing
> everything in reverse order?

Sorry for the imprecision: there's nothing to rollback... It's a test
conversion.  More fully:

If encoding conversion is necessary for the write, there are two passes
through the buffer:

First pass: for every character in the buffer, ensure that it can be
converted to the target encoding.  (This is in memory, so it's not as
time-consuming as a full "practice write" to disk.)

Second pass: write the file to disk, using the target encoding
(converting on-the-fly, or however the current process works).


Not sure how this would work for BufWriteCmd hackery.  (But I'm not sure
how any of the proposals would work w.r.t. BufWriteCmd)


> IMHO, the 'writebackup' option should be ignored when a character
> encoding conversion is needed.  Warning the user when that fails.

IMHO, the 'writebackup' option should be kept to whatever the user sets
it.

Vim simply shouldn't destroy the user's data due to a failed character
conversion.  Despite its rarity, it's a completely-preventable condition
that Vim can detect before the original file is truncated.

--
Best,
Ben H

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Christian Brabandt]

Hi Benjamin!

On Do, 17 Mai 2012, Benjamin R. Haskell wrote:

> If encoding conversion is necessary for the write, there are two
> passes through the buffer:
>
> First pass: for every character in the buffer, ensure that it can be
> converted to the target encoding.  (This is in memory, so it's not
> as time-consuming as a full "practice write" to disk.)
>
> Second pass: write the file to disk, using the target encoding
> (converting on-the-fly, or however the current process works).

That is what my patch does.

> Vim simply shouldn't destroy the user's data due to a failed
> character conversion.  Despite its rarity, it's a
> completely-preventable condition that Vim can detect before the
> original file is truncated.

Yes.

regards,
Christian
--

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Benjamin R. Haskell-8]

On Thu, 17 May 2012, Christian Brabandt wrote:

> Hi Benjamin!
>
> On Do, 17 Mai 2012, Benjamin R. Haskell wrote:
>
>> If encoding conversion is necessary for the write, there are two
>> passes through the buffer:
>>
>> First pass: for every character in the buffer, ensure that it can be
>> converted to the target encoding.  (This is in memory, so it's not as
>> time-consuming as a full "practice write" to disk.)
>>
>> Second pass: write the file to disk, using the target encoding
>> (converting on-the-fly, or however the current process works).
>
> That is what my patch does.

The updated one?  Or did I misinterpret the test results from this
thread?  It sounded as though the change doubled the time it would take
to write a large file.


>> Vim simply shouldn't destroy the user's data due to a failed
>> character conversion.  Despite its rarity, it's a
>> completely-preventable condition that Vim can detect before the
>> original file is truncated.
>
> Yes.

If your patch doesn't double the time it takes to write large files, and
it prevents data loss, then it seems clearly to be the right thing to
do.

--
Best,
Ben H

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Christian Brabandt]

Hi Benjamin!

On Do, 17 Mai 2012, Benjamin R. Haskell wrote:

> On Thu, 17 May 2012, Christian Brabandt wrote:
>
> >Hi Benjamin!
> >
> >On Do, 17 Mai 2012, Benjamin R. Haskell wrote:
> >
> >>If encoding conversion is necessary for the write, there are two
> >>passes through the buffer:
> >>
> >>First pass: for every character in the buffer, ensure that it
> >>can be converted to the target encoding.  (This is in memory, so
> >>it's not as time-consuming as a full "practice write" to disk.)
> >>
> >>Second pass: write the file to disk, using the target encoding
> >>(converting on-the-fly, or however the current process works).
> >
> >That is what my patch does.
>
> The updated one?  Or did I misinterpret the test results from this
> thread?  It sounded as though the change doubled the time it would
> take to write a large file.

Both did. I was also surprised, that just converting the buffer content
without writing almost doubled the write speed. The updated one just
kicked in, when no backup was done.


regards,
Christian
--

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Ben Fritz]

On Thu, May 17, 2012 at 4:00 AM, Christian Brabandt <[hidden email]> wrote:

> Hi Benjamin!
>
> On Mi, 16 Mai 2012, Benjamin Fritz wrote:
>
>> Is there a reason we couldn't do the encoding conversion in memory,
>> and if it succeeds, only then open the file to write the data? That
>> seems like the sanest way to do things. Only open the file when you
>
> Well, I wouldn't rather not have 2 copies of the buffer in memory.
>

Why not? I doubt very much somebody will be so low on memory that they
can't add another buffer to Vim. It would only be there temporarily as
well. And it wouldn't be an exact copy, unless fenc==enc, in which
case the conversion cannot fail and we can skip the conversion step.

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php