BOM in QuickFix window

classic Classic list List threaded Threaded
5 messages Options
BAA
Reply | Threaded
Open this post in threaded view
|

BOM in QuickFix window

BAA

Hi.

I'm using few MONO apps (Nemerle compiler and COCO/R parser generator)
through compiler plugins in VIM under Linux, and the problem is that
they produce output starting with FEFF.
I had this problem about a half a year ago and being (and staying) a
newbie, thought that the problem was with *errorformat* but didn't
cope defining it correctly - not sure if it is possible.
I found workaround, setting LANG=en_US in compiler call command.

I have encoding and termencoding set to 'utf-8' and locale is
'en_US.utf8'.

Is there a better way to solve this problem?

Thanks

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: BOM in QuickFix window

Tony Mechelynck

On 08/07/09 22:48, Aleksey wrote:

>
> Hi.
>
> I'm using few MONO apps (Nemerle compiler and COCO/R parser generator)
> through compiler plugins in VIM under Linux, and the problem is that
> they produce output starting with FEFF.
> I had this problem about a half a year ago and being (and staying) a
> newbie, thought that the problem was with *errorformat* but didn't
> cope defining it correctly - not sure if it is possible.
> I found workaround, setting LANG=en_US in compiler call command.
>
> I have encoding and termencoding set to 'utf-8' and locale is
> 'en_US.utf8'.
>
> Is there a better way to solve this problem?
>
> Thanks

Make sure your 'fileencodings' option (plural) starts with "ucs-bom". In
that case, for editfiles at least (not sure about quickfix error files),
Vim will recognise the Unicode codepoint U+FEFF (known as the BOM for
Byte Order Mark though actually it's more than that) when it happens at
the very start of a file, and set 'fileencoding' (singular) and 'bomb'
accordingly for that file, as follows:

Lead bytes (hex)     'fileencoding'      'bomb'
EF BB BF             utf-8               bomb
00 00 FE FF          ucs-4               bomb
FF FE 00 00          ucs-4le             bomb
FE FF                utf-16              bomb
FF FE                utf-16le            bomb
anything else        not yet known       nobomb

As you can see, the BOM allows us to identify all Unicode encodings
(endianness included, but treating UCS-2 as a particular case of UTF-16,
which it is) assuming that no little-endian UTF-16 file will ever start
with a NULL codepoint, which I think is a reasonable assumption. ("Not
yet known" in the table above means "try the next entry in
'fileencodings'".)

If, after trying it, you find that it doesn't work for the quickfix
errorfile, come back to report it, and in that case maybe Bram will pass
by and add it to his TODO list. But even if he does, don't set your
hopes too high: it's a very long list, see |todo.txt| in the Vim help
(the file is currently 4705 lines long as of 2 July 2009, but I didn't
count how many todo items (which should be fewer ;-) ) are in it.

IIUC, even if 'fileencodings' doesn't work for quickfix error files,
setting LC_MESSAGES should be enough (LANG is used as a default for any
LC_<something> which is unset, or LC_ALL if present overrides any other
LC_* even if present). Maybe even just using ":language messages en-US"
or ":lang mess C" near the top of your vimrc could be enough.



Best regards,
Tony.
--
hundred-and-one symptoms of being an internet addict:
66. You create a homepage with the impression to cure the afflicted...but
     your hidden agenda is to receive more e-mail.

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

BAA
Reply | Threaded
Open this post in threaded view
|

Re: BOM in QuickFix window

BAA

> Make sure your 'fileencodings' option (plural) starts with "ucs-bom".

> IIUC, even if 'fileencodings' doesn't work for quickfix error files,
> setting LC_MESSAGES should be enough (LANG is used as a default for any
> LC_<something> which is unset, or LC_ALL if present overrides any other
> LC_* even if present). Maybe even just using ":language messages en-US"
> or ":lang mess C" near the top of your vimrc could be enough.
>
> Best regards,
> Tony.
> --
> hundred-and-one symptoms of being an internet addict:
> 66. You create a homepage with the impression to cure the afflicted...but
>      your hidden agenda is to receive more e-mail.

Thanks for your answer.

fileencoding already had 'ucs-bom' in its start, so it didnt' help.
I also have :language messages C, so it didn't help either.
I didn't quite understand what you wrote about LC_ stuff, because I
lack even basic knowledge, but does it mean my solution with setting
LANG=en_US is fine?

Reading docs for VIM 7.2 (:help quickfix) I found the following
paragraph:
    When 'encoding' differs from the locale, the error messages may
have a
    different encoding from what Vim is using.  To convert the
messages you can
    use this code:
        function QfMakeConv()
           let qflist = getqflist()
           for i in qflist
              let i.text = iconv(i.text, "cp936", "utf-8")
           endfor
           call setqflist(qflist)
        endfunction

        au QuickfixCmdPost make call QfMakeConv()

I've tried putting this to .vimrc replacing "cp936" with ucs-bom, but
not sure if that was right. It didn't change anything.

I don't think this problem is worth of Bram's attention since it's not
really wide-spread and the workaround is available.
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: BOM in QuickFix window

Tony Mechelynck

On 09/07/09 08:52, Aleksey wrote:
>
>> Make sure your 'fileencodings' option (plural) starts with "ucs-bom".
------------------------------------------^^^^^^

>
>> IIUC, even if 'fileencodings' doesn't work for quickfix error files,
>> setting LC_MESSAGES should be enough (LANG is used as a default for any
>> LC_<something>  which is unset, or LC_ALL if present overrides any other
>> LC_* even if present). Maybe even just using ":language messages en-US"
>> or ":lang mess C" near the top of your vimrc could be enough.
>>
>> Best regards,
>> Tony.
>> --
>> hundred-and-one symptoms of being an internet addict:
>> 66. You create a homepage with the impression to cure the afflicted...but
>>       your hidden agenda is to receive more e-mail.
>
> Thanks for your answer.
>
> fileencoding already had 'ucs-bom' in its start, so it didnt' help.

should be 'fileencodings' with s at the end. There are two different
options, with and without s, and they don't have the same meaning.
Without s, it's the encoding used on disk for one file at a time. With
s, it's the heuristics to find what to use when opening an existing file.

> I also have :language messages C, so it didn't help either.
> I didn't quite understand what you wrote about LC_ stuff, because I
> lack even basic knowledge, but does it mean my solution with setting
> LANG=en_US is fine?

yeah, sure.

>
> Reading docs for VIM 7.2 (:help quickfix) I found the following
> paragraph:
>      When 'encoding' differs from the locale, the error messages may
> have a
>      different encoding from what Vim is using.  To convert the
> messages you can
>      use this code:
> function QfMakeConv()
>   let qflist = getqflist()
>   for i in qflist
>      let i.text = iconv(i.text, "cp936", "utf-8")
>   endfor
>   call setqflist(qflist)
> endfunction
>
> au QuickfixCmdPost make call QfMakeConv()
>
> I've tried putting this to .vimrc replacing "cp936" with ucs-bom, but
> not sure if that was right. It didn't change anything.

no, iconv doesn't know about ucs-bom.

>
> I don't think this problem is worth of Bram's attention since it's not
> really wide-spread and the workaround is available.

How can one know how widespread it is? Sounds like the age-old "Nobody
uses it" used without proof by developers of some applications (not by
Bram) when speaking of a feature they want to remove because they don't
use it themselves and they don't want to support it.

OK, so IIUC the problem seems to be: When a compiler errorfile starts
with a BOM, Vim doesn't know how to handle it. I'm not sure if there's
an 'errorfile' setting allowing to get over it because I've never looked
into that option myself.


Best regards,
Tony.
--
"Benson, you are so free of the ravages of intelligence"
                -- Time Bandits

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

BAA
Reply | Threaded
Open this post in threaded view
|

Re: BOM in QuickFix window

BAA

On Jul 9, 1:51 pm, Tony Mechelynck <[hidden email]>
wrote:

> should be 'fileencodings' with s at the end. There are two different
> options, with and without s, and they don't have the same meaning.
> Without s, it's the encoding used on disk for one file at a time. With
> s, it's the heuristics to find what to use when opening an existing file.

Sorry, mistyped it. It's 'fileencodings' certainly, and 'ucs-bom' is
the list.


> > I don't think this problem is worth of Bram's attention since it's not
> > really wide-spread and the workaround is available.
>
> How can one know how widespread it is? Sounds like the age-old "Nobody
> uses it" used without proof by developers of some applications (not by
> Bram) when speaking of a feature they want to remove because they don't
> use it themselves and they don't want to support it.

I didn't mean that. I've made such decision because I searched a lot
for similar
 problem and found almost nothing except your recent answers to 'Match
a BOM'
in a file. Which deals with different issue, but it made me think
about mine.

I wanted to say, that since this issue has a solution and isn't
reported by other developers,
people know how to live with it or don't care about it. It's just
about not wasting time on low priority stuff, IMHO.
I'm just glad to understand what the problem is.

>
> OK, so IIUC the problem seems to be: When a compiler errorfile starts
> with a BOM, Vim doesn't know how to handle it. I'm not sure if there's
> an 'errorfile' setting allowing to get over it because I've never looked
> into that option myself.
>

Yeah, you've got it right. Thank you for your time, I'll settle down
on LANG=en_US which is just ok for me.
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---