utf problems with regular vim (not macvim) on os x

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

utf problems with regular vim (not macvim) on os x

lanzz-2

hello everybody,
i have problems getting vim to work with unicode in OS X. it can open
unicode text files fine, properly detects that they are in utf8, and
displays them fine; however if i try to enter unicode (codes above
127) via the keyboard, vim interprets the input as single-byte latin1
input and re-encodes it from latin1 to utf8, which does not produce
the correct results.

vim is :set termencoding=utf8, LC_CTYPE is en_US.utf8. i've verified
that the OS X is producing the proper utf8 sequences. utf8 works
correctly in bash (properly displayed and text length is calculated
correctly). the strange thing is that if i ssh from a linux box to
the
mac, vim works correctly with utf8. it is apparently something
related
to the OS X terminal, but i cannot figure out what it is, since
typing
"echo <some-cyrillic-text> | xxd" in shell produces exactly the same
byte sequence on the mac as the same command on linux.

any insights?


--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: utf problems with regular vim (not macvim) on os x

Tony Mechelynck

lanzz wrote:

> hello everybody,
> i have problems getting vim to work with unicode in OS X. it can open
> unicode text files fine, properly detects that they are in utf8, and
> displays them fine; however if i try to enter unicode (codes above
> 127) via the keyboard, vim interprets the input as single-byte latin1
> input and re-encodes it from latin1 to utf8, which does not produce
> the correct results.
>
> vim is :set termencoding=utf8, LC_CTYPE is en_US.utf8. i've verified
> that the OS X is producing the proper utf8 sequences. utf8 works
> correctly in bash (properly displayed and text length is calculated
> correctly). the strange thing is that if i ssh from a linux box to
> the
> mac, vim works correctly with utf8. it is apparently something
> related
> to the OS X terminal, but i cannot figure out what it is, since
> typing
> "echo <some-cyrillic-text> | xxd" in shell produces exactly the same
> byte sequence on the mac as the same command on linux.
>
> any insights?

When you do

        vim -N -u NONE
                :verbose set enc? tenc?

what is the reply?

If 'encoding' is not set to UTF-8 at startup, and 'termencoding' is empty, you
should preserve the console encoding as follows:

        if &enc !~? '^u' " nothing to do if already Unicode
                if &tenc == ''
                        " avoid clobbering the keyboard encoding
                        let &tenc = &enc
                endif
                set enc=utf-8

  " the following line (defaults for new files) is optional
                setglobal bomb fenc=latin1

                " set encoding detection heuristics
                set fencs=ucs-bom,utf-8,latin1
        endif



Best regards,
Tony.
--
hundred-and-one symptoms of being an internet addict:
29. Your phone bill comes to your doorstep in a box.

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: utf problems with regular vim (not macvim) on os x

lanzz-2

On Sep 28, 4:06 pm, Tony Mechelynck <[hidden email]>
wrote:

> When you do
>
>         vim -N -u NONE
>                 :verbose set enc? tenc?
>
> what is the reply?

  encoding=utf-8
  termencoding=


--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: utf problems with regular vim (not macvim) on os x

Stefán-78

Hi

I am probably too late to answer this thread, but I ran into the same
problem.

It seems to be a problem with the settings of Terminal.app . By going
to the menu Terminal -> Window Settings... , then the Emulation page
there and unchecking "Escape non-ASCII characters" the problem seems
to have disappeared. If it works, you have to click the "Use Settings
as Default" to make the change pernanent. (Bash, after applying the
hints at http://www.macosxhints.com/article.php?story=20060825071728278
seems not to care about this "Escape non-ASCII characters" business.
But that is another matter.)

Also note that even with this solution, Terminal.app does a pretty
lousy job of utf-8 rendering, as can be seen from trying "cat" or
"vim" with UTF-8-demo.txt . Both mess up eg. polytonic greek pretty
badly.

Stefán

On Sep 28, 5:21 am, lanzz <[hidden email]> wrote:

> On Sep 28, 4:06 pm, Tony Mechelynck <[hidden email]>
> wrote:
>
> > When you do
>
> >         vim -N -u NONE
> >                 :verbose set enc? tenc?
>
> > what is the reply?
>
>   encoding=utf-8
>   termencoding=


--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: utf problems with regular vim (not macvim) on os x

David Sewell-2
On Fri, 9 Nov 2007, Stefán wrote:

> It seems to be a problem with the settings of Terminal.app . By going
> to the menu Terminal -> Window Settings... , then the Emulation page
> there and unchecking "Escape non-ASCII characters" the problem seems
> to have disappeared. If it works, you have to click the "Use Settings
> as Default" to make the change pernanent. (Bash, after applying the
> hints at http://www.macosxhints.com/article.php?story=20060825071728278
> seems not to care about this "Escape non-ASCII characters" business.
> But that is another matter.)
>
> Also note that even with this solution, Terminal.app does a pretty
> lousy job of utf-8 rendering, as can be seen from trying "cat" or
> "vim" with UTF-8-demo.txt . Both mess up eg. polytonic greek pretty
> badly.
In the Terminal.app Display preferences, are both the options for "Wide
glyphs" unchecked? Greek and most of the other fonts display properly
for me with these boxes unchecked.

--
David Sewell, Editorial and Technical Manager
Electronic Imprint, The University of Virginia Press
PO Box 400318, Charlottesville, VA 22904-4318 USA
Courier: 310 Old Ivy Way, Suite 302, Charlottesville VA 22903
Email: [hidden email]   Tel: +1 434 924 9973
Web: http://www.ei.virginia.edu/
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: utf problems with regular vim (not macvim) on os x

Stefán-78

Thanks

Unchecking the two "wide glyphs" options helped a lot. Now the
polytonic greek works properly. I think I finally made my Tiger
terminal work as I want it. (And they just released Leopard :)

Stefán

On Nov 10, 6:26 am, David Sewell <[hidden email]> wrote:

> On Fri, 9 Nov 2007, Stefán wrote:
> > It seems to be a problem with the settings of Terminal.app . By going
> > to the menu Terminal -> Window Settings... , then the Emulation page
> > there and unchecking "Escape non-ASCII characters" the problem seems
> > to have disappeared. If it works, you have to click the "Use Settings
> > as Default" to make the change pernanent. (Bash, after applying the
> > hints athttp://www.macosxhints.com/article.php?story=20060825071728278
> > seems not to care about this "Escape non-ASCII characters" business.
> > But that is another matter.)
>
> > Also note that even with this solution, Terminal.app does a pretty
> > lousy job of utf-8 rendering, as can be seen from trying "cat" or
> > "vim" with UTF-8-demo.txt . Both mess up eg. polytonic greek pretty
> > badly.
>
> In the Terminal.app Display preferences, are both the options for "Wide
> glyphs" unchecked? Greek and most of the other fonts display properly
> for me with these boxes unchecked.
>
> --
> David Sewell, Editorial and Technical Manager
> Electronic Imprint, The University of Virginia Press
> PO Box 400318, Charlottesville, VA 22904-4318 USA
> Courier: 310 Old Ivy Way, Suite 302, Charlottesville VA 22903
> Email: [hidden email]   Tel: +1 434 924 9973
> Web:http://www.ei.virginia.edu/


--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: utf problems with regular vim (not macvim) on os x

lanzz-2
In reply to this post by Stefán-78

On Nov 10, 12:14 am, Stefán <[hidden email]> wrote:
> It seems to be a problem with the settings of Terminal.app . By going
> to the menu Terminal -> Window Settings... , then the Emulation page
> there and unchecking "Escape non-ASCII characters" the problem seems
> to have disappeared.

great, thanks! this fixed the problem.

> Also note that even with this solution, Terminal.app does a pretty
> lousy job of utf-8 rendering, as can be seen from trying "cat" or
> "vim" with UTF-8-demo.txt . Both mess up eg. polytonic greek pretty
> badly.

fortunately cyrillic works fine :)
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---