bug in experimental renderer

[dv1445]

I think There is a bug in MacVim regarding multibyte characters.  It
only happens when the Experimental Renderer is turned ON.

To reproduce:

1. make a new folder "bar".
2. In bar, do "touch föo", where there is an umlaut over the first 'o'.
3. Start MacVim with mvim -u NONE -U NONE.
4. In MacVim, do "cd bar".
5. Now do ":r !ls".
6. Put a couple of blank lines after the result of that.
7. Now do :let @a=glob("*"), followed by :put a

Expected behavior: There should be two instances of "föo" in the buffer.

Actual behavior: Depending on what your value of 'gfn' is, you might see
that one of the "föo" lacks an umlaut.  For instance, if you use
Consolas or Monaco, the one that was the result of "ls" (i.e., the top
one) will lack an umlaut while the bottom one will look correct.  If you
use Courier New or Menlo, then both words have the umlaut.

Actual behavior part 2: Here's where it gets weird.  Navigate to each of
the "o" that are supposed to have an umlaut (the first of each pair) and
do "ga".  Vim's feedback for the top-foo (which came from ls) is:

<o>  111,  Hex 6f,  Octal 157 < ̈> 776, Hex 0308, Octal 1410

Vim's feedback for the bottom-foo (which came from glob()) is:

<ö> 246, Hex 00f6, Octal 366

Again, depending on your font, "ga" might give umlauts to both, or it
might not.  What's weird is that "ga" shows the two "o"s to in fact be
different characters.

Observation: The difference in "ga" happens also if you use MacVim's
inner-Vim as a console vim (i.e., replace step 3 with
'/MacVim.app/Contents/MacOS/Vim -u NONE', which might require you make
a bash alias for launching).  However, I get umlauts displayed for
each in this case.

Observation 2: But, it does NOT happen on a true console vim (one
built from vim's mercurial repo).  By "not happen" I mean that in a
true console vim, the output of doing "ga" over the characters in
question is identical, and is the same as the output that MacVim gives
to the top case (from ls).  Also, umlauts are present in each.

I don't know anything about multibute or composing characters, and am
not a programmer.  But I conjecture that there is a bug in the
experimental renderer that manifests itself depending on what font one
uses (the sometimes-missing umlauts), and there is probably an
*additional* bug in MacVim that makes the two "ga" yield different
results (which happens even when using MacVim in the console where the
umlauts show correctly).  Of course, it might be one bug at bottom
with two manifestations.

Summary:

In MacVim...    show umlauts?          "ga" yields...
                --------------         ----------------
from ls         NO                     111 & 776
from glob()     YES                    246
           
In MacVim within console...
                show umlauts?          "ga" yields...
                ----------------       ----------------
from ls         YES                    111 & 776
from glob()     YES                    246

In a true console vim....
                show umlauts?          "ga" yields...
                ----------------       ----------------
from ls         YES                    111 & 776
from glob()     YES                    111 & 776

-gmn

--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Björn Winckler]

On Tue, Mar 20, 2012 at 11:30 PM,  wrote:

> I think There is a bug in MacVim regarding multibyte characters.  It
> only happens when the Experimental Renderer is turned ON.
>
> To reproduce:
>
> 1. make a new folder "bar".
> 2. In bar, do "touch föo", where there is an umlaut over the first 'o'.
> 3. Start MacVim with mvim -u NONE -U NONE.
> 4. In MacVim, do "cd bar".
> 5. Now do ":r !ls".
> 6. Put a couple of blank lines after the result of that.
> 7. Now do :let @a=glob("*"), followed by :put a
>
> Expected behavior: There should be two instances of "föo" in the buffer.
>
> Actual behavior: Depending on what your value of 'gfn' is, you might see
> that one of the "föo" lacks an umlaut.  For instance, if you use
> Consolas or Monaco, the one that was the result of "ls" (i.e., the top
> one) will lack an umlaut while the bottom one will look correct.  If you
> use Courier New or Menlo, then both words have the umlaut.
>
> Actual behavior part 2: Here's where it gets weird.  Navigate to each of
> the "o" that are supposed to have an umlaut (the first of each pair) and
> do "ga".  Vim's feedback for the top-foo (which came from ls) is:
>
> <o>  111,  Hex 6f,  Octal 157 < ̈> 776, Hex 0308, Octal 1410
>
> Vim's feedback for the bottom-foo (which came from glob()) is:
>
> <ö> 246, Hex 00f6, Octal 366

This has to do with Unicode normalization [1].  The output from !ls is
not composed, whereas when you paste it in Vim it is composed.  The
problem arises because you are piping characters from an external
command.

I suppose we could try to normalize all input from external commands
but the problem is that it need not be in Unicode format so this may
cause new problems and/or be slower.  I'll put a note on my todo list.

Björn

[1] http://en.wikipedia.org/wiki/Unicode_equivalence#Normalization

--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[dv1445]

On 03/24/12 at 13:07 +0100, björn wrote:

> On Tue, Mar 20, 2012 at 11:30 PM,  wrote:
> > I think There is a bug in MacVim regarding multibyte characters.  It
> > only happens when the Experimental Renderer is turned ON.
> >
> > To reproduce:
> >
> > 1. make a new folder "bar".
> > 2. In bar, do "touch föo", where there is an umlaut over the first 'o'.
> > 3. Start MacVim with mvim -u NONE -U NONE.
> > 4. In MacVim, do "cd bar".
> > 5. Now do ":r !ls".
> > 6. Put a couple of blank lines after the result of that.
> > 7. Now do :let @a=glob("*"), followed by :put a
> >
> > Expected behavior: There should be two instances of "föo" in the buffer.
> >
> > Actual behavior: Depending on what your value of 'gfn' is, you might see
> > that one of the "föo" lacks an umlaut.  For instance, if you use
> > Consolas or Monaco, the one that was the result of "ls" (i.e., the top
> > one) will lack an umlaut while the bottom one will look correct.  If you
> > use Courier New or Menlo, then both words have the umlaut.
> >
> > Actual behavior part 2: Here's where it gets weird.  Navigate to each of
> > the "o" that are supposed to have an umlaut (the first of each pair) and
> > do "ga".  Vim's feedback for the top-foo (which came from ls) is:
> >
> > <o>  111,  Hex 6f,  Octal 157 < ̈> 776, Hex 0308, Octal 1410
> >
> > Vim's feedback for the bottom-foo (which came from glob()) is:
> >
> > <ö> 246, Hex 00f6, Octal 366
>
> This has to do with Unicode normalization [1].  The output from !ls is
> not composed, whereas when you paste it in Vim it is composed.  The
> problem arises because you are piping characters from an external
> command.
>
> I suppose we could try to normalize all input from external commands
> but the problem is that it need not be in Unicode format so this may
> cause new problems and/or be slower.  I'll put a note on my todo list.
>
> Björn
>
> [1] http://en.wikipedia.org/wiki/Unicode_equivalence#Normalization

Hi Björn,

Thanks for putting this on the TODO list.  I just want to clear up a bit of
confusion on my part.

I thought that (1) the outputs of both ":r !ls" and glob() should be
indentical, and also that (2) they should both do it the way that !ls does it.
The reason I think this is because that's how "true" console vim (from Bram's
hg repo) does it, and we want to stay as much like true vim as is humanly
possible.

Is this right?

If it is right, then what would need fixing is the output of glob(), not that
of !ls.  It seemed to me in your message that you were thinking about
tinkering with !ls (and pipe-ins generally), but I wasn't sure.

It also now seems that the non-showing of umlauts (depending on the font) is
really a separate bug, probably in the experimental renderer; this separate
bug was only *revealed* by MacVim's treating !ls and glob() differently, not
caused by it.

-gmn

--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Björn Winckler]

On Sat, Jul 7, 2012 at 6:55 PM,  <[hidden email]> wrote:
>
> I thought that (1) the outputs of both ":r !ls" and glob() should be
> indentical, and also that (2) they should both do it the way that !ls does it.
> The reason I think this is because that's how "true" console vim (from Bram's
> hg repo) does it, and we want to stay as much like true vim as is humanly
> possible.
>
> Is this right?

They are not identical, ":r !ls" is like typing "ls" in Terminal
whereas glob() is implemented in the Vim source code.

The other difference here is that console Vim handles output from
commands like ":!ls" differently from GUI Vim (specifically, GUI Vim
does not come with a terminal emulator).

> If it is right, then what would need fixing is the output of glob(), not that
> of !ls.  It seemed to me in your message that you were thinking about
> tinkering with !ls (and pipe-ins generally), but I wasn't sure.

Are you saying that the problem only happens when you use glob() but
not ":e !ls"?

> It also now seems that the non-showing of umlauts (depending on the font) is
> really a separate bug, probably in the experimental renderer; this separate
> bug was only *revealed* by MacVim's treating !ls and glob() differently, not
> caused by it.

I think that there are problems related to rendering decomposed UTF.
If I remember correctly HFS+ stores filenames in a decomposed format
(decomposed form NFD) whereas for things to work they need to be in
normalization form C (some parts of the code uses routines in
Miscellaneous.h to do this conversion).

If it is only glob() needs to be patched, then it may be possible to
do the conversion to NFC to the relevant parts of the Vim source code
(see f_glob() in eval.c).  I'm not 100% sure if I am understanding you
correctly though.

Björn

--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Ben Schmidt]

On 8/07/12 2:55 AM, [hidden email] wrote:

> On 03/24/12 at 13:07 +0100, björn wrote:
>> On Tue, Mar 20, 2012 at 11:30 PM,  wrote:
>>> I think There is a bug in MacVim regarding multibyte characters.  It
>>> only happens when the Experimental Renderer is turned ON.
>>>
>>> To reproduce:
>>>
>>> 1. make a new folder "bar".
>>> 2. In bar, do "touch föo", where there is an umlaut over the first 'o'.
>>> 3. Start MacVim with mvim -u NONE -U NONE.
>>> 4. In MacVim, do "cd bar".
>>> 5. Now do ":r !ls".
>>> 6. Put a couple of blank lines after the result of that.
>>> 7. Now do :let @a=glob("*"), followed by :put a
>>>
>>> Expected behavior: There should be two instances of "föo" in the buffer.
>>>
>>> Actual behavior: Depending on what your value of 'gfn' is, you might see
>>> that one of the "föo" lacks an umlaut.  For instance, if you use
>>> Consolas or Monaco, the one that was the result of "ls" (i.e., the top
>>> one) will lack an umlaut while the bottom one will look correct.  If you
>>> use Courier New or Menlo, then both words have the umlaut.
>>>
>>> Actual behavior part 2: Here's where it gets weird.  Navigate to each of
>>> the "o" that are supposed to have an umlaut (the first of each pair) and
>>> do "ga".  Vim's feedback for the top-foo (which came from ls) is:
>>>
>>> <o>    111,  Hex 6f,  Octal 157<  ̈>  776, Hex 0308, Octal 1410
>>>
>>> Vim's feedback for the bottom-foo (which came from glob()) is:
>>>
>>> <ö>  246, Hex 00f6, Octal 366
>>
>> This has to do with Unicode normalization [1].

Yes, but that just *reveals* the bug; it isn't the bug.

The subject line correctly identifies this as a rendering bug.

It should be possible to edit both composed and non-composed characters
in Vim. And indeed it is. For example, :set delcombine, and then issue x
on top of the composed character and it will delete just the umlaut.

However, you can't see this happen in MacVim because for some reason
MacVim doesn't overlay the composing characters, but just doesn't
display them at all. However, console Vim displays them fine in
Terminal.app.

>> I suppose we could try to normalize all input from external commands
>> but the problem is that it need not be in Unicode format so this may
>> cause new problems and/or be slower.  I'll put a note on my todo list.

This isn't a good idea at all. MacVim shouldn't manipulate its input. It
should just display it properly. Doing it just for external commands
wouldn't solve the problem when composing characters are found in a file
being edited, either.

We need to find and fix the bug in the renderer.

(I actually thought the bug had been fixed, or at least made some
progress, because Hebrew with vowels and accents seems to be displaying
OK, and it's got composing characters everywhere. It's strange that it's
working at least to some extent for Hebrew but not for simpler European
characters.)

Smiles,

Ben.



--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[dv1445]

björn wrote on 07/10/12 at 23:18:38 +0200:

> On Sat, Jul 7, 2012 at 6:55 PM,  <[hidden email]> wrote:
> > I thought that (1) the outputs of both ":r !ls" and glob() should be
> > indentical, and also that (2) they should both do it the way that
> > !ls does it.  The reason I think this is because that's how "true"
> > console vim (from Bram's hg repo) does it, and we want to stay as
> > much like true vim as is humanly possible.
> >
> > Is this right?
>
> They are not identical, ":r !ls" is like typing "ls" in Terminal
> whereas glob() is implemented in the Vim source code.

Right, but it seems to me that if at all possible, their *outputs*
should be identical, in the following sense: If there is a lone
directory or file called 'föo' in the cwd, and we're looking at an empty
vim buffer, then the two commands

   ":r !ls" and

   "let @a = glob('*') | put a"

should put the same characters into the buffer.  They do not.
 

> The other difference here is that console Vim handles output from
> commands like ":!ls" differently from GUI Vim (specifically, GUI Vim
> does not come with a terminal emulator).
>
> > If it is right, then what would need fixing is the output of glob(),
> > not that of !ls.  It seemed to me in your message that you were
> > thinking about tinkering with !ls (and pipe-ins generally), but I
> > wasn't sure.
>
> Are you saying that the problem only happens when you use glob() but
> not ":e !ls"?

There are two "problems" here.  The big problem is that 'ga' over the
umlauted 'o' yields different values when the characters entered the
buffer because of glob() than when the chars entered the buffer because
of !ls.  True console vim has their values as identical, and therefore
if at all possible so should MacVim; also true console vim has them both
behaving the way :!ls does, and therefore MacVim should too if at all
possible.  Summary: MacVim's glob() is not doing what it should, where
'should' is understood as 'doing things the way true vim does'.

The second problem was the titular bug in the experimental renderer;
this bug (the dropping of the umlaut) only manifests itself when the
:!ls procedure is used (and only with some fonts).  That is, this second
bug only appears when MacVim is putting chars in the buffer the right
way.  This experimental rendering bug has been masked by MacVim making
glob() do something different from :!ls.

> > It also now seems that the non-showing of umlauts (depending on the
> > font) is really a separate bug, probably in the experimental
> > renderer; this separate bug was only *revealed* by MacVim's treating
> > !ls and glob() differently, not caused by it.
>
> I think that there are problems related to rendering decomposed UTF.  If I
> remember correctly HFS+ stores filenames in a decomposed format (decomposed
> form NFD) whereas for things to work they need to be in normalization form C
> (some parts of the code uses routines in Miscellaneous.h to do this
> conversion).
>
> If it is only glob() needs to be patched, then it may be possible to do the
> conversion to NFC to the relevant parts of the Vim source code (see f_glob()
> in eval.c).  I'm not 100% sure if I am understanding you correctly though.

Well, I don't know any programming language other than some VimScript,
so I can't help with the code.  Maybe these two bugs are dinky and it's
not worth the effort to fix them, for all I know.  Maybe they aren't
even fixable.  I just thought it was interesting enough to report.

It may be worth noting that the two methods of putting 'föo' in the
buffer, i.e., !ls and glob(), affect certain VimScript functions like
strlen() and strchars()---though I can't recall which did what.  One or
both of those Vim functions returned different values depending on
whether its input came from glob() or from !ls.

-gmn

--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Ben Schmidt]

>> They are not identical, ":r !ls" is like typing "ls" in Terminal
>> whereas glob() is implemented in the Vim source code.
>
> Right, but it seems to me that if at all possible, their *outputs*
> should be identical, in the following sense: If there is a lone
> directory or file called 'föo' in the cwd, and we're looking at an empty
> vim buffer, then the two commands
>
>     ":r !ls" and
>
>     "let @a = glob('*') | put a"
>
> should put the same characters into the buffer.  They do not.

I wouldn't bother myself. ls might change one day. And other commandline
utilities that output filenames might do differently. You're never going
to be able to please everybody or match every utility. There's nothing
wrong with what happens at the moment. Might as well leave it as is. If
you really want to compose/decompose all characters, find some unicode
utility that can do it and pipe your data through it.

> There are two "problems" here.  The big problem is that 'ga' over the
> umlauted 'o' yields different values when the characters entered the
> buffer because of glob() than when the chars entered the buffer because
> of !ls.  True console vim has their values as identical, and therefore
> if at all possible so should MacVim; also true console vim has them both
> behaving the way :!ls does, and therefore MacVim should too if at all
> possible.  Summary: MacVim's glob() is not doing what it should, where
> 'should' is understood as 'doing things the way true vim does'.

This is indeed interesting. It is not a console Vim vs. MacVim issue,
though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470 I
compiled myself. I don't know whether it's something about the way it
was compiled or the version difference that's altered things. Here is my
full :version output:

This has glob() and r !ls the same:

VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Jul 31 2011 19:27:29)
Compiled by [hidden email]
Normal version without GUI.  Features included (+) or not (-):
-arabic +autocmd -balloon_eval -browse +builtin_terms +byte_offset +cindent
-clientserver -clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
-conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con +diff +digraphs
-dnd -ebcdic -emacs_tags +eval +ex_extra +extra_search -farsi +file_in_path
+find_in_path +float +folding -footer +fork() -gettext -hangul_input +iconv
+insert_expand +jumplist -keymap -langmap +libcall +linebreak +lispindent
+listcmds +localmap -lua +menu +mksession +modify_fname +mouse -mouseshape
-mouse_dec -mouse_gpm -mouse_jsbterm -mouse_netterm -mouse_sysmouse
+mouse_xterm +multi_byte +multi_lang -mzscheme +netbeans_intg -osfiletype
+path_extra -perl +persistent_undo +postscript +printer -profile -python
-python3 +quickfix +reltime -rightleft -ruby +scrollbind +signs +smartindent
-sniff +startuptime +statusline -sun_workshop +syntax +tag_binary
+tag_old_static -tag_any_white -tcl +terminfo +termresponse +textobjects +title
  -toolbar +user_commands +vertsplit +virtualedit +visual +visualextra +viminfo
+vreplace +wildignore +wildmenu +windows +writebackup -X11 -xfontset -xim -xsmp
  -xterm_clipboard -xterm_save
    system vimrc file: "$VIM/vimrc"
      user vimrc file: "$HOME/.vimrc"
       user exrc file: "$HOME/.exrc"
   fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -D_FORTIFY_SOURCE=0 -Iproto -DHAVE_CONFIG_H -arch i386
-arch x86_64 -g -Os -pipe
Linking: gcc -arch i386 -arch x86_64 -o vim -lncurses

These have glob() and r !ls different:

VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Mar  9 2012 00:45:32)
MacOS X (unix) version
Included patches: 1-470
Modified by Ben Schmidt et al.
Compiled by [hidden email]
Huge version without GUI.  Features included (+) or not (-):
+arabic +autocmd -balloon_eval -browse ++builtin_terms +byte_offset +cindent
+clientserver +clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
+conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con +diff +digraphs
-dnd -ebcdic +emacs_tags +eval +ex_extra +extra_search +farsi +file_in_path
+find_in_path +float +folding -footer +fork() -gettext -hangul_input -iconv
+insert_expand +jumplist +keymap +langmap +libcall +linebreak +lispindent
+listcmds +localmap -lua +menu +mksession +modify_fname +mouse -mouseshape
+mouse_dec -mouse_gpm -mouse_jsbterm +mouse_netterm -mouse_sysmouse
+mouse_xterm +mouse_urxvt +multi_byte +multi_lang -mzscheme +netbeans_intg
+path_extra +perl +persistent_undo +postscript +printer +profile +python
-python3 +quickfix +reltime +rightleft +ruby +scrollbind +signs +smartindent
-sniff +startuptime +statusline -sun_workshop +syntax +tag_binary
+tag_old_static -tag_any_white +tcl +terminfo +termresponse +textobjects +title
  -toolbar +user_commands +vartabs +vertsplit +virtualedit +visual +visualextra
+viminfo +vreplace +wildignore +wildmenu +windows +writebackup +X11 +xfontset
-xim +xsmp_interact +xterm_clipboard -xterm_save
    system vimrc file: "$VIM/vimrc"
      user vimrc file: "$HOME/.vimrc"
       user exrc file: "$HOME/.exrc"
   fall-back for $VIM: "/usr/local/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X_UNIX -no-cpp-precomp
-g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
-I/System/Library/Frameworks/Tcl.framework/Headers  -D_REENTRANT=1
-D_THREAD_SAFE=1  -D_DARWIN_C_SOURCE=1
Linking: gcc   -L.        -g -L/opt/local/lib -L/usr/local/lib -o vim    -lXt
-lX11 -lSM -lICE -lncurses  -framework Cocoa    -L/opt/local/lib
-fstack-protector  -L/opt/local/lib/perl5/5.12.3/darwin-multi-2level/CORE -lperl
-lm -lutil -lc -framework Python  -F/System/Library/Frameworks -framework Tcl
-framework CoreFoundation -lruby

VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Mar 11 2012 01:07:11)
MacOS X (unix) version
Included patches: 1-456
Modified by Ben Schmidt et al.
Compiled by [hidden email]
Huge version with MacVim GUI.  Features included (+) or not (-):
+arabic +autocmd +balloon_eval +browse ++builtin_terms +byte_offset +cindent
+clientserver +clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
+conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con_gui +diff
+digraphs +dnd -ebcdic +emacs_tags +eval +ex_extra +extra_search +farsi
+file_in_path +find_in_path +float +folding -footer +fork() +fullscreen -gettext
  -hangul_input -iconv +insert_expand +jumplist +keymap +langmap +libcall
+linebreak +lispindent +listcmds +localmap -lua +menu +mksession +modify_fname
+mouse +mouseshape +mouse_dec -mouse_gpm -mouse_jsbterm +mouse_netterm
-mouse_sysmouse +mouse_xterm +mouse_urxvt +multi_byte +multi_lang -mzscheme
+netbeans_intg +odbeditor +path_extra +perl +persistent_undo +postscript
+printer +profile +python -python3 +quickfix +reltime +rightleft +ruby
+scrollbind +signs +smartindent -sniff +startuptime +statusline -sun_workshop
+syntax +tag_binary +tag_old_static -tag_any_white +tcl +terminfo +termresponse
+textobjects +title +toolbar +transparency +user_commands +vartabs +vertsplit
+virtualedit +visual +visualextra +viminfo +vreplace +wildignore +wildmenu
+windows +writebackup -X11 -xfontset +xim -xsmp -xterm_clipboard -xterm_save
    system vimrc file: "$VIM/vimrc"
      user vimrc file: "$HOME/.vimrc"
       user exrc file: "$HOME/.exrc"
   system gvimrc file: "$VIM/gvimrc"
     user gvimrc file: "$HOME/.gvimrc"
     system menu file: "$VIMRUNTIME/menu.vim"
   fall-back for $VIM: "/Applications/MacVim.app/Contents/Resources/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_MACVIM -Wall
-Wno-unknown-pragmas -pipe  -DMACOS_X_UNIX -no-cpp-precomp  -g -O2
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
-I/System/Library/Frameworks/Tcl.framework/Headers  -D_REENTRANT=1
-D_THREAD_SAFE=1  -D_DARWIN_C_SOURCE=1
Linking: gcc   -L.    -L.        -L/opt/local/lib -L/usr/local/lib -o Vim
-framework Cocoa -framework Carbon      -lncurses  -framework Cocoa
-L/opt/local/lib  -fstack-protector
-L/opt/local/lib/perl5/5.12.3/darwin-multi-2level/CORE -lperl -lm -lutil -lc
-framework Python  -F/System/Library/Frameworks -framework Tcl -framework
CoreFoundation -framework Ruby

What do you get for :version (use :redir to capture it)?

> The second problem was the titular bug in the experimental renderer;
> this bug (the dropping of the umlaut) only manifests itself when the
> :!ls procedure is used (and only with some fonts).  That is, this second
> bug only appears when MacVim is putting chars in the buffer the right
> way.  This experimental rendering bug has been masked by MacVim making
> glob() do something different from :!ls.

This seems like a much bigger problem to me. The other issue may be
inconvenient, but there's nothing actually wrong. In both cases, the
filename is validly represented. However, this issue affects any time
Unicode text with composing characters is edited, and the display is
incorrect as it doesn't represent the contents of the buffer accurately.

> It may be worth noting that the two methods of putting 'föo' in the
> buffer, i.e., !ls and glob(), affect certain VimScript functions like
> strlen() and strchars()---though I can't recall which did what.  One or
> both of those Vim functions returned different values depending on
> whether its input came from glob() or from !ls.

That is expected.

Cheers,

Ben.



--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Ben Schmidt]

>> There are two "problems" here. The big problem is that 'ga' over the
>> umlauted 'o' yields different values when the characters entered the
>> buffer because of glob() than when the chars entered the buffer
>> because of !ls. True console vim has their values as identical, and
>> therefore if at all possible so should MacVim; also true console vim
>> has them both behaving the way :!ls does, and therefore MacVim should
>> too if at all possible. Summary: MacVim's glob() is not doing what it
>> should, where 'should' is understood as 'doing things the way true
>> vim does'.
>
> This is indeed interesting. It is not a console Vim vs. MacVim issue,
> though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470
> I compiled myself. I don't know whether it's something about the way
> it was compiled or the version difference that's altered things.

I have tracked down what this is.

MacVim is compiled with Darwin support, but Apple's Vim is not (go
figure). That means that MacVim calls mac_precompose_path() while
evaluating glob(), but Apple's Vim doesn't. This function "converts a
decomposed HFS+ UTF-8 path to precomposed UTF-8". This explains why
there is a difference between glob() and :r !ls in MacVim but not in
Apple's Vim: MacVim does some extra processing for glob().

MacVim won't compile with --disable-darwin, though that may be more
because of the logic in configure than because it would actually cause
problems with the codebase. So we are stuck with this for now.

However, it does raise some questions.

- Why was it ever thought necessary to do that conversion?
- Is it still necessary?
- Why doesn't Apple's Vim have support for their operating system
   compiled in?

Ben.



--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Björn Winckler]

On Thu, Jul 19, 2012 at 2:31 PM, Ben Schmidt wrote:

>>> There are two "problems" here. The big problem is that 'ga' over the
>>> umlauted 'o' yields different values when the characters entered the
>>> buffer because of glob() than when the chars entered the buffer
>>> because of !ls. True console vim has their values as identical, and
>>> therefore if at all possible so should MacVim; also true console vim
>>> has them both behaving the way :!ls does, and therefore MacVim should
>>> too if at all possible. Summary: MacVim's glob() is not doing what it
>>> should, where 'should' is understood as 'doing things the way true
>>> vim does'.
>>
>>
>> This is indeed interesting. It is not a console Vim vs. MacVim issue,
>> though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470
>> I compiled myself. I don't know whether it's something about the way
>> it was compiled or the version difference that's altered things.
>
>
> I have tracked down what this is.
>
> MacVim is compiled with Darwin support, but Apple's Vim is not (go
> figure). That means that MacVim calls mac_precompose_path() while
> evaluating glob(), but Apple's Vim doesn't. This function "converts a
> decomposed HFS+ UTF-8 path to precomposed UTF-8". This explains why
> there is a difference between glob() and :r !ls in MacVim but not in
> Apple's Vim: MacVim does some extra processing for glob().

Thanks for tracking this down.

Precomposing filenames is the "right thing" to do in this case.  In
order for Vim to function correctly filenames have to be precomposed.
I don't remember the exact details, but I think e.g. things like
comparing filenames fails if they are not precomposed.  However, and I
admittedly conflated these two issues in a previous post, this should
not affect rendering of text.

> MacVim won't compile with --disable-darwin, though that may be more
> because of the logic in configure than because it would actually cause
> problems with the codebase. So we are stuck with this for now.
>
> However, it does raise some questions.
>
> - Why was it ever thought necessary to do that conversion?
> - Is it still necessary?
> - Why doesn't Apple's Vim have support for their operating system
>   compiled in?

I have answered the first two above, the third I cannot answer.

So, to conclude and hopefully put this thread to rest:

- The experimental renderer has problems rendering decomposed UTF

- Vim expects filenames to be precomposed UTF: glob() does the right
thing by precomposing, ":!ls" does not precompose since filenames are
stored in decomposed form in the filesystem (and we should not /
cannot change this).  If you are passing filenames to Vim (e.g. to
open a file) then they need to be precomposed.  The fact that console
Vim does not precompose UTF is actually a bug (in part due to Apple
not compiling Vim properly).


Björn

--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[dv1445]

Ben Schmidt, on 07/19/12 at 22:31:49 +1000, wrote:
> >This is indeed interesting. It is not a console Vim vs. MacVim issue,
> >though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470
> >I compiled myself. I don't know whether it's something about the way
> >it was compiled or the version difference that's altered things.
>
> I have tracked down what this is.

Nice work!
 
> MacVim is compiled with Darwin support, but Apple's Vim is not (go
> figure). That means that MacVim calls mac_precompose_path() while
> evaluating glob(), but Apple's Vim doesn't. This function "converts a
> decomposed HFS+ UTF-8 path to precomposed UTF-8". This explains why
> there is a difference between glob() and :r !ls in MacVim but not in
> Apple's Vim: MacVim does some extra processing for glob().

Ah, my "true console" vim isn't the factory one from Apple, but compiled
myself with a flag of ... sigh ... --disable-darwin!  

As to why this flag is in my bash alias for running ./configure for
console vim, I can no longer recall; I know that I *had* a reason, and I
*think* it had something to do with not wanting certain hard-wired
mapping-related stuff that --enable-darwin stuck in there.

Since --enable-darwin is really the way to go, for precomposing reasons
as Björn pointed out, I will try to find out if my reason was actually
any good.  If it was, perhaps we can fiddle with what --enable-darwin
does.

Prediction: my reason was not any good. :)

-gmn

--
You received this message from the "vim_mac" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php