Re: Solution for the problem of gvimext in UTF-8 locale

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Solution for the problem of gvimext in UTF-8 locale

Bram Moolenaar


[this message was stuck in my outbox for a few weeks, sorry]

Rainux wrote:

> When using gvimext in UTF-8 locale, e.g. zh_CN.UTF-8, it can not
> display the context menu item correctly, simply because the
> libintl.dll in official dist is built with MS VC, so it can not
> automatically convert the output message to the current code page of
> Windows.
>
> To solve this problem, just replace the linbintl.dll by the one on
> GnuWin32 page (http://gnuwin32.sourceforge.net/packages/libintl.htm).
>
> Hope this can be done officially, thanks. :p

I'm very careful with replacing things like this.  The page you refer to
mentions requiring libiconv too.  I would prefer a solution that uses
the native Win32 conversion, so that you don't need yet another .dll.
I think libiconv.dll is quite big.

--
Back up my hard drive?  I can't find the reverse switch!

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Solution for the problem of gvimext in UTF-8 locale

MURAOKA Taro

Vim loads libintl.dll dynamically, so we can try multiple
implementations of libintl.dll easily.

In attached patch, it try to load libintl2.dll (which requres
libiconv.dll too) first, when it was failed, try to load libintl.dll
(not require libiconv.dll).  After applied this patch, we would
distribute libintl2.dll and libiconv.dll separately from vim as an
"option".
--
MURAOKA Taro <[hidden email]>


diff -c ./src/os_win32.c.orig ./src/os_win32.c
*** ./src/os_win32.c.orig Mon Jul 30 06:04:40 2007
--- ./src/os_win32.c Mon Jul 30 06:15:57 2007
***************
*** 248,253 ****
--- 248,256 ----
  }

  #if defined(DYNAMIC_GETTEXT) || defined(PROTO)
+ # ifndef GETTEXT_DLL_WITHENC
+ #  define GETTEXT_DLL_WITHENC "libintl2.dll"
+ # endif
  # ifndef GETTEXT_DLL
  #  define GETTEXT_DLL "libintl.dll"
  # endif
***************
*** 285,291 ****
      if (hLibintlDLL)
  return 1;
      /* Load gettext library (libintl.dll) */
!     hLibintlDLL = LoadLibrary(libname != NULL ? libname :
GETTEXT_DLL);
      if (!hLibintlDLL)
      {
  char_u    dirname[_MAX_PATH];
--- 288,311 ----
      if (hLibintlDLL)
  return 1;
      /* Load gettext library (libintl.dll) */
!     /*
!      * Priority:
!      * 1. libname if it isn't NULL.
!      * 2. GETTEXT_DLL_WITHENC to convert encodings.
!      * 3. GETTEXT_DLL to use message in only native encoding.
!      */
!     if (libname != NULL)
!     {
! hLibintlDLL = LoadLibrary(libname);
! /* If failed to load libname, try to load default libraries. */
!     }
!     if (hLibintlDLL == NULL)
!     {
! hLibintlDLL = LoadLibrary(GETTEXT_DLL_WITHENC);
! if (hLibintlDLL == NULL)
!    hLibintlDLL = LoadLibrary(GETTEXT_DLL);
!     }
!     /* Get functions of gettext library. */
      if (!hLibintlDLL)
      {
  char_u    dirname[_MAX_PATH];


--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Solution for the problem of gvimext in UTF-8 locale

Rainux
In reply to this post by Bram Moolenaar

I have received this message before you resend it :p
But seems my reply to it haven't delivered.

I think Vim always need libiconv (iconv.dll or libiconv.dll) when
compiled with ICONV=yes, and I just remember Vim 6.3 can not handle
ucs-2le files correctly without libiconv in some environment, but
seems now both 6.3 and 7.1 can works correctly without libiconv on my
Windows XP Professional (Simplified Chinese version), I just be
confused.

On 7/29/07, Bram Moolenaar <[hidden email]> wrote:

>
> [this message was stuck in my outbox for a few weeks, sorry]
>
> Rainux wrote:
>
> > When using gvimext in UTF-8 locale, e.g. zh_CN.UTF-8, it can not
> > display the context menu item correctly, simply because the
> > libintl.dll in official dist is built with MS VC, so it can not
> > automatically convert the output message to the current code page of
> > Windows.
> >
> > To solve this problem, just replace the linbintl.dll by the one on
> > GnuWin32 page (http://gnuwin32.sourceforge.net/packages/libintl.htm).
> >
> > Hope this can be done officially, thanks. :p
>
> I'm very careful with replacing things like this.  The page you refer to
> mentions requiring libiconv too.  I would prefer a solution that uses
> the native Win32 conversion, so that you don't need yet another .dll.
> I think libiconv.dll is quite big.
>
> --
> Back up my hard drive?  I can't find the reverse switch!
>
>  /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
> ///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\        download, build and distribute -- http://www.A-A-P.org        ///
>  \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///
>


--
Best Regards

Rainux

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Solution for the problem of gvimext in UTF-8 locale

Bram Moolenaar
In reply to this post by MURAOKA Taro


Taro Muraoka wrote:

> Vim loads libintl.dll dynamically, so we can try multiple
> implementations of libintl.dll easily.
>
> In attached patch, it try to load libintl2.dll (which requres
> libiconv.dll too) first, when it was failed, try to load libintl.dll
> (not require libiconv.dll).  After applied this patch, we would
> distribute libintl2.dll and libiconv.dll separately from vim as an
> "option".

The libiconv.dll is big.  I noticed one that is almost 1 Mbyte.  While
Win32 has native support for encoding conversions, thus mostly there is
no reason to use libiconv.dll.  Conversions not natively supported are
likely to have problems anyway.

The libintl.dll that's included with the big self-installing Vim is a
bit old.  I hesitate to replace it, because it should work with all
different compilers.

Is the problem that the ports of GNU gettext to Win32 have not been
modified to use the Win32 API?

--
hundred-and-one symptoms of being an internet addict:
56. You leave the modem speaker on after connecting because you think it
    sounds like the ocean wind...the perfect soundtrack for "surfing the net".

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Solution for the problem of gvimext in UTF-8 locale

MURAOKA Taro

Hi Bram.


I see 3 items to think in your reply:
  1. Should vim continue to use iconv.dll?
  2. Is the newer libintl.dll portable?
  3. Can we modify gettext to use Windows native converter?
I'll answer one by one.


= 1. Should vim continue to use iconv.dll? =

YES.  Count of encodings which Windows support is different depends on
edition or installed other modules.  Ex:Vista has large number of
encodings support, but 2000 has fewer.  So it may become hard to
warrant
and provide enough number of encodings to real users who need encoding
conversion.


= 2. Is the newer libintl.dll portable? =

YES.  I have tried to compile a newer gettext (0.14.2) by MSVC 7.1.
And it have succeeded.


= 3. Can we modify gettext to use Windows native converter? =

YES.  It will be a work to implement another iconv implementation by
using Windows native encoding converter.  And vim will become a
important example at the moment, to creating a patch.
BUT, I think it is not easy to be applied the patch by the gettext
maintainer.  If it isn't applied, we must maintain and release
branched gettext codes.
--
MURAOKA Taro <[hidden email]>


On 7月31日, 午前5:11, Bram Moolenaar <[hidden email]> wrote:

> Taro Muraoka wrote:
> > Vim loads libintl.dll dynamically, so we can try multiple
> > implementations of libintl.dll easily.
>
> > In attached patch, it try to load libintl2.dll (which requres
> > libiconv.dll too) first, when it was failed, try to load libintl.dll
> > (not require libiconv.dll).  After applied this patch, we would
> > distribute libintl2.dll and libiconv.dll separately from vim as an
> > "option".
>
> The libiconv.dll is big.  I noticed one that is almost 1 Mbyte.  While
> Win32 has native support for encoding conversions, thus mostly there is
> no reason to use libiconv.dll.  Conversions not natively supported are
> likely to have problems anyway.
>
> The libintl.dll that's included with the big self-installing Vim is a
> bit old.  I hesitate to replace it, because it should work with all
> different compilers.
>
> Is the problem that the ports of GNU gettext to Win32 have not been
> modified to use the Win32 API?
>
> --
> hundred-and-one symptoms of being an internet addict:
> 56. You leave the modem speaker on after connecting because you think it
>     sounds like the ocean wind...the perfect soundtrack for "surfing the net".
>
>  /// Bram Moolenaar -- [hidden email] --http://www.Moolenaar.net  \\\
> ///        sponsor Vim, vote for features --http://www.Vim.org/sponsor/\\\
> \\\        download, build and distribute --http://www.A-A-P.org       ///
>  \\\            help me help AIDS victims --http://ICCF-Holland.org   ///


--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Solution for the problem of gvimext in UTF-8 locale

Rainux

Wonderful answer!

Yes, I think the problem that Vim can not handle ucs-2le encoding file
I've encoutered, is produced by the problem of native encoding
conversion in Win32.

On 7/31/07, MURAOKA Taro <[hidden email]> wrote:

>
> Hi Bram.
>
>
> I see 3 items to think in your reply:
>   1. Should vim continue to use iconv.dll?
>   2. Is the newer libintl.dll portable?
>   3. Can we modify gettext to use Windows native converter?
> I'll answer one by one.
>
>
> = 1. Should vim continue to use iconv.dll? =
>
> YES.  Count of encodings which Windows support is different depends on
> edition or installed other modules.  Ex:Vista has large number of
> encodings support, but 2000 has fewer.  So it may become hard to
> warrant
> and provide enough number of encodings to real users who need encoding
> conversion.
>
>
> = 2. Is the newer libintl.dll portable? =
>
> YES.  I have tried to compile a newer gettext (0.14.2) by MSVC 7.1.
> And it have succeeded.
>
>
> = 3. Can we modify gettext to use Windows native converter? =
>
> YES.  It will be a work to implement another iconv implementation by
> using Windows native encoding converter.  And vim will become a
> important example at the moment, to creating a patch.
> BUT, I think it is not easy to be applied the patch by the gettext
> maintainer.  If it isn't applied, we must maintain and release
> branched gettext codes.
> --
> MURAOKA Taro <[hidden email]>
>
>
> On 7月31日, 午前5:11, Bram Moolenaar <[hidden email]> wrote:
> > Taro Muraoka wrote:
> > > Vim loads libintl.dll dynamically, so we can try multiple
> > > implementations of libintl.dll easily.
> >
> > > In attached patch, it try to load libintl2.dll (which requres
> > > libiconv.dll too) first, when it was failed, try to load libintl.dll
> > > (not require libiconv.dll).  After applied this patch, we would
> > > distribute libintl2.dll and libiconv.dll separately from vim as an
> > > "option".
> >
> > The libiconv.dll is big.  I noticed one that is almost 1 Mbyte.  While
> > Win32 has native support for encoding conversions, thus mostly there is
> > no reason to use libiconv.dll.  Conversions not natively supported are
> > likely to have problems anyway.
> >
> > The libintl.dll that's included with the big self-installing Vim is a
> > bit old.  I hesitate to replace it, because it should work with all
> > different compilers.
> >
> > Is the problem that the ports of GNU gettext to Win32 have not been
> > modified to use the Win32 API?
> >
> > --
> > hundred-and-one symptoms of being an internet addict:
> > 56. You leave the modem speaker on after connecting because you think it
> >     sounds like the ocean wind...the perfect soundtrack for "surfing the net".
> >
> >  /// Bram Moolenaar -- [hidden email] --http://www.Moolenaar.net  \\\
> > ///        sponsor Vim, vote for features --http://www.Vim.org/sponsor/\\\
> > \\\        download, build and distribute --http://www.A-A-P.org       ///
> >  \\\            help me help AIDS victims --http://ICCF-Holland.org   ///
>
>
> >
>


--
Best Regards

Rainux

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Solution for the problem of gvimext in UTF-8 locale

Bram Moolenaar
In reply to this post by MURAOKA Taro


Taro Muraoka wrote:

> I see 3 items to think in your reply:
>   1. Should vim continue to use iconv.dll?
>   2. Is the newer libintl.dll portable?
>   3. Can we modify gettext to use Windows native converter?
> I'll answer one by one.
>
>
> = 1. Should vim continue to use iconv.dll? =
>
> YES.  Count of encodings which Windows support is different depends on
> edition or installed other modules.  Ex:Vista has large number of
> encodings support, but 2000 has fewer.  So it may become hard to
> warrant
> and provide enough number of encodings to real users who need encoding
> conversion.

We rarely remove existing support.  However, keep in mind that iconv is
loaded dynamically, it's only really loaded when the user has it
installed in the path.  It's not included in the Vim distribution, so
it's actally rare that people use it.  Most people rely on the Windows
conversion functionality.  But you can add the iconv library when you
need it.

> = 2. Is the newer libintl.dll portable? =
>
> YES.  I have tried to compile a newer gettext (0.14.2) by MSVC 7.1.
> And it have succeeded.

I wasn't talking about the source code being portable.  I was wondering
if the binary libintl.dll is portable.  That means it works no matter
with which compiler Vim was compiled.  And it doesn't require any other
library.  That's how it works at the moment.  In the past there have
been problems that a dll compiled with compiler X does not work with Vim
compiled with compiler Y.  That defeats the idea of independent
libraries.

> = 3. Can we modify gettext to use Windows native converter? =
>
> YES.  It will be a work to implement another iconv implementation by
> using Windows native encoding converter.  And vim will become a
> important example at the moment, to creating a patch.
> BUT, I think it is not easy to be applied the patch by the gettext
> maintainer.  If it isn't applied, we must maintain and release
> branched gettext codes.

So the libintl.dll that supports more encodings requires iconv, it
doesn't use the Windows native conversions?  And if we want it otherwise
we need to build our own libintl.dll?

It's not surprising, actually, that people somehow accept having to
install a 1 Mbyte library just to make a 50 Kbyte library work (and then
probably not use the features that the 1 Mbyte library offers).  It
sounds like work was saved by not converting the library to MS-Windows.

I think the best solution would be that gettext (libintl.dll) tries to
dynamically load iconv, then when that fails falls back to using the
Windows native conversion functions.  The first part should be
relatively easy to implement, then we at least have a backwards
compatible version of libintl.dll, not requiring the 1 Mbyte
libiconv.dll (and people who need the iconv functionality can get it).
The second part might be a bit more work.

--
hundred-and-one symptoms of being an internet addict:
66. You create a homepage with the impression to cure the afflicted...but
    your hidden agenda is to receive more e-mail.

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Solution for the problem of gvimext in UTF-8 locale

Bram Moolenaar
In reply to this post by Rainux


Rainux wrote:

> Wonderful answer!
>
> Yes, I think the problem that Vim can not handle ucs-2le encoding file
> I've encoutered, is produced by the problem of native encoding
> conversion in Win32.

What is your 'encoding' set to then?  Win32 has reasonable Unicode
support, and Vim does some conversion itself too.  Thus this would only
fail if your 'encoding' is set to something that MS-Windows doesn't
support.

It might be that your 'fileencodings' setting has a wrong value.

--
hundred-and-one symptoms of being an internet addict:
70. ISDN lines are added to your house on a hourly basis

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Solution for the problem of gvimext in UTF-8 locale

Rainux

This is the relative settings in my .vimrc

   set fileencodings=ucs-bom,utf-8,chinese
   set encoding=utf-8

I'm sorry now I can not recur that problem, it not occur on all Win32
environment.

Sorry for the repeat, I just forgot to "Reply to all".

On 8/1/07, Bram Moolenaar <[hidden email]> wrote:

>
> Rainux wrote:
>
> > Wonderful answer!
> >
> > Yes, I think the problem that Vim can not handle ucs-2le encoding file
> > I've encoutered, is produced by the problem of native encoding
> > conversion in Win32.
>
> What is your 'encoding' set to then?  Win32 has reasonable Unicode
> support, and Vim does some conversion itself too.  Thus this would only
> fail if your 'encoding' is set to something that MS-Windows doesn't
> support.
>
> It might be that your 'fileencodings' setting has a wrong value.
>
> --
> hundred-and-one symptoms of being an internet addict:
> 70. ISDN lines are added to your house on a hourly basis
>
>  /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
> ///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\        download, build and distribute -- http://www.A-A-P.org        ///
>  \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///
>


--
Best Regards

Rainux

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_multibyte" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---