Re: Python IDE-ish features (was: Vim 7.0 sandbox changes break a lot of my configuration)

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Python IDE-ish features (was: Vim 7.0 sandbox changes break a lot of my configuration)

G. Sumner Hayes
Adding vim-dev back in, I accidentally stripped it
before--for those who missed it, the list of python
support features I'm working on is at the end of this
message.

Bram Moolenaar <[hidden email]> wrote:
> G. Sumner Hayes wrote:
> > If there were some programmatic way to find what
> > file/line a tag matched, I could avoid
> > that--e.g.exposing the :tselect menu as a list
> > (preferrably with line numbers as well),  with a
> > taglist("tagname") function returning that list.
>
> Why can't you use the taglist() function?

Ahh!  Didn't catch that addition (I just started
toying with Vim 7 yesterday).

If it could add line numbers that'd be ideal.  But I
understand why _not_ to do that (it entails opening up
all those files to find the lines).  

Since the ex cmds tend to just be "/regex/" I can just
do a regex search from within Python (theoretically it
can be any ex command, right?  But ctags seems to only
use /, at least for my tags files--supporting that and
? is easy enough).

> Very good.  Did you have a look at Agide
> (www.agide.org)?

Not really, I looked at the web page once when it was
mentioned here and saw "NOTE: Agide is still in the
development phase: Only a few things work properly."
Which made me reluctant to install right now.

I figured I'd just do some work with what I have for
now, which is all inside of vim and shouldn't
duplicate agide's work much. I'll certainly take
another look before I get to the point of duplicating
features that Agide claims to support.

> Hmm, perhaps you can have a look at making omni
completion
> work for Python.  It's one of the things on my list.
 Python
> would be the first where running the interpreter
could help to
> provide information about things that can be
completed.  Should
> actually be very well possible, since Python exposes
nearly
> everything in dictionaries.

Short answer: yeah, I'm interested in doing this.
It's just not an easy task even with the embedded
interpreter, so it'll be punted to later in the cycle
(especially since in practice, completing on tags is
generally good enough for me personally).

Long answer: I've thought about it.  In general, it's
a hard problem for Python (and other dynamically typed
languages).
For instance, if I'm in a function:
cmp(x, y):
    if x.^X^O

and I try to complete, I have no idea what the types
of x and y are.  Indeed, one place in the code might
call cmp with x and y as DollarValues and another
might call it with TestScores (and this kind of
automatic polymorphism is a strong suit of languages
like Python, Ruby, Smalltalk, etc).

Or I could have:
aClass = loadClass("happyObject")
a=aClass()
a.^X^O

Same idea; loadClass could return any class at all.

Now, for some things it _might_ be more obvious.  But
a reasonable solution would mean parsing the entire
project to see what values are actually being passed
around and doing some amount of type propogation.
Probably means a ctags-style type inference engine,
and even then it's not perfect in cases like
input = sys.read()
a = exec(input)

where you simply can't know the type of a statically.
But a 95% solution is certainly doable and IDEs like
Wing attempt it, so it's worth looking into.


Not snipping the following since I didn't send it to
the list before:
> > Having an embedded interpreter makes this kind of
> > support for dynamic languages much easier to drop
in.
> > Among the features I have "kind of working" are:
> > 1. Help functionality, so if I type "cmp(" then
the
> > status bar (while I'm still inserting) displays
> > "cmp(x, y) Compare the two objects X and Y and
return
> > an integer according to the outcome." and hitting
F1
> > displays the full help text.  This pulls from the
> > Python docs for core functions, and pulls function
> > definitions, doc strings, and preceding comments
for
> > code that's in the current source tree.
> > 2. Syntax checking, so if I type "if a=1:" and hit
> > enter then I get an error message saying it's a
syntax
> > error (just like C you need == in comparisons).
> > Ideally I'll highlight errors with undercurls
> > eventually.
> > 3. Class browsing stuff; I have a GUI menu showing
the
> > parent class, child classes, and methods of
whatever
> > class I'm in (and selecting any of those from the
menu
> > jumps there).
>
> > I'd like to get those working and a pdb (debugger)
> > integration with ballon expressions and package it
up
> > as an easy drop-in plugin (open source).
>


               
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

Reply | Threaded
Open this post in threaded view
|

[PATCH] Python interface support for evals that return dict/list

G. Sumner Hayes
"G. Sumner Hayes" <[hidden email]> wrote:
> Bram Moolenaar <[hidden email]> wrote:
> > Why can't you use the taglist() function?
>
> Ahh!  Didn't catch that addition (I just started
> toying with Vim 7 yesterday).

Problem: the python interface doesn't support lists or
dicts, so vim.eval("taglist('foo')") fails.

I've attached a proof-of-concept patch against that
(not
ready for general use, see problem note).  Apologies
if the Python interface maintainer already has
something in the works.

Essentially, rather than using eval_to_str for
Python's vim.eval, it'll use eval_expr.  If that
returns a string or a number, it returns a Python
string
(using a Python integer for
numbers would probably break backward compatibility).
If it returns a dict or list, it'll recursively build
up a python dict or list.

PROBLEM: it doesn't free up the list returned; I'm
very unclear on how the memory management in vim
works.

It looks like the dict_free/list_free helper functions
from eval.c would be handy for this, but they're
static; any chance they'll be externed?

I already pilfered the first 4 lines of the patch from
eval.c, I'd be happier seeing them moved out to a
commonly available place than doing copy-and-paste
coding.


               
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

--- src/if_python.c 2005-10-13 14:43:52.000000000 -0400
+++ ../vim/src/if_python.c 2006-01-05 19:26:34.408749540 -0500
@@ -1082,6 +1082,59 @@
     return result;
 }
 
+/*
+ * In a hashtab item "hi_key" points to "di_key" in a dictitem.
+ * This avoids adding a pointer to the hashtab item.
+ * DI2HIKEY() converts a dictitem pointer to a hashitem key pointer.
+ * HIKEY2DI() converts a hashitem key pointer to a dictitem pointer.
+ * HI2DI() converts a hashitem pointer to a dictitem pointer.
+ */
+static dictitem_T dumdi;
+#define DI2HIKEY(di) ((di)->di_key)
+#define HIKEY2DI(p)  ((dictitem_T *)(p - (dumdi.di_key - (char_u *)&dumdi)))
+#define HI2DI(hi)     HIKEY2DI((hi)->hi_key)
+
+    static PyObject *
+VimToPython(typval_T *our_tv)
+{
+    PyObject *result;
+    if (our_tv->v_type == VAR_STRING) {
+        result = Py_BuildValue("s", our_tv->vval.v_string);
+    } else if (our_tv->v_type == VAR_NUMBER) {
+        char buf[NUMBUFLEN];
+        sprintf(buf, "%ld", (long)our_tv->vval.v_number);
+        result = Py_BuildValue("s", buf);
+    } else if (our_tv->v_type == VAR_LIST){
+        int i=0;
+        struct listvar_S *list = our_tv->vval.v_list;
+        listitem_T *curr = list->lv_first;
+        result = PyList_New(0);
+        for(i=0; i<list->lv_len; i++, curr = curr->li_next) {
+            PyList_Append(result, VimToPython(&curr->li_tv));
+        }
+        //Free list here
+
+    } else if (our_tv->v_type = VAR_DICT) {
+        result = PyDict_New();
+        int i=0;
+        dictitem_T *di;
+        hashtab_T ht = our_tv->vval.v_dict->dv_hashtab;
+        hashitem_T *hi = ht.ht_array;
+        if (hi==NULL) hi = ht.ht_smallarray;
+        for(i=0; i<ht.ht_mask+1; i++) {
+            if(hi->hi_key && hi->hi_key != HI_KEY_REMOVED) {
+                di = HI2DI(hi);
+                PyDict_SetItemString(result, hi->hi_key, VimToPython(&di->di_tv));
+
+            }
+            hi++;
+        }
+        //Free dict here
+    } else {
+        result = Py_BuildValue("s", "not string");
+    }
+    return result;
+}
 /*ARGSUSED*/
     static PyObject *
 VimEval(PyObject *self, PyObject *args)
@@ -1089,6 +1142,7 @@
 #ifdef FEAT_EVAL
     char *expr;
     char *str;
+    typval_T   *our_tv;
     PyObject *result;
 
     if (!PyArg_ParseTuple(args, "s", &expr))
@@ -1096,21 +1150,27 @@
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    str = (char *)eval_to_string((char_u *)expr, NULL);
+    //str = (char *)eval_to_string((char_u *)expr, NULL);
+    our_tv = eval_expr((char_u *) expr, NULL);
+
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
-    if (str == NULL)
+    if (our_tv == NULL)
     {
  PyErr_SetVim(_("invalid expression"));
  return NULL;
-    }
+    }
+
+    result = VimToPython(our_tv);
 
-    result = Py_BuildValue("s", str);
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    vim_free(str);
+    if (our_tv->v_type == VAR_STRING) {
+        vim_free(our_tv->vval.v_string);
+    }
+    vim_free(our_tv);
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
Reply | Threaded
Open this post in threaded view
|

Re: Python IDE-ish features (was: Vim 7.0 sandbox changes break a lot of my configuration)

Bram Moolenaar
In reply to this post by G. Sumner Hayes

G. Sumner Hayes wrote:

> > Very good.  Did you have a look at Agide
> > (www.agide.org)?
>
> Not really, I looked at the web page once when it was
> mentioned here and saw "NOTE: Agide is still in the
> development phase: Only a few things work properly."
> Which made me reluctant to install right now.

It's true that only a few things got implemented.  But it's useful, I am
using it for Vim debugging all the time.  One current problem is that
the gdb window doesn't scroll up automatically.  It stopped working
after installing a new version of WxPython...

> I figured I'd just do some work with what I have for
> now, which is all inside of vim and shouldn't
> duplicate agide's work much. I'll certainly take
> another look before I get to the point of duplicating
> features that Agide claims to support.

Please do so.  Comments are welcome.

> > Hmm, perhaps you can have a look at making omni completion
> > work for Python.  It's one of the things on my list.  Python
> > would be the first where running the interpreter could help to
> > provide information about things that can be completed.  Should
> > actually be very well possible, since Python exposes nearly
> > everything in dictionaries.
>
> Short answer: yeah, I'm interested in doing this.
> It's just not an easy task even with the embedded
> interpreter, so it'll be punted to later in the cycle
> (especially since in practice, completing on tags is
> generally good enough for me personally).
>
> Long answer: I've thought about it.  In general, it's
> a hard problem for Python (and other dynamically typed
> languages).
> For instance, if I'm in a function:
> cmp(x, y):
>     if x.^X^O
>
> and I try to complete, I have no idea what the types
> of x and y are.  Indeed, one place in the code might
> call cmp with x and y as DollarValues and another
> might call it with TestScores (and this kind of
> automatic polymorphism is a strong suit of languages
> like Python, Ruby, Smalltalk, etc).

In this case you simply don't know what the types are.  That's the
disadvantage of a runtime typed language like Python.  I wouldn't even
try to do clever completion for this.

> Or I could have:
> aClass = loadClass("happyObject")
> a=aClass()
> a.^X^O
>
> Same idea; loadClass could return any class at all.
>
> Now, for some things it _might_ be more obvious.  But
> a reasonable solution would mean parsing the entire
> project to see what values are actually being passed
> around and doing some amount of type propogation.
> Probably means a ctags-style type inference engine,
> and even then it's not perfect in cases like
> input = sys.read()
> a = exec(input)
>
> where you simply can't know the type of a statically.
> But a 95% solution is certainly doable and IDEs like
> Wing attempt it, so it's worth looking into.

I don't think users expect the omni completion to be that clever.  Just
make it do what can obviously be known about what's in front of the
cursor.  For example with names of modules and functions they provide.
Thus when I type "string." it completes what the string module contains.

--
How To Keep A Healthy Level Of Insanity:
10. Ask people what sex they are. Laugh hysterically after they answer.

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://www.ICCF.nl         ///
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

Bram Moolenaar
In reply to this post by G. Sumner Hayes

G. Sumner Hayes wrote:

> "G. Sumner Hayes" <[hidden email]> wrote:
> > Bram Moolenaar <[hidden email]> wrote:
> > > Why can't you use the taglist() function?
> >
> > Ahh!  Didn't catch that addition (I just started
> > toying with Vim 7 yesterday).
>
> Problem: the python interface doesn't support lists or
> dicts, so vim.eval("taglist('foo')") fails.

Right, this wasn't supported yet.

> I've attached a proof-of-concept patch against that (not
> ready for general use, see problem note).  Apologies
> if the Python interface maintainer already has
> something in the works.
>
> Essentially, rather than using eval_to_str for
> Python's vim.eval, it'll use eval_expr.  If that
> returns a string or a number, it returns a Python
> string (using a Python integer for
> numbers would probably break backward compatibility).
> If it returns a dict or list, it'll recursively build
> up a python dict or list.

Great!  Please try it out a bit more, I'll include it later.

> PROBLEM: it doesn't free up the list returned; I'm very unclear on how
> the memory management in vim works.
>
> It looks like the dict_free/list_free helper functions
> from eval.c would be handy for this, but they're
> static; any chance they'll be externed?

I'm trying to keep most manipulation with script variables local to
eval.c.  But it's starting to leak out, there are a few places in the
code where using lists is so handy.  And here you really have a place
where Vim and Python things are together.

I think we can make some functions in eval.c global.  But we should be
careful about the interface that we provide to the rest of Vim.
Avoiding too many structs to be moved from eval.c to structs.h.

> I already pilfered the first 4 lines of the patch from
> eval.c, I'd be happier seeing them moved out to a
> commonly available place than doing copy-and-paste
> coding.

Duplicating code is certainly a bad idea.

Since C itself doesn't support lists and dicts it's tempting to somehow
add this.  I find that sometimes it's easier to write Vim script than C
code to implement a feature...

--
How To Keep A Healthy Level Of Insanity:
11. Specify that your drive-through order is "to go".

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://www.ICCF.nl         ///
Reply | Threaded
Open this post in threaded view
|

Re: Python IDE-ish features (was: Vim 7.0 sandbox changes break a lot of my configuration)

Mikołaj Machowski
In reply to this post by G. Sumner Hayes
Dnia czwartek, 5 stycznia 2006 22:41, G. Sumner Hayes napisał:

> and I try to complete, I have no idea what the types
> of x and y are.  Indeed, one place in the code might
> call cmp with x and y as DollarValues and another
> might call it with TestScores (and this kind of
> automatic polymorphism is a strong suit of languages
> like Python, Ruby, Smalltalk, etc).
>
> Or I could have:
> aClass = loadClass("happyObject")
> a=aClass()
> a.^X^O
>
> Same idea; loadClass could return any class at all.
>
> Now, for some things it _might_ be more obvious.  But
> a reasonable solution would mean parsing the entire
> project to see what values are actually being passed
> around and doing some amount of type propogation.
> Probably means a ctags-style type inference engine,
> and even then it's not perfect in cases like
> input = sys.read()
> a = exec(input)

That could be partially solved by omnicompletion providing some kind of
structure. I was thinking about JavaScript but there you have very
little to differentiate between variable/object name, function name and
DOM elements. In some cases you can guess but not ever be near 95% you
mentioned.

If omnicompletion could take also Dictionaries as argument and present
them as structure...

m.

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

G. Sumner Hayes
In reply to this post by Bram Moolenaar
Note: I think I may have found a memory leak in vim7,
see below.

Bram Moolenaar <[hidden email]> wrote:
> G. Sumner Hayes wrote:
> > I've attached a proof-of-concept patch against
> > that (not ready for general use, see problem
note).
[SNIP]
> > PROBLEM: it doesn't free up the list returned; I'm
> > very unclear on how the memory management in vim
> > works.

> I'm trying to keep most manipulation with script
> variables local to eval.c.

After looking at it a bit more, it seems like the
function I need is exported already (clear_tv).

2 questions:
1. If I do clear_tv(my_tv), that should empty out any
lists, strings, dicts, etc within the tv, right?
2. assuming that my_tv was originally dynamically
allocated via "my_tv = (typval_t *)
alloc(sizeof(typval_T));" then I still need to call
vim_free(my_tv) after the clear_tv call, right?

If I'm right on both of those, then the last if block
in VimFree from my patch should be changed by
eliminating the "if(our_tv->v_type == VAL_STRING)"
block and replacing it with "clear_tv(our_tv);"

New diff attached.  I think this is ready for other
people to test (or to apply), as far as I can see the
memory management is okay now.  If someone else out
there knows the if_python code, reading my diff
wouldn't hurt.

Bram, feel free to use/distribute under the standard
vim license.

POSSIBLE MEMORY LEAK: quickfix.c at line 2998 clears a
tv that it got from eval_expr.  Since eval_expr
allocates the tv dynamically, I believe a line should
be added after that "vim_free(tv);".  Is that right?


               
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

G. Sumner Hayes
In reply to this post by G. Sumner Hayes
Bram Moolenaar <[hidden email]> wrote:
> > New diff attached.  I think this is ready for
> > other people to test (or to apply), as far as I
> > can see the memory management is okay now.  
> > If someone else out there knows the if_python
> > code, reading my diff wouldn't hurt.
>
> You didn't attach a patch...

Gah!  Here it is.  It uses clear_tv/vim_free, that
could easily be changed to free_tv when you make that global.


               
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

python_dict_list_v2.diff (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

Bram Moolenaar

G. Sumner Hayes wrote:

> Bram Moolenaar <[hidden email]> wrote:
> > > New diff attached.  I think this is ready for
> > > other people to test (or to apply), as far as I
> > > can see the memory management is okay now.  
> > > If someone else out there knows the if_python
> > > code, reading my diff wouldn't hurt.
> >
> > You didn't attach a patch...
>
> Gah!  Here it is.  It uses clear_tv/vim_free, that
> could easily be changed to free_tv when you make that global.

Very good.  So it's completely functional now, right?

I'll put it in the todo list to be included.  But perhaps you can fix
the formatting to match the Vim source code.  And I also need an update
for the documentation.

--
'Well, here's something to occupy you and keep your mind off things.'
'It won't work, I have an exceptionally large mind.'
                -- Douglas Adams, "The Hitchhiker's Guide to the Galaxy"

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://www.ICCF.nl         ///
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

G. Sumner Hayes
Bram Moolenaar <[hidden email]> wrote:
> Very good.  So it's completely functional now,
> right?

Yes, it looks functional to me.  The taglist function
returns a list of dictionaries keyed/valued by strings
and so seems to test every case except integers, which
I've tested independently (and they do come back as
strings, for backward compatibility).

I've also done things like:
  let a = {1:'one', 2:['two', 'three']}
  py print vim.eval('a')[2]  #and so forth
to test nesting dicts/lists in the other direction.

I haven't exactly tried throwing huge data structures
at it but I see no reason it shouldn't work.

> I'll put it in the todo list to be included.  But
> perhaps you can fix the formatting to match the Vim
> source code.  And I also need an update for the
> documentation.

I tried to fix formatting (mostly braces) to conform
with the rest of the vim code.  Attached.  Most
functions in that file have an /*ARGSUSED*/ comment
(probably for lint or similar) that I don't know the
exact meaning of and so did not duplicate.

Also gave a shot at documentation updates.  Attached.

Open issues:
1. What should we do if the eval doesn't return a
supported type (I return None, presumably a vim
exception will happen and I'm not sure that we want to
raise a Python exception.)
2. 4 lines are duplicated from eval.c, we might want
to move at least the #defines into a public header and
avoid duplication
3. I use clear_tv/vim_free(tv) to free memory.  You
may want to switch that to free_tv when you publish
that function.


               
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

--- /home/sumner/src/vim/runtime/doc/if_pyth.txt 2005-10-14 10:11:26.000000000 -0400
+++ runtime/doc/if_pyth.txt 2006-01-09 14:56:33.390853000 -0500
@@ -132,12 +132,22 @@
 vim.eval(str) *python-eval*
  Evaluates the expression str using the vim internal expression
  evaluator (see |expression|).  Returns the expression result as a
- string.
+ string if the Vim expression evaluates to a string or integer.
+        Returns the expression as a dictionary if the vim expression
+        evaluates to a Vim dictionary.  Returns the expression as a list
+        if the vim expression evaluates to a vim list.  Dictionaries and
+        lists are recursively expanded.
  Examples: >
     :py text_width = vim.eval("&tw")
     :py str = vim.eval("12+12") # NB result is a string! Use
  # string.atoi() to convert to
  # a number.
+            :py tagList = vim.eval('taglist("eval_expr")')
+        The latter will return a python list of python dicts, for instance:
+        [{'cmd': '/^eval_expr(arg, nextcmd)$/', 'static': 0, 'name':
+        'eval_expr', 'kind': 'f', 'filename': './src/eval.c'}]
+
+                                                      
 
 Error object of the "vim" module
 

--- src/if_python.c 2005-10-13 14:43:52.000000000 -0400
+++ ../vim/src/if_python.c 2006-01-09 15:15:41.240306605 -0500
@@ -1082,6 +1082,73 @@
     return result;
 }
 
+/*
+ * In a hashtab item "hi_key" points to "di_key" in a dictitem.
+ * This avoids adding a pointer to the hashtab item.
+ * DI2HIKEY() converts a dictitem pointer to a hashitem key pointer.
+ * HIKEY2DI() converts a hashitem key pointer to a dictitem pointer.
+ * HI2DI() converts a hashitem pointer to a dictitem pointer.
+ */
+static dictitem_T dumdi;
+#define DI2HIKEY(di) ((di)->di_key)
+#define HIKEY2DI(p)  ((dictitem_T *)(p - (dumdi.di_key - (char_u *)&dumdi)))
+#define HI2DI(hi)     HIKEY2DI((hi)->hi_key)
+
+/*
+ * Function to translate a typval_T into a PyObject; this will recursively
+ * translate lists/dictionaries into their Python equivalents.
+ */
+    static PyObject *
+VimToPython(typval_T *our_tv)
+{
+    PyObject *result;
+    if (our_tv->v_type == VAR_STRING)
+    {
+        result = Py_BuildValue("s", our_tv->vval.v_string);
+    }
+    else if (our_tv->v_type == VAR_NUMBER)
+    {
+        char buf[NUMBUFLEN];
+        sprintf(buf, "%ld", (long)our_tv->vval.v_number);
+        result = Py_BuildValue("s", buf);
+    }
+    else if (our_tv->v_type == VAR_LIST)
+    {
+        int i=0;
+        struct listvar_S *list = our_tv->vval.v_list;
+        listitem_T *curr = list->lv_first;
+        result = PyList_New(0);
+        for(i=0; i<list->lv_len; i++, curr = curr->li_next)
+        {
+            PyList_Append(result, VimToPython(&curr->li_tv));
+        }
+    }
+    else if (our_tv->v_type = VAR_DICT)
+    {
+        result = PyDict_New();
+        int i=0;
+        dictitem_T *di;
+        hashtab_T ht = our_tv->vval.v_dict->dv_hashtab;
+        hashitem_T *hi = ht.ht_array;
+        if (hi==NULL) hi = ht.ht_smallarray;
+        for(i=0; i<ht.ht_mask+1; i++)
+        {
+            if(hi->hi_key && hi->hi_key != HI_KEY_REMOVED)
+            {
+                di = HI2DI(hi);
+                PyDict_SetItemString(result, hi->hi_key, VimToPython(&di->di_tv));
+
+            }
+            hi++;
+        }
+    }
+    else
+    {
+        Py_INCREF(Py_None);
+        return Py_None;
+    }
+    return result;
+}
 /*ARGSUSED*/
     static PyObject *
 VimEval(PyObject *self, PyObject *args)
@@ -1089,6 +1156,7 @@
 #ifdef FEAT_EVAL
     char *expr;
     char *str;
+    typval_T   *our_tv;
     PyObject *result;
 
     if (!PyArg_ParseTuple(args, "s", &expr))
@@ -1096,21 +1164,25 @@
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    str = (char *)eval_to_string((char_u *)expr, NULL);
+    //str = (char *)eval_to_string((char_u *)expr, NULL);
+    our_tv = eval_expr((char_u *) expr, NULL);
+
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
-    if (str == NULL)
+    if (our_tv == NULL)
     {
  PyErr_SetVim(_("invalid expression"));
  return NULL;
-    }
+    }
+
+    result = VimToPython(our_tv);
 
-    result = Py_BuildValue("s", str);
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    vim_free(str);
+    clear_tv(our_tv);
+    vim_free(our_tv);
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

Bram Moolenaar

G. Sumner Hayes wrote:

> Bram Moolenaar <[hidden email]> wrote:
> > Very good.  So it's completely functional now,
> > right?
>
> Yes, it looks functional to me.  The taglist function
> returns a list of dictionaries keyed/valued by strings
> and so seems to test every case except integers, which
> I've tested independently (and they do come back as
> strings, for backward compatibility).
>
> I've also done things like:
>   let a = {1:'one', 2:['two', 'three']}
>   py print vim.eval('a')[2]  #and so forth
> to test nesting dicts/lists in the other direction.
>
> I haven't exactly tried throwing huge data structures
> at it but I see no reason it shouldn't work.

Well, you could try doing something complicated and see if Vim crashes.
Recursive references (e.g., a dict containing itself) are always great
fun.

And when you have the ccmalloc library you could check for memory leaks
(search the Makefile for -DEXITFREE).

> > I'll put it in the todo list to be included.  But
> > perhaps you can fix the formatting to match the Vim
> > source code.  And I also need an update for the
> > documentation.
>
> I tried to fix formatting (mostly braces) to conform
> with the rest of the vim code.  Attached.  Most
> functions in that file have an /*ARGSUSED*/ comment
> (probably for lint or similar) that I don't know the
> exact meaning of and so did not duplicate.
>
> Also gave a shot at documentation updates.  Attached.

Thanks for the update!

> Open issues:
> 1. What should we do if the eval doesn't return a
> supported type (I return None, presumably a vim
> exception will happen and I'm not sure that we want to
> raise a Python exception.)

Using None might be the best.  Or transform it to a string with
tv2string().

> 2. 4 lines are duplicated from eval.c, we might want
> to move at least the #defines into a public header and
> avoid duplication

This part is a bit problemous, knowledge about the Dictionary
implemenation is being used.  It's difficult to avoid though.  Instead
of the #define HI2DI you could use a function that is in eval.c.

> 3. I use clear_tv/vim_free(tv) to free memory.  You
> may want to switch that to free_tv when you publish
> that function.

Yes, I made a note about that.

- Bram

--
Nothing is fool-proof to a sufficiently talented fool.

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://www.ICCF.nl         ///
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

G. Sumner Hayes
Thanks for looking at this Bram.

Bram Moolenaar <[hidden email]> wrote:
> G. Sumner Hayes wrote:
> Well, you could try doing something complicated and
> see if Vim crashes.

Deep structures seem okay.

> Recursive references (e.g., a dict containing
> itself) are always great fun.

These break.  I'm tempted to punt on properly handling
recursive objects for now, and just put in a maximum
recursion depth (as Vim script does when you echo such
a structure); a proper fix requires keeping a table of
visited objects.

> And when you have the ccmalloc library you could
> check for memory leaks (search the Makefile for
> -DEXITFREE).

I don't have ccmalloc.  I use dynamic memory in 2
ways:
1. The tv returned by eval_expr, which is always freed
unless it's NULL.  Should be fine.
2. Python objects are allocated; these are all
reference counted and stored, so if the if_python
interface treated the returned objects properly before
then it should still be fine.

Obviously that's the theory, if someone out there has
ccmalloc then testing would be a good idea.

> > Open issues:
> > 1. What should we do if the eval doesn't return a
> > supported type (I return None, presumably a vim
> > exception will happen and I'm not sure that we
> > want to raise a Python exception.)
>
> Using None might be the best.  Or transform it to a
> string with tv2string().

I don't like a string, since it makes it harder for
Python extensions to test whether the call failed.  An
exception might be reasonable, but None is also
reasonable and already implemented so unless someone
has a good argument for the exception I'm inclined to
leave as-is.

> > 2. 4 lines are duplicated from eval.c, we might
> > want to move at least the #defines into a
> > public header and avoid duplication
>
> This part is a bit problemous, knowledge about the
> Dictionary implemenation is being used.  It's
> difficult to avoid though.  Instead of the #define
> HI2DI you could use a function that is in eval.c.

Good idea, I'll do that.

Thanks again for your time,

Sumner


               
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

G. Sumner Hayes
New patch attached.  I'm not re-sending the doc patch
as it's unchanged.

"G. Sumner Hayes" <[hidden email]> wrote:
> Bram Moolenaar <[hidden email]> wrote:
> > Recursive references (e.g., a dict containing
> > itself) are always great fun.
>
> These break.  I'm tempted to punt on properly
> handling recursive objects for now, and just
> put in a maximum recursion depth (as Vim
> script does when you echo such a structure)

Done.  Dealing properly with recursive objects is on
my TODO but I want to look at another issue first.  So
for now I cap recursion.

> > HI2DI you could use a function that is in eval.c.
>
> Good idea, I'll do that.

Done, but I don't know where to put the prototype.
It's in structs.h now which seems wrong.  Feel free to
move it to another header.

Sumner


               
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

--- /home/sumner/src/vim2/src/structs.h 2005-12-19 13:26:27.000000000 -0500
+++ /home/sumner/src/vim/src/structs.h 2006-01-09 19:58:16.405682598 -0500
@@ -1049,6 +1049,7 @@
 };
 
 typedef struct dictitem_S dictitem_T;
+extern dictitem_T * dict_lookup(hashitem_T *hi);
 
 #define DI_FLAGS_RO 1 /* "di_flags" value: read-only variable */
 #define DI_FLAGS_RO_SBX 2 /* "di_flags" value: read-only in the sandbox */
--- /home/sumner/src/vim2/src/eval.c 2005-12-20 04:33:37.000000000 -0500
+++ /home/sumner/src/vim/src/eval.c 2006-01-09 19:48:04.555133449 -0500
@@ -5312,6 +5312,12 @@
     return item1 == NULL && item2 == NULL;
 }
 
+    dictitem_T *
+dict_lookup(hi)
+    hashitem_T *hi;
+{
+    return HI2DI(hi);
+}
 /*
  * Return TRUE when two dictionaries have exactly the same key/values.
  */
--- /home/sumner/src/vim2/src/if_python.c 2005-10-13 14:43:52.000000000 -0400
+++ /home/sumner/src/vim/src/if_python.c 2006-01-09 19:48:36.164538002 -0500
@@ -1082,6 +1082,75 @@
     return result;
 }
 
+/*
+ * Function to translate a typval_T into a PyObject; this will recursively
+ * translate lists/dictionaries into their Python equivalents.
+ *
+ * The depth parameter is too avoid infinite recursion, set it to 1 when
+ * you call VimToPython.
+ */
+    static PyObject *
+VimToPython(typval_T *our_tv, int depth)
+{
+    PyObject *result;
+
+    /* Avoid infinite recursion */
+    if(depth > 100)
+    {
+        Py_INCREF(Py_None);
+        result = Py_None;
+        return result;
+    }
+
+    depth = depth + 1;
+    if (our_tv->v_type == VAR_STRING)
+    {
+        result = Py_BuildValue("s", our_tv->vval.v_string);
+    }
+    else if (our_tv->v_type == VAR_NUMBER)
+    {
+        char buf[NUMBUFLEN];
+        sprintf(buf, "%ld", (long)our_tv->vval.v_number);
+        result = Py_BuildValue("s", buf);
+    }
+    else if (our_tv->v_type == VAR_LIST)
+    {
+        int i=0;
+        struct listvar_S *list = our_tv->vval.v_list;
+        listitem_T *curr = list->lv_first;
+        result = PyList_New(0);
+        for(i=0; i<list->lv_len; i++, curr = curr->li_next)
+        {
+            PyList_Append(result, VimToPython(&curr->li_tv, depth));
+        }
+    }
+    else if (our_tv->v_type = VAR_DICT)
+    {
+        result = PyDict_New();
+        int i=0;
+        dictitem_T *di;
+        hashtab_T ht = our_tv->vval.v_dict->dv_hashtab;
+        hashitem_T *hi = ht.ht_array;
+        if (hi==NULL) hi = ht.ht_smallarray;
+        for(i=0; i<ht.ht_mask+1; i++)
+        {
+            if(hi->hi_key && hi->hi_key != HI_KEY_REMOVED)
+            {
+                di = dict_lookup(hi);
+                PyDict_SetItemString(result, hi->hi_key, VimToPython(&di->di_tv, depth));
+
+            }
+            hi++;
+        }
+    }
+    else
+    {
+        Py_INCREF(Py_None);
+        result = Py_None;
+    }
+    depth = depth - 1;
+    return result;
+}
 /*ARGSUSED*/
     static PyObject *
 VimEval(PyObject *self, PyObject *args)
@@ -1089,6 +1158,7 @@
 #ifdef FEAT_EVAL
     char *expr;
     char *str;
+    typval_T   *our_tv;
     PyObject *result;
 
     if (!PyArg_ParseTuple(args, "s", &expr))
@@ -1096,21 +1166,25 @@
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    str = (char *)eval_to_string((char_u *)expr, NULL);
+    //str = (char *)eval_to_string((char_u *)expr, NULL);
+    our_tv = eval_expr((char_u *) expr, NULL);
+
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
-    if (str == NULL)
+    if (our_tv == NULL)
     {
  PyErr_SetVim(_("invalid expression"));
  return NULL;
-    }
+    }
+
+    result = VimToPython(our_tv, 1);
 
-    result = Py_BuildValue("s", str);
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    vim_free(str);
+    clear_tv(our_tv);
+    vim_free(our_tv);
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

Bram Moolenaar
In reply to this post by G. Sumner Hayes

G. Sumner Hayes wrote:

> Bram Moolenaar <[hidden email]> wrote:
> > G. Sumner Hayes wrote:
> > Well, you could try doing something complicated and
> > see if Vim crashes.
>
> Deep structures seem okay.
>
> > Recursive references (e.g., a dict containing
> > itself) are always great fun.
>
> These break.  I'm tempted to punt on properly handling
> recursive objects for now, and just put in a maximum
> recursion depth (as Vim script does when you echo such
> a structure); a proper fix requires keeping a table of
> visited objects.

Have a look at how items are deep-copied in eval.c  Search for copyID.
I don't know how Python does this, but I would not be surprised if it
works similarly.

> > > Open issues:
> > > 1. What should we do if the eval doesn't return a
> > > supported type (I return None, presumably a vim
> > > exception will happen and I'm not sure that we
> > > want to raise a Python exception.)
> >
> > Using None might be the best.  Or transform it to a
> > string with tv2string().
>
> I don't like a string, since it makes it harder for
> Python extensions to test whether the call failed.  An
> exception might be reasonable, but None is also
> reasonable and already implemented so unless someone
> has a good argument for the exception I'm inclined to
> leave as-is.

OK.  This also keeps future extensions open.

--
The 50-50-90 rule: Anytime you have a 50-50 chance of getting
something right, there's a 90% probability you'll get it wrong.

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://www.ICCF.nl         ///
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

G. Sumner Hayes
Bram Moolenaar <[hidden email]> wrote:

> G. Sumner Hayes wrote:
> > These break.  I'm tempted to punt on properly
> > handling recursive objects for now, and just
> > put in a maximum recursion depth (as Vim script
> > does when you echo such a structure); a proper
> > fix requires keeping a table of visited objects.
>
> Have a look at how items are deep-copied in eval.c
> Search for copyID. I don't know how Python does
> this, but I would not be surprised if it works
similarly.

Hmm.  I'm not 100% clear on how it works, before I
read through it can you verify that it'll properly
handle mutually recursive structures?

e.g.
b={}
a=[b]
b[0]=a

and then have deepcopy work okay?

Anyway, I did the mapping solution already.  Attached.

Basically, whenever I'm calling VimToPython on a tv, I
store a mapping from the tv to the Python object.
Then if I get called on something I've already
converted, I just get the Python object out of the
mapping instead of building a new one.

Tested on mutually recursive objects, works for me.

Sumner

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 
--- /home/sumner/src/vim2/src/structs.h 2005-12-19 13:26:27.000000000 -0500
+++ /home/sumner/src/vim/src/structs.h 2006-01-09 19:58:16.405682598 -0500
@@ -1049,6 +1049,7 @@
 };
 
 typedef struct dictitem_S dictitem_T;
+extern dictitem_T * dict_lookup(hashitem_T *hi);
 
 #define DI_FLAGS_RO 1 /* "di_flags" value: read-only variable */
 #define DI_FLAGS_RO_SBX 2 /* "di_flags" value: read-only in the sandbox */
--- /home/sumner/src/vim2/src/eval.c 2005-12-20 04:33:37.000000000 -0500
+++ /home/sumner/src/vim/src/eval.c 2006-01-09 19:48:04.555133449 -0500
@@ -5312,6 +5312,12 @@
     return item1 == NULL && item2 == NULL;
 }
 
+    dictitem_T *
+dict_lookup(hi)
+    hashitem_T *hi;
+{
+    return HI2DI(hi);
+}
 /*
  * Return TRUE when two dictionaries have exactly the same key/values.
  */
--- /home/sumner/src/vim2/src/if_python.c 2005-10-13 14:43:52.000000000 -0400
+++ /home/sumner/src/vim/src/if_python.c 2006-01-10 18:07:49.701076185 -0500
@@ -1082,6 +1082,91 @@
     return result;
 }
 
+/*
+ * Function to translate a typval_T into a PyObject; this will recursively
+ * translate lists/dictionaries into their Python equivalents.
+ *
+ * The depth parameter is too avoid infinite recursion, set it to 1 when
+ * you call VimToPython.
+ */
+    static PyObject *
+VimToPython(typval_T *our_tv, int depth, PyObject *lookupDict)
+{
+    PyObject *result;
+    char ptrBuf[255];
+
+    /* Avoid infinite recursion */
+    if(depth > 100)
+    {
+        Py_INCREF(Py_None);
+        result = Py_None;
+        return result;
+    }
+
+    depth = depth + 1;
+    sprintf(ptrBuf, "%p", our_tv);
+    result=PyDict_GetItemString(lookupDict, ptrBuf);
+    if(result)
+    {
+        Py_INCREF(result);
+    }
+    else if (our_tv->v_type == VAR_STRING)
+    {
+        result = Py_BuildValue("s", our_tv->vval.v_string);
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+    }
+    else if (our_tv->v_type == VAR_NUMBER)
+    {
+        char buf[NUMBUFLEN];
+        sprintf(buf, "%ld", (long)our_tv->vval.v_number);
+        result = Py_BuildValue("s", buf);
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+    }
+    else if (our_tv->v_type == VAR_LIST)
+    {
+        int i=0;
+        struct listvar_S *list = our_tv->vval.v_list;
+        listitem_T *curr = list->lv_first;
+        result = PyList_New(0);
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+        for(i=0; i<list->lv_len; i++, curr = curr->li_next)
+        {
+            PyList_Append(result, VimToPython(&curr->li_tv, depth, lookupDict));
+        }
+    }
+    else if (our_tv->v_type = VAR_DICT)
+    {
+        result = PyDict_New();
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+        int i=0;
+        dictitem_T *di;
+        hashtab_T ht = our_tv->vval.v_dict->dv_hashtab;
+        hashitem_T *hi = ht.ht_array;
+        if (hi==NULL) hi = ht.ht_smallarray;
+        for(i=0; i<ht.ht_mask+1; i++)
+        {
+            if(hi->hi_key && hi->hi_key != HI_KEY_REMOVED)
+            {
+                PyObject *newObject;
+                di = dict_lookup(hi);
+
+                newObject = VimToPython(&di->di_tv, depth, lookupDict);
+                /* PyDict_SetItemString doesn't steal a reference */
+                PyDict_SetItemString(result, hi->hi_key, newObject);
+                Py_DECREF(newObject);
+
+            }
+            hi++;
+        }
+    }
+    else
+    {
+        Py_INCREF(Py_None);
+        result = Py_None;
+    }
+    depth = depth - 1;
+    return result;
+}
 /*ARGSUSED*/
     static PyObject *
 VimEval(PyObject *self, PyObject *args)
@@ -1089,28 +1174,36 @@
 #ifdef FEAT_EVAL
     char *expr;
     char *str;
+    typval_T   *our_tv;
     PyObject *result;
+    PyObject    *lookup_dict;
 
     if (!PyArg_ParseTuple(args, "s", &expr))
  return NULL;
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    str = (char *)eval_to_string((char_u *)expr, NULL);
+    //str = (char *)eval_to_string((char_u *)expr, NULL);
+    our_tv = eval_expr((char_u *) expr, NULL);
+
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
-    if (str == NULL)
+    if (our_tv == NULL)
     {
  PyErr_SetVim(_("invalid expression"));
  return NULL;
-    }
+    }
+
+    lookup_dict=PyDict_New();
+    result = VimToPython(our_tv, 1, lookup_dict);
+    Py_DECREF(lookup_dict);
 
-    result = Py_BuildValue("s", str);
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    vim_free(str);
+    clear_tv(our_tv);
+    vim_free(our_tv);
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

G. Sumner Hayes
So after further code review/testing I've found a
reference counting problem in my code that could leak
memory.  Attached should fix.

I _think_ it's feature-complete now that recursive
structures are handled, and I've done a fair amount of
testing on it so hopefully it's ready for
beta-testing.

Certainly seems sufficient for getting the output of
taglist(), which is the only reason I wrote it in the
first place.  :-)

Sumner

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 
--- /home/sumner/src/vim2/src/structs.h 2005-12-19 13:26:27.000000000 -0500
+++ /home/sumner/src/vim/src/structs.h 2006-01-09 19:58:16.405682598 -0500
@@ -1049,6 +1049,7 @@
 };
 
 typedef struct dictitem_S dictitem_T;
+extern dictitem_T * dict_lookup(hashitem_T *hi);
 
 #define DI_FLAGS_RO 1 /* "di_flags" value: read-only variable */
 #define DI_FLAGS_RO_SBX 2 /* "di_flags" value: read-only in the sandbox */
--- /home/sumner/src/vim2/src/eval.c 2005-12-20 04:33:37.000000000 -0500
+++ /home/sumner/src/vim/src/eval.c 2006-01-09 19:48:04.555133449 -0500
@@ -5312,6 +5312,12 @@
     return item1 == NULL && item2 == NULL;
 }
 
+    dictitem_T *
+dict_lookup(hi)
+    hashitem_T *hi;
+{
+    return HI2DI(hi);
+}
 /*
  * Return TRUE when two dictionaries have exactly the same key/values.
  */
--- /home/sumner/src/vim2/src/if_python.c 2005-10-13 14:43:52.000000000 -0400
+++ /home/sumner/src/vim/src/if_python.c 2006-01-10 18:22:42.836091209 -0500
@@ -1082,6 +1082,93 @@
     return result;
 }
 
+/*
+ * Function to translate a typval_T into a PyObject; this will recursively
+ * translate lists/dictionaries into their Python equivalents.
+ *
+ * The depth parameter is too avoid infinite recursion, set it to 1 when
+ * you call VimToPython.
+ */
+    static PyObject *
+VimToPython(typval_T *our_tv, int depth, PyObject *lookupDict)
+{
+    PyObject *result;
+    char ptrBuf[255];
+
+    /* Avoid infinite recursion */
+    if(depth > 100)
+    {
+        Py_INCREF(Py_None);
+        result = Py_None;
+        return result;
+    }
+
+    depth = depth + 1;
+    sprintf(ptrBuf, "%p", our_tv);
+    result=PyDict_GetItemString(lookupDict, ptrBuf);
+    if(result)
+    {
+        Py_INCREF(result);
+    }
+    else if (our_tv->v_type == VAR_STRING)
+    {
+        result = Py_BuildValue("s", our_tv->vval.v_string);
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+    }
+    else if (our_tv->v_type == VAR_NUMBER)
+    {
+        char buf[NUMBUFLEN];
+        sprintf(buf, "%ld", (long)our_tv->vval.v_number);
+        result = Py_BuildValue("s", buf);
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+    }
+    else if (our_tv->v_type == VAR_LIST)
+    {
+        int i=0;
+        struct listvar_S *list = our_tv->vval.v_list;
+        listitem_T *curr = list->lv_first;
+        result = PyList_New(0);
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+        for(i=0; i<list->lv_len; i++, curr = curr->li_next)
+        {
+            PyObject *newObj = VimToPython(&curr->li_tv, depth, lookupDict);
+            PyList_Append(result, newObj);
+            Py_DECREF(newObj);
+        }
+    }
+    else if (our_tv->v_type = VAR_DICT)
+    {
+        result = PyDict_New();
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+        int i=0;
+        dictitem_T *di;
+        hashtab_T ht = our_tv->vval.v_dict->dv_hashtab;
+        hashitem_T *hi = ht.ht_array;
+        if (hi==NULL) hi = ht.ht_smallarray;
+        for(i=0; i<ht.ht_mask+1; i++)
+        {
+            if(hi->hi_key && hi->hi_key != HI_KEY_REMOVED)
+            {
+                PyObject *newObject;
+                di = dict_lookup(hi);
+
+                newObject = VimToPython(&di->di_tv, depth, lookupDict);
+                /* PyDict_SetItemString doesn't steal a reference */
+                PyDict_SetItemString(result, hi->hi_key, newObject);
+                Py_DECREF(newObject);
+
+            }
+            hi++;
+        }
+    }
+    else
+    {
+        Py_INCREF(Py_None);
+        result = Py_None;
+    }
+    depth = depth - 1;
+    return result;
+}
 /*ARGSUSED*/
     static PyObject *
 VimEval(PyObject *self, PyObject *args)
@@ -1089,28 +1176,36 @@
 #ifdef FEAT_EVAL
     char *expr;
     char *str;
+    typval_T   *our_tv;
     PyObject *result;
+    PyObject    *lookup_dict;
 
     if (!PyArg_ParseTuple(args, "s", &expr))
  return NULL;
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    str = (char *)eval_to_string((char_u *)expr, NULL);
+    //str = (char *)eval_to_string((char_u *)expr, NULL);
+    our_tv = eval_expr((char_u *) expr, NULL);
+
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
-    if (str == NULL)
+    if (our_tv == NULL)
     {
  PyErr_SetVim(_("invalid expression"));
  return NULL;
-    }
+    }
+
+    lookup_dict=PyDict_New();
+    result = VimToPython(our_tv, 1, lookup_dict);
+    Py_DECREF(lookup_dict);
 
-    result = Py_BuildValue("s", str);
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    vim_free(str);
+    clear_tv(our_tv);
+    vim_free(our_tv);
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Python interface support for evals that return dict/list

Bram Moolenaar
In reply to this post by G. Sumner Hayes

G. Sumner Hayes wrote:

> Bram Moolenaar <[hidden email]> wrote:
> > G. Sumner Hayes wrote:
> > > These break.  I'm tempted to punt on properly
> > > handling recursive objects for now, and just
> > > put in a maximum recursion depth (as Vim script
> > > does when you echo such a structure); a proper
> > > fix requires keeping a table of visited objects.
> >
> > Have a look at how items are deep-copied in eval.c
> > Search for copyID. I don't know how Python does
> > this, but I would not be surprised if it works
> similarly.
>
> Hmm.  I'm not 100% clear on how it works, before I
> read through it can you verify that it'll properly
> handle mutually recursive structures?
>
> e.g.
> b={}
> a=[b]
> b[0]=a
>
> and then have deepcopy work okay?

That should work.  I tested it a long time ago.

> Anyway, I did the mapping solution already.  Attached.
>
> Basically, whenever I'm calling VimToPython on a tv, I
> store a mapping from the tv to the Python object.
> Then if I get called on something I've already
> converted, I just get the Python object out of the
> mapping instead of building a new one.
>
> Tested on mutually recursive objects, works for me.

Good to see you found a solution.  It's similar to what happens for
deepcopy in eval.c, except that there it stores the info with the copied
list or dict (using lv_copyID and lv_copylist or dv_copydict).

--
hundred-and-one symptoms of being an internet addict:
40. You tell the cab driver you live at
    http://123.elm.street/house/bluetrim.html
41. You actually try that 123.elm.street address.

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://www.ICCF.nl         ///
Reply | Threaded
Open this post in threaded view
|

[PATCH] [BETA-ready] Python interface support for evals that return dict/list

G. Sumner Hayes
In reply to this post by G. Sumner Hayes
This patch allows Python extensions to call vim.eval()
on vim commands that return dict or list structures.
So you can do things like:
:py tagmatches = vim.eval("taglist('stat')")

I've done a bit more testing and I think it's ready
for beta-test/inclusion.  Please let me know on
vim-dev if there are any problems.

Bram, this is the same as the final version I sent in
the development thread (the one that fixed the last
memory leak I know about).

Thanks for your time,

Sumner

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 
--- /home/sumner/src/vim2/src/structs.h 2005-12-19 13:26:27.000000000 -0500
+++ /home/sumner/src/vim/src/structs.h 2006-01-09 19:58:16.405682598 -0500
@@ -1049,6 +1049,7 @@
 };
 
 typedef struct dictitem_S dictitem_T;
+extern dictitem_T * dict_lookup(hashitem_T *hi);
 
 #define DI_FLAGS_RO 1 /* "di_flags" value: read-only variable */
 #define DI_FLAGS_RO_SBX 2 /* "di_flags" value: read-only in the sandbox */
--- /home/sumner/src/vim2/src/eval.c 2005-12-20 04:33:37.000000000 -0500
+++ /home/sumner/src/vim/src/eval.c 2006-01-09 19:48:04.555133449 -0500
@@ -5312,6 +5312,12 @@
     return item1 == NULL && item2 == NULL;
 }
 
+    dictitem_T *
+dict_lookup(hi)
+    hashitem_T *hi;
+{
+    return HI2DI(hi);
+}
 /*
  * Return TRUE when two dictionaries have exactly the same key/values.
  */
--- /home/sumner/src/vim2/src/if_python.c 2005-10-13 14:43:52.000000000 -0400
+++ /home/sumner/src/vim/src/if_python.c 2006-01-10 18:22:42.836091209 -0500
@@ -1082,6 +1082,93 @@
     return result;
 }
 
+/*
+ * Function to translate a typval_T into a PyObject; this will recursively
+ * translate lists/dictionaries into their Python equivalents.
+ *
+ * The depth parameter is too avoid infinite recursion, set it to 1 when
+ * you call VimToPython.
+ */
+    static PyObject *
+VimToPython(typval_T *our_tv, int depth, PyObject *lookupDict)
+{
+    PyObject *result;
+    char ptrBuf[255];
+
+    /* Avoid infinite recursion */
+    if(depth > 100)
+    {
+        Py_INCREF(Py_None);
+        result = Py_None;
+        return result;
+    }
+
+    depth = depth + 1;
+    sprintf(ptrBuf, "%p", our_tv);
+    result=PyDict_GetItemString(lookupDict, ptrBuf);
+    if(result)
+    {
+        Py_INCREF(result);
+    }
+    else if (our_tv->v_type == VAR_STRING)
+    {
+        result = Py_BuildValue("s", our_tv->vval.v_string);
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+    }
+    else if (our_tv->v_type == VAR_NUMBER)
+    {
+        char buf[NUMBUFLEN];
+        sprintf(buf, "%ld", (long)our_tv->vval.v_number);
+        result = Py_BuildValue("s", buf);
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+    }
+    else if (our_tv->v_type == VAR_LIST)
+    {
+        int i=0;
+        struct listvar_S *list = our_tv->vval.v_list;
+        listitem_T *curr = list->lv_first;
+        result = PyList_New(0);
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+        for(i=0; i<list->lv_len; i++, curr = curr->li_next)
+        {
+            PyObject *newObj = VimToPython(&curr->li_tv, depth, lookupDict);
+            PyList_Append(result, newObj);
+            Py_DECREF(newObj);
+        }
+    }
+    else if (our_tv->v_type = VAR_DICT)
+    {
+        result = PyDict_New();
+        PyDict_SetItemString(lookupDict, ptrBuf, result);
+        int i=0;
+        dictitem_T *di;
+        hashtab_T ht = our_tv->vval.v_dict->dv_hashtab;
+        hashitem_T *hi = ht.ht_array;
+        if (hi==NULL) hi = ht.ht_smallarray;
+        for(i=0; i<ht.ht_mask+1; i++)
+        {
+            if(hi->hi_key && hi->hi_key != HI_KEY_REMOVED)
+            {
+                PyObject *newObject;
+                di = dict_lookup(hi);
+
+                newObject = VimToPython(&di->di_tv, depth, lookupDict);
+                /* PyDict_SetItemString doesn't steal a reference */
+                PyDict_SetItemString(result, hi->hi_key, newObject);
+                Py_DECREF(newObject);
+
+            }
+            hi++;
+        }
+    }
+    else
+    {
+        Py_INCREF(Py_None);
+        result = Py_None;
+    }
+    depth = depth - 1;
+    return result;
+}
 /*ARGSUSED*/
     static PyObject *
 VimEval(PyObject *self, PyObject *args)
@@ -1089,28 +1176,36 @@
 #ifdef FEAT_EVAL
     char *expr;
     char *str;
+    typval_T   *our_tv;
     PyObject *result;
+    PyObject    *lookup_dict;
 
     if (!PyArg_ParseTuple(args, "s", &expr))
  return NULL;
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    str = (char *)eval_to_string((char_u *)expr, NULL);
+    //str = (char *)eval_to_string((char_u *)expr, NULL);
+    our_tv = eval_expr((char_u *) expr, NULL);
+
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
-    if (str == NULL)
+    if (our_tv == NULL)
     {
  PyErr_SetVim(_("invalid expression"));
  return NULL;
-    }
+    }
+
+    lookup_dict=PyDict_New();
+    result = VimToPython(our_tv, 1, lookup_dict);
+    Py_DECREF(lookup_dict);
 
-    result = Py_BuildValue("s", str);
 
     Py_BEGIN_ALLOW_THREADS
     Python_Lock_Vim();
-    vim_free(str);
+    clear_tv(our_tv);
+    vim_free(our_tv);
     Python_Release_Vim();
     Py_END_ALLOW_THREADS
 
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] [BETA-ready] Python interface support for evals that return dict/list

Bram Moolenaar

G. Sumner Hayes wrote:

> This patch allows Python extensions to call vim.eval()
> on vim commands that return dict or list structures.
> So you can do things like:
> :py tagmatches = vim.eval("taglist('stat')")
>
> I've done a bit more testing and I think it's ready
> for beta-test/inclusion.  Please let me know on
> vim-dev if there are any problems.
>
> Bram, this is the same as the final version I sent in
> the development thread (the one that fixed the last
> memory leak I know about).

Thanks.  I hope send out a snapshot with updated spell checking tonight,
I'll include a few patches tomorrow.

--
Q: Should I clean my house or work on Vim?
A: Whatever contains more bugs.

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://www.ICCF.nl         ///
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] [BETA-ready] Python interface support for evals that return dict/list

G. Sumner Hayes
Bram Moolenaar <[hidden email]> wrote:
> Thanks.  I hope send out a snapshot with updated
> spell checking tonight,
> I'll include a few patches tomorrow.

Cool, I'm not trying to rush you just making sure it's
clear what's ready for inclusion.  I'll mark 'em with
the BETA tag when I think they're feature-complete and
I've finished up my testing.

Sumner

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 
Reply | Threaded
Open this post in threaded view
|

Re: TODO for vim 7

G. Sumner Hayes
In reply to this post by Bram Moolenaar
Bram Moolenaar <[hidden email]> wrote:
> Thanks.  I hope send out a snapshot with updated
> spell checking tonight,
> I'll include a few patches tomorrow.
>

Is there a public list of features that you're
thinking of implementing for Vim 7?  Something I could
look at and see if anything looks interesting that I
could run with to help you out.

Thanks,

Sumner

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 
12