efficient way for building class browsers.

[Sridhar R]

I am looking for a class browser that has these features.

1. Given a symbol (class, method or function) it should giveback
the lineno n source code
2. It should be efficient and quick.

I looked at the `pyclbr` module, but it's slower. I will be _often_
regenerating the symbols from exactly one python source file. But for
an average python script in my machine it takes 2 secs for pyclbr to
build the dictionary of symbols.

When I profiled pyclbr, I found that most of the time was spent on the
tokenize.get_token** functions, and the remaining time was spent on
pyclbr._readmodule function.

Remember I need a class browser, that also tells me the line no in
source code of that symbol. (So compiler module (ast) can't be used -
Is it so? )

Are there any better ways for achieving this?

 

[Josiah Carlson]

I am looking for a class browser that has these features.

Only a browser? Text or GUI? What OS? Any particular GUI toolkit?

Remember I need a class browser, that also tells me the line no in
source code of that symbol. (So compiler module (ast) can't be used -
Is it so? )

compiler.ast can be used for this, but only produces proper results when
the file has no syntax errors. Don't get me wrong, using the output of
compiler.ast can be a pain in the ass, and slow as hell (5 seconds for
100k of source on a celeron 400), but it can be used.

Are there any better ways for achieving this?

You can write a limited parser (subject to certain flaws), one that
doesn't use tokenize or compiler.ast, and get pretty fast results. I
wrote one to go into my own editor PyPE (pype.sourceforge.net), and it
has no problems parsing a 100k Python source file and displaying the
source tree in a GUI in less than 1/2 second on a celeron 400.

It wouldn't be too bad to pull the tree portion out, what kind of
browser do you need?

- Josiah

 

[Lothar Scholz]

Only a browser? Text or GUI? What OS? Any particular GUI toolkit?


compiler.ast can be used for this, but only produces proper results when
the file has no syntax errors. Don't get me wrong, using the output of
compiler.ast can be a pain in the ass, and slow as hell (5 seconds for
100k of source on a celeron 400), but it can be used.


You can write a limited parser (subject to certain flaws), one that
doesn't use tokenize or compiler.ast, and get pretty fast results. I
wrote one to go into my own editor PyPE (pype.sourceforge.net), and it
has no problems parsing a 100k Python source file and displaying the
source tree in a GUI in less than 1/2 second on a celeron 400.

Thats slow i have a parser that parses 100k in less then 15 ms on a
1,2 GHz system. It will be part of "http://www.python-ide.com"
It is a 100% GNU Eiffel solution which generates plain C code.

I think the 1/2 second could also be done with some intelligent
regexpr's. There is something like this in the IDLE code.

 

[Josiah Carlson]

You can write a limited parser (subject to certain flaws), one that

Thats slow i have a parser that parses 100k in less then 15 ms on a
1,2 GHz system. It will be part of "http://www.python-ide.com"
It is a 100% GNU Eiffel solution which generates plain C code.

I never said it was the fastest, I said it was "pretty fast", and
provided results that it was faster than the pyclbr and compiler.ast
modules.

It is also written in Python, which seems to be more of what Sridhar was
looking for. Furthermore, the source for the fast parser is freely
available. Considering that Arachno Python IDE is a commercial effort,
the liklihood of the parser being available in source or loadable module
format seems fairly unlikely.

I think the 1/2 second could also be done with some intelligent
regexpr's. There is something like this in the IDLE code.

Checking the IDLE source for version 1.0 (I would check CVS, but CVS
seems to be having issues at sourceforge), it uses pyclbr.

- Josiah

 

[Michele Simionato]

Checking the IDLE source for version 1.0 (I would check CVS, but CVS
seems to be having issues at sourceforge), it uses pyclbr.

The inspect module uses regexps to extract the source code from classes
(and it is always a very good reading).

Michele

 

[Sridhar R]

Only a browser? Text or GUI? What OS? Any particular GUI toolkit?

IDE. Linux,Windows,Mac. PyGTK

compiler.ast can be used for this, but only produces proper results when
the file has no syntax errors. Don't get me wrong, using the output of
compiler.ast can be a pain in the ass, and slow as hell (5 seconds for
100k of source on a celeron 400), but it can be used.


You can write a limited parser (subject to certain flaws), one that
doesn't use tokenize or compiler.ast, and get pretty fast results. I
wrote one to go into my own editor PyPE (pype.sourceforge.net), and it
has no problems parsing a 100k Python source file and displaying the
source tree in a GUI in less than 1/2 second on a celeron 400.

I'll try that. I have come across tagmanager (C) code written for
the Anjuta (http://anjuta.org) project. I should give it a try too.

It wouldn't be too bad to pull the tree portion out, what kind of
browser do you need?

I will put it in brief. The class browser can't support all
standard and third party python packages, since most of them is
available as shared objects. So mine is focused only on the current
project, i.e. user created (or application) python files. So the
browser should be able to fetch the tags (class, functions, methods)
from source code along with the line number information. That's all
about. Any further possiblities in this regard is also expected.

 

[Josiah Carlson]

I am looking for a class browser that has these features.

IDE. Linux,Windows,Mac. PyGTK

Seemingly this suggests that you want an IDE that runs on linux,
windows, mac, and that uses the PyGTK toolkit.

I personally don't spend much time looking through PyGTK apps, so I
can't be of much help. On the other hand, if you have a parser,
constructing the tree control/widget from the parsing is very easy.

I'll try that. I have come across tagmanager (C) code written for
the Anjuta (http://anjuta.org) project. I should give it a try too.

The control in PyPE is written for wxPython, but the parser is platform
independant.

I will put it in brief. The class browser can't support all
standard and third party python packages, since most of them is
available as shared objects. So mine is focused only on the current
project, i.e. user created (or application) python files. So the
browser should be able to fetch the tags (class, functions, methods)
from source code along with the line number information. That's all
about. Any further possiblities in this regard is also expected.

Fetching the various function/class/method names is easy, look at the
parser included with PyPE. It isn't perfect, but works pretty well
considering that it is neither a regular expression, nor does it have a
tokenizer.

- Josiah