ANN: PyKDE now does KDE plugins

[Jim Bublitz]

The latest release of PyKDE (3.8.0) includes the ability to write
KDE panel applets completely in Python -- absolutely no C++
required. This is the first in what's planned to be a number of
extensions for PyKDE that allow plugins and related objects to
be created entirely in Python; David Boddie is nearing release
of modules for authoring KParts for export (PyKDE already
imports KParts), KDE Control Center modules, and IOSlaves.

Future plans include allowing QWidget subclasses created in
Python to be imported into Qt Designer with complete
functionality, and possibly Python scripting and plugins for KDE
apps like KOffice and Kontact. The underlying mechanisms and
code are similar in all cases.

In some cases, specific .so libs will still be required (depends
on the plugin loader), but the Python modules will include
autogeneration of the necessary C++ code, along with installers
to simplify the task of making the plugins available.

PyKDE 3.8.0 requires SIP 3.8 and PyQt 3.8. While the PyKDE 3.8.0
tarball is available now, it will be several days or more before
RPM and Debian packages are available. You can check the PyKDE
mailing list for more info. PyKDE-3.8.0 supports any version of
KDE from 3.0.0 through 3.1.4.

PyKDE 3.8.0 is also the last release that will support Python
versions below 2.3, and the last that will work with sip 3.8
(sip 4.0 is already in pre-release). However, support on 3.8.0
(bug fixes and some enhancements) will continue for the
forseeable future.

PyKDE, PyQt and sip are available at:
http://riverbankcomputing.co.uk

Info on KDE panel applets is at:
http://www.riverbankcomputing.co.uk/pykde/docs/panapp1.html

The PyKDE mailing list is at:
http://mats.imk.fraunhofer.de/mailman/listinfo/pykde

Jim

 

[John J. Lee]

required. This is the first in what's planned to be a number of
extensions for PyKDE that allow plugins and related objects to
be created entirely in Python; David Boddie is nearing release
of modules for authoring KParts for export (PyKDE already
imports KParts), KDE Control Center modules, and IOSlaves.

Future plans include allowing QWidget subclasses created in
Python to be imported into Qt Designer with complete
functionality, and possibly Python scripting and plugins for KDE
apps like KOffice and Kontact. The underlying mechanisms and
code are similar in all cases.

[...]

Wow, sounds like impressive stuff. How does it work? Is there a
standard .so proxy that can be used by all Python plugins? What sort
of memory footprint do these plugins have?

In some cases, specific .so libs will still be required (depends
on the plugin loader), but the Python modules will include
autogeneration of the necessary C++ code, along with installers
to simplify the task of making the plugins available.

What makes the difference between all-Python and needing a bit of C++
here?


John

 

[Jim Bublitz]

required. This is the first in what's planned to be a number
of extensions for PyKDE that allow plugins and related objects
to be created entirely in Python; David Boddie is nearing
release of modules for authoring KParts for export (PyKDE
already imports KParts), KDE Control Center modules, and
IOSlaves.

Future plans include allowing QWidget subclasses created in
Python to be imported into Qt Designer with complete
functionality, and possibly Python scripting and plugins for
KDE apps like KOffice and Kontact. The underlying mechanisms
and code are similar in all cases.

[...]

Wow, sounds like impressive stuff. How does it work? Is there
a standard .so proxy that can be used by all Python plugins?

Basically yes - it's a little more convoluted than that. Panel
applets need a .desktop file which specifies the .so lib to
load. The .so specified is symlinked to the proxy, so it's
filename is correct and unique, but still runs the proxy code.
(There's a fake .la file in this case too, since the loader
expects one)

The .so proxy starts another .so that wraps the Python
interpreter, and from the config file name that the plugin
loader passes in it computes the name of the .py file. The .py
file is loaded into the interpreter, and the Python factory
function is located and called. That constructs the actual
applet (from Python via PyKDE/PyQt). A PyObject is returned to
the proxy .so which uses sip/PyKDE to convert it to a C++
pointer to the required type which is passed back to the loading
app.

The same scheme will work for almost any plugin - the major
difference is that in some cases the plugin loader doesn't
provide enough info to allow a proxy for the .so file (for
example, to load "somePlugin" the loader might expect
libsomePlugin with an init_somePlugin factory function in C++).
For those cases, the plan is to autogenerate the plugin-specific
..so file (naming is the only thing that varies), but it still
requires a C++ 'make" step.

For panel applets, there's an installer that does most of the
work (except for actually writing the py file) and does things
like validate and serve as a test harness. Plugins can be hard
to debug, and a failed load rarely returns any error info, so t
it's helpful to be able to run the plugin outside the loading
app.

It takes more lines than to describe than to implement - it
really isn't much code. David Boddie, who did a lot of the
development on panel applets has a bunch of other stuff coming
soon - KParts (including a Wiki KPart), IOSlaves, KDE Control
Center plugins. The same scheme will also work for styles,
QtDesigner plugins and apps that take plugins. Quite a while ago
I had a similar KSpread (KOffice spreadsheet) plugin working
using a cruder version of the same basic idea. Apps require a
Python wrapper for their API to be useful, but that's also
pretty simple to write using sip (for a well-written app
anyway).

What sort of memory footprint do these plugins have?

The "Presidential" answer is: not very big at all. The real
answer is that by the time you load PyQt and the necessary PyKDE
modules you're up around 10MB or more I'd guess (plus other
shared libs that will already have been loaded). The Python
interpreter .so adds about another MB, the plugin code itself is
in Python and only maybe 5KB to a few hundred KB depending on
complexity. The proxy .so is very small too.

My opinion is that this is great for prototyping, but some people
are willing to spend that much memory to be able to do plugins
in Python. I had one applet running for a few weeks, and I'd
have to say that on a machine with 1GB of memory it isn't a big
deal, but I still find it a little mind-boggling - maybe 20X
what the original IBM PC came with just to run a 40x40 pixel
panel applet.

There's a couple places I find this really worthwhile though - if
you're writing an app with PyQt/PyKDE in the first place or
regularly have an app running based on those, the memory cost is
minimal because it's all in shared libs. The other really useful
place is a situation like the spreadsheet plugin I mentioned -
it only took about 2MB total and allowed all kinds of stuff the
app itself couldn't do (eg manipulate a spreadsheet in the
background). That's based on a thin wrapper instead of a
complete set of bindings and no GUI programming in the plugin
(basically no PyQt or PyKDE).

What makes the difference between all-Python and needing a bit
of C++ here?

Just what the app loading the plugin expects to find. If the
factory or init function of the plugin is uniquely named, there
isn't a convenient way to intercept the call - well there
probably is, but I haven't figured it out yet. If someone knows
more about .so loading and structure than I do and sees a
solution I'd be interested in hearing about it. I'm sure there
must be a way to hack the .so's symbol table or some such, but
it's beyond me at them moment. I suppose if the parameters
passed to the plugin don't convey the necessary info, that would
be another case, but I don't think that occurs in KDE or Qt.

On the other hand, there's really no reason why a proxy wouldn't
*always* work if C++ developers allowed for it - always call a
factory function with the same name ("init", "create". whatever)
and pass in an identifying string for locating the plugin.
Probably makes C++ plugins a little harder to write though.

Jim

 

[John J. Lee]

The same scheme will work for almost any plugin - the major
difference is that in some cases the plugin loader doesn't
provide enough info to allow a proxy for the .so file (for
example, to load "somePlugin" the loader might expect
libsomePlugin with an init_somePlugin factory function in C++).

[...]

Sounds like some gentle persuasion could be beneficial here!

[from end of Jim's post]

On the other hand, there's really no reason why a proxy wouldn't
*always* work if C++ developers allowed for it - always call a
factory function with the same name ("init", "create". whatever)
and pass in an identifying string for locating the plugin.
Probably makes C++ plugins a little harder to write though.

Not if a (C++) interface is *added*, not replaced. The KDE plugin
loader could check for init_blah first, and if that's not there, check
for plain old init.

I had a similar KSpread (KOffice spreadsheet) plugin working
using a cruder version of the same basic idea. Apps require a
Python wrapper for their API to be useful, but that's also
pretty simple to write using sip (for a well-written app
anyway).

[...]

Well, it might be if there were any sip documentation I had some
success using sip before, but it was hard work. I noticed that Phil
promised docs for PyQt/sip 4, though!


John

 

[Jim Bublitz]

Not if a (C++) interface is *added*, not replaced. The KDE
plugin loader could check for init_blah first, and if that's
not there, check for plain old init.

There sre several different loading schemes. I've always taken
the approach that PyKDE and related stuff shouldn't require any
modifications to KDE (although I did file one feature request
before we figured out how to load panel applets reliably).
Considering any scripting language can use the same approach and
there are other bindings in various stages of development, it
would probably be a good idea to bring this up with KDE
eventually. I'd prefer to have more running code before doing
that.

Well, it might be if there were any sip documentation I had
some
success using sip before, but it was hard work. I noticed that
Phil promised docs for PyQt/sip 4, though!

Yeah - I promised Phil I'd write a sip tutorial too. Notice there
aren't any links

Actually I have a tutorial about half done, but sip 4.0 is in
pre-release with lots of improvements (less handwritten code),
so I'm waiting for that to be completed. There are some links to
docs at tiverbankcomputing.co.uk too.

I also have a tool in development that generates sip files
automatically from h files (it's how PyKDE is generated) - it's
probably about 6 months (or more) away from release. The code
generation works well, but the doc generation and build
machinery generation need a lot of work yet. That will need
updating for sip 4 also.

Jim

 

[John J. Lee]

success using sip before, but it was hard work. I noticed that Phil
promised docs for PyQt/sip 4, though!

[...]

meant to say sip 4, not PyQt/sip 4.


John

 

[John J. Lee]

Yeah - I promised Phil I'd write a sip tutorial too. Notice there
aren't any links

Actually I have a tutorial about half done, but sip 4.0 is in
pre-release with lots of improvements (less handwritten code),

Great, I look forward to it.

I also have a tool in development that generates sip files
automatically from h files (it's how PyKDE is generated) - it's
probably about 6 months (or more) away from release. The code
generation works well, but the doc generation and build
machinery generation need a lot of work yet. That will need
updating for sip 4 also.

Have you seen GCC-XML and pyste (part of Boost Python)?


John

 

[Jim Bublitz]

Have you seen GCC-XML and pyste (part of Boost Python)?

I've looked at both just out of curiousity, but not in enough
detail to say anything intelligent about either. I'm very
comfortable with sip, and from that bias didn't see anything
that would make me want to switch methods.

The kde-bindings people are using gcc-xml for "Smoke" I believe,
which as I understand it is kind of a generalpurpose
multi-language bindings generator.


Jim

 

[John J. Lee]

I've looked at both just out of curiousity, but not in enough
detail to say anything intelligent about either. I'm very
comfortable with sip, and from that bias didn't see anything
that would make me want to switch methods.

[...]

I was thinking of it as a way to generate sip files, not to replace
sip.


John

 

[Jim Bublitz]

Have you seen GCC-XML and pyste (part of Boost Python)?

I've looked at both just out of curiousity, but not in enough
detail to say anything intelligent about either. I'm very
comfortable with sip, and from that bias didn't see anything
that would make me want to switch methods. [...]

I was thinking of it as a way to generate sip files, not to
replace sip.

Wouldn't work for me (directly). I need an intermediate format
for both the h files AND the previous version's sip files to
generate the new version's sip files. I need to read in PyQt's
sip files too. That's both for versioning (change detection) and
to transfer forward any handwritten code, doc markup, ignored
methods (like the "virtualHook" stuff that's everywhere in KDE),
auxiliary code like mapped types, etc.

The same parser builds a symbol table for each set of files (the
other two above wouldn't handle sip syntax) and then merges the
two symbol tables and spits out the sip files. I also need to
build some kind of hierarchy so I can scope names - sip needs
fully-qualified names in a lot of places C++ will allow implicit
scope (eg namespaces). It took me 8-10 hours to do just scoping
manually last time I did it that way - the whole KDE generation
run takes 2-3 minutes now (everything except new handwritten
code that's needed, and those are flagged).

Jim