embedded python?

[Alexander May]

Hi,

I love Python! I've been using it for a couple of years now and have found
it to be a highly productive language. I evangelize it to my developer
friends and am probably responsible for the sale of at least 10 Alex
Martelli books. I am now in the fortunate position of being able to use
Python for a large project, and as such I have a question.

We are developing a distributed application running on approximately six
thousand nodes with somewhat limited hardware resources. Our hardware
target is 66 MHz ARM style processor with 16 Mb ram. We haven't selected
specific hardware yet; the hardware target is what we are trying to fit into
based on price constraints. Each device needs to be able to handle about 2
kbs (yes kilo, not mega) worth of network traffic.

I intend to a least prototype the system in Python. It would be great if we
could we could use Python in production by embedding it in the hardware. My
question is does anyone have any real world experience using python in an
embedded system? Two general categories of questions:

1) Are there any embedded Pythons out there? The nodes will likely be
running some form of Linux, but I don't particularly feel like devoting
resrouces to porting python. Any embedded Linuxes supporting Python?
Thoughts in general?

2) What are the resource requirements of Python? How much overhead do the
network related modules add? Obviously I'll be able to determine our
application's resource usage once the prototype is written, but I curious
about other people's past experience.

In short, is embedding python realistic?

Thoughts and comments are greatly appreciated!

Thanks,
Alex

 

[Peter Hansen]

We are developing a distributed application running on approximately six
thousand nodes with somewhat limited hardware resources. Our hardware
target is 66 MHz ARM style processor with 16 Mb ram. We haven't selected
specific hardware yet; the hardware target is what we are trying to fit into
based on price constraints. Each device needs to be able to handle about 2
kbs (yes kilo, not mega) worth of network traffic.

You mention network, so presumably you have Ethernet (?) interfaces, and
plan on TCP or UDP, in which case you probably first need to consider
operating systems... Python does not automatically give you support for
networking, it just uses the underlying OS support.

I intend to a least prototype the system in Python. It would be great if we
could we could use Python in production by embedding it in the hardware. My
question is does anyone have any real world experience using python in an
embedded system?

Yes, there are a few people around here who have worked on embedded
Python systems, including me (at a past employer).

1) Are there any embedded Pythons out there? The nodes will likely be
running some form of Linux, but I don't particularly feel like devoting
resrouces to porting python. Any embedded Linuxes supporting Python?
Thoughts in general?

I doubt you'll need to do much "porting". Unless the Linices you
are considering are quite unusual, they ought to look a lot like any
other Linux, and Python may well work largely unchanged. Any required
changes could be in areas you don't need, anyway. Again, make sure
you consider right away exactly what OS resources, other than
networking, that you actually need. Multitasking? Signals? Audio?
GUI? etc...

2) What are the resource requirements of Python? How much overhead do the
network related modules add?

_socket.pyd is 49212 on Win32, and 124040 (probably with debugging
symbol table included?) on Linux, for Python 2.3.4. This includes
lots of doc-strings, though, so you can probably shrink it pretty
easily with a configurable setting during compilation.

In short, is embedding python realistic?

Definitely. We did it first on a 33MHz 386 (or 286?) compatible
board with only 1MB of flash and 1MB of RAM, and a tiny embedded
OS with no networking. That took a little slashing, such as removing
floating point support, but for the most part such things were
supported well by #defines during compilation. I think we based
that one on Python 1.5.2. (It was a few years ago.) Performance,
however, turned out to be inadequate and we decided we'd get
farther faster by keeping Python and switching to faster hardware
(than by ditching Python and doing it all in C).

The next version was a 100MHz 486 compatible with 32MB RAM and
Compact Flash as a hard drive. We picked 32MB flash as the target,
but used 64MB for prototyping and kept using it thereafter because
CF cards kept getting cheaper. This version ran vanilla Linux
with everything non-essential removed, and stock Python recompiled
from sources with, I believe, no changes. We stripped a few library
files from the resulting binary (e.g. unicodedata.so) to reduce the
footprint.

I know others have done similar things on hardware in this range,
but I don't know of anything freely available yet. Not sure that
would make sense, either, as everybody uses different hardware in
this sort of thing...

The main advice I can give you from this experience is that if
your hardware doesn't quite cut it, considering putting the money
into better hardware rather than immediately abandoning any thought
of using Python. On the other hand, you have 6K nodes, and we
had less than a few hundred, so the "economies of scale" just
weren't there as much in our case and development costs were
the largest consideration. In your case, saving $100 on each
unit might be more important than saving a few thousand hours
of development time... your call.

-Peter

 

[Christopher T King]

1) Are there any embedded Pythons out there? The nodes will likely be

running some form of Linux, but I don't particularly feel like devoting
resrouces to porting python. Any embedded Linuxes supporting Python?
Thoughts in general?

Python runs fine on ARM-based PDAs running Linux -- see
http://www.vanille.de/projects/python.spy for info and a surprising amount
of ARM binaries. Note that the packages are in ipkg (.ipk) format. These
are similar to Debian's .deb files, and you can get at the contents using
tar a couple of times.

 

[Lars]

Hi,

Python (at least version 1.something) was running fine on my Psion 5mx
(36 Mhz ARM, 16MB RAM). No problem.


Jirka

 

[Alexander May]

You mention network, so presumably you have Ethernet (?) interfaces, and

plan on TCP or UDP...

This is the very start of the project, so very little is locked down. Most
likely we will have Ethernet interfaces and a standard protocol stack, but
we are exploring some mesh networking options such as zigbee.

I doubt you'll need to do much "porting". Unless the Linices you
are considering are quite unusual, they ought to look a lot like any
other Linux, and Python may well work largely unchanged.

One of my concerns is the lack of confidence I'd have in the build. Any
hard to diagnose error that arose could potentially be a logic error or an
interpreter bug (or a hardware bug). On a six thousand node distributed
system, I want to be quite sure of the foundation, and minimize possible
sources of error. I've never compiled python before. Is the test suite
comprehensive enough to justify a high level of confidence in a new build?

An unexplored suggestion was to use Jython and target an embedded chip
designed to accelerate Java bytecode. I know little about Jython or Java
chips, so I can't yet make any sense of this idea.

... rather than immediately abandoning any thought
of using Python.

No chance of that! I agree that the productivity benefit is a very strong
incentive. We are price constrained per unit, so the hardware can't get too
expensive. Your experience is encouraging.in that regards.

Thanks for the info,
Alex

 

[Phil Frost]

If you prototype in Python and find the performance to be inadaquate, a
good option might be Pyrex
<http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>.

 

[Alexander May]

Thanks. Have you had good experiences with this distro?

I also found this distro linked from riverbank's defunct distro. Do you
know if this continuation of that distro, or an entirely new one?

Alex

 

[Alexander May]

PS:

I doubt you'll need to do much "porting". Unless the Linices you
are considering are quite unusual, they ought to look a lot like any
other Linux, and Python may well work largely unchanged.

from http://www.vanille.de/projects/python.spy (from Christopher T King's
post)

"You won't find many Python distributions for arm-linux, because Python is
pretty finnicky when it comes to cross-compiling. The build process of
Python compiles a core part of it (the parser generator pgen) and tries to
execute that later in the build process. This - of course - doesn't work for
cross compiling. My patches to the Python build scripts are available @
sourceforge and are likely to be included into the main Python tree."

-----

Thoughts?

 

[Peter Hansen]

> Peter Hansen recklessly declaimed(*):

from http://www.vanille.de/projects/python.spy (from Christopher T King's
post)

"You won't find many Python distributions for arm-linux, because Python is
pretty finnicky when it comes to cross-compiling. The build process of
Python compiles a core part of it (the parser generator pgen) and tries to
execute that later in the build process. This - of course - doesn't work for
cross compiling. My patches to the Python build scripts are available @
sourceforge and are likely to be included into the main Python tree."

Thoughts?

Yeah, my Linux experience is limited to x86 systems, and my definition
of "unusual" therefore includes anything outside of that. Sorry. :-(

-Peter

 

[Peter Hansen]

One of my concerns is the lack of confidence I'd have in the build. Any
hard to diagnose error that arose could potentially be a logic error or an
interpreter bug (or a hardware bug). On a six thousand node distributed
system, I want to be quite sure of the foundation, and minimize possible
sources of error. I've never compiled python before. Is the test suite
comprehensive enough to justify a high level of confidence in a new build?

You can't necessarily have confidence in someone else's build either,
unless it's very widely used, and used in the same way yours will be
used. Otherwise you are likely to use functionality that others do
not, and you may still encounter problems.

The test suite is extensive. Personally, I would have pretty high
confidence in any working port, after some basic testing with sample
code that reflects the sort of thing my application needs to do.

I am, however, pretty used to tracking down compiler bugs and such,
so I'm a little blase about that sort of thing.

An unexplored suggestion was to use Jython and target an embedded chip
designed to accelerate Java bytecode. I know little about Jython or Java
chips, so I can't yet make any sense of this idea.

My only comment on that is that mixing many different technologies
together will increase the complexity exponentially. I'd try to
avoid it, at the least for the prototype. Note also that I'm a very
YAGNI(*) sort now, what with Extreme Programming ideas having seeped
into my head so far. I tend to believe I'll find ways around any
issues that arise, and generally do...

-Peter

 

[Brad Clements]

PS:


from http://www.vanille.de/projects/python.spy (from Christopher T King's
post)

"You won't find many Python distributions for arm-linux, because Python is
pretty finnicky when it comes to cross-compiling. The build process of
Python compiles a core part of it (the parser generator pgen) and tries to
execute that later in the build process. This - of course - doesn't work for
cross compiling. My patches to the Python build scripts are available @
sourceforge and are likely to be included into the main Python tree."

Don't worry about this. I've ported Python to a 68K board (8 meg ram, 4 meg
flash) running uClinux. That was a squeeze, due to uClinux limitations I had
to statically link in libxml2, libxslt and other goodies I needed.

It worked, and the build process was easy to tidy up (once you know how it
works). I cross compiled from RH9 (386) using gcc tools. Put the binaries on
NFS and tested from the 68K box. Later, burned a flash rom image with the
binaries.

 

[Alexander May]

You can't necessarily have confidence in someone else's build either,

unless it's very widely used, and used in the same way yours will be
used. Otherwise you are likely to use functionality that others do
not, and you may still encounter problems.

Good point.

My only comment on that is that mixing many different technologies
together will increase the complexity exponentially. I'd try to
avoid it, at the least for the prototype. Note also that I'm a very

I wasn't terribly fond of it either and had not invested any time exploring
it. Still I was curious on this newsgroups experience.

Note also that I'm a very
YAGNI(*) sort now

I'm just the lazy sort, which seems to lead to the same result.

Thanks again,
Alex

 

[Alexander May]

What distro are you running?
Thanks,
Alex

Hi,

Python (at least version 1.something) was running fine on my Psion 5mx
(36 Mhz ARM, 16MB RAM). No problem.


Jirka

 

[Christopher T King]

Thanks. Have you had good experiences with this distro?

I've used in on my Zaurus under the OpenZaurus Linux build (with which
it's included), and haven't had any problems, but I haven't used it that
extensively, either.

I also found this distro linked from riverbank's defunct distro. Do you
know if this continuation of that distro, or an entirely new one?

I'm not sure. My guess is it's not, since it seems to have been around
independently for quite a few years now (the visible news items go back to
2002).