Python for Large Projects

[Ixokai]

Hello all.

I've been a long time Python fan, and have fairly recently (with the
support of a coworker who surprised me by mentioning my pet language one
day) convinced my company to begin the colossal task of basically rewriting
all of our software in Python. Woohoo.
Previously we used a few different development environments, mostly
Borland, for different products in our 'system' of thick clients sort of
operating with eachother as they dug against a database. The whole thing is
woefully out of date at this point, and platform-dependant, so we needed to
switch to something.. and Python won. Yay.
In the new platform, we'll be building a distributed system with CORBA
(omniORB + omniORBpy, etc) with Python making up most of our development for
the new edition of our software. Woohoo. Much fun.

My question is not 'how suitable is Python for large projects?', but
instead more, 'Do you have any advice for large projects with Python?'.

For example, our system will be made up of numerous applications --
three or four 'servers', multiple clients, all talking to eachother, sharing
a great deal of code. Any suggestions for organization? Our current plan is
to put everything under the 'site-packages' tree of a Python installation we
provide/control, with a fairly deeply nested tree:

apt
apt\library
apt\library\corba.py
apt\library\database\*
apt\library\controls\*
apt\application\globalapp1\
apt\application\globalapp2\
apt\system\editorial\application\client\
apt\system\editorial\application\client2\
apt\system\editorial\library\elementgrid.py

.... etc. You get the idea, I hope. We have apt.library where we store all of
our 'global', 'shared' libraries that everything can make use of, and
apt.application with some global, shared apps. Then we have 'systems', which
are a collection of applications and libraries in a single domain. These
mostly get sold as a single product, really. Etc, etc.

I end up doing a lot of:

from apt.library.corba import *
from apt.library.database import Connection
from apt.library import machine
from apt.system.editorial.library.elementgrid import ElementGrid

... And such. When PEP328 gets in, this may become more attractive, but
occasionally I'm alarmed by the depth of the whole thing. But I also want to
keep stuff logical and separated.

And how about distribution? Freezing and such won't work, because of the
shared nature of most of this code. Currently I'm probably planning on
looking into finding the build scripts for python, win32all, and wxPython
and seeing if I can combine them into one single install. Don't know if
that's the best thing to do or not, but expecting my customers to install
this stuff... is expecting too much of them.

For example: Traditionally my company has avoided installing software on
a users local machine. Instead we install to a file server and have everyone
run the software from there. A bit of a performance hit but the advantage of
administration is very important, because our customers do not necessarily
have a very strong IT staff. There is some concern that if we do this same
thing with Python, the performance hit of loading an interpreted environment
over the shared filesystem will be prohibitive. Some basic testing has made
this seem to be the case.

hmm. That's all I have for now. I think.

Thanks in advance.

--Stephen

 

[Peter Hansen]

There is some concern that if we do this same
thing with Python, the performance hit of loading an interpreted environment
over the shared filesystem will be prohibitive. Some basic testing has made
this seem to be the case.

Just focusing on this one point: it doesn't make sense to me that
this would be the case, and in my experience it is not. The
interpreter (python23.dll for example on Win32) is less than 1MB
and that's basically _tiny_ these days. We always had a single
shared instance of Python and never noticed more than a small
fraction of second delay compared to loading it locally. This
was a network with only about fifteen developers as users, and
100MBps throughout, but I would think almost any program these
days is going to be 1MB or more, so I can't see that Python has
any disadvantage in this area.

-Peter

 

[john fabiani]

Just focusing on this one point: it doesn't make sense to me that
this would be the case, and in my experience it is not. The
interpreter (python23.dll for example on Win32) is less than 1MB
and that's basically _tiny_ these days. We always had a single
shared instance of Python and never noticed more than a small
fraction of second delay compared to loading it locally. This
was a network with only about fifteen developers as users, and
100MBps throughout, but I would think almost any program these
days is going to be 1MB or more, so I can't see that Python has
any disadvantage in this area.

-Peter

I'm a newbie so take what I say with a grain of salt!
In the VFP (visual foxpro) world the runtime environment is loaded on
the local computer (client) and the program is often loaded on to the
server. I'm wondering if there would be any advanage to placing the
python runtime on the clients along with the standard modules. Then
placing the program code on to a server where all the clients/users
would access the code. It would seem that the advanage would be that
program load time would about the same but the excutions would be faster
because of the local runtime. Can this even been done? Would it be faster?

John

 

[Jorge Godoy]

I'm a newbie so take what I say with a grain of salt!
In the VFP (visual foxpro) world the runtime environment is loaded on
the local computer (client) and the program is often loaded on to the
server. I'm wondering if there would be any advanage to placing the
python runtime on the clients along with the standard modules. Then
placing the program code on to a server where all the clients/users
would access the code. It would seem that the advanage would be that
program load time would about the same but the excutions would be faster
because of the local runtime. Can this even been done? Would it be
faster?

If I understood you correctly, this is what we do on our clients. It also
makes it easier to make bugfixes, updates, etc. in our code.

 

[john fabiani]

If I understood you correctly, this is what we do on our clients. It also
makes it easier to make bugfixes, updates, etc. in our code.

So it can be done. I'll have to learn a little more about path-ing.
Thanks
John

 

[Ixokai]

Just focusing on this one point: it doesn't make sense to me that
this would be the case, and in my experience it is not. The
interpreter (python23.dll for example on Win32) is less than 1MB
and that's basically _tiny_ these days. We always had a single
shared instance of Python and never noticed more than a small
fraction of second delay compared to loading it locally. This
was a network with only about fifteen developers as users, and
100MBps throughout, but I would think almost any program these
days is going to be 1MB or more, so I can't see that Python has
any disadvantage in this area.

These are GUI programs and we've decided to use wxPython, so that may be
adding to it. I will do more testing to see if it was just a fluke the last
time I tried it out. It might have been

--Stephen

 

[Scott David Daniels]

>
These are GUI programs and we've decided to use wxPython, so that
> may be adding to it. I will do more testing to see if it was just
a fluke the last time I tried it out. It might have been

One thing to investigate is file openings. If you combine all of
python (and in your case wxPython's) .pyc files into a single zip
wish an actual directory structure matching the packages you want to
use, you will only have to open the zip file once, rather than pull
each module in in several file operations.

 

[David Fraser]

Hello all.

I've been a long time Python fan, and have fairly recently (with the
support of a coworker who surprised me by mentioning my pet language one
day) convinced my company to begin the colossal task of basically rewriting
all of our software in Python. Woohoo.
Previously we used a few different development environments, mostly
Borland, for different products in our 'system' of thick clients sort of
operating with eachother as they dug against a database. The whole thing is
woefully out of date at this point, and platform-dependant, so we needed to
switch to something.. and Python won. Yay.
In the new platform, we'll be building a distributed system with CORBA
(omniORB + omniORBpy, etc) with Python making up most of our development for
the new edition of our software. Woohoo. Much fun.

My question is not 'how suitable is Python for large projects?', but
instead more, 'Do you have any advice for large projects with Python?'.

For example, our system will be made up of numerous applications --
three or four 'servers', multiple clients, all talking to eachother, sharing
a great deal of code. Any suggestions for organization? Our current plan is
to put everything under the 'site-packages' tree of a Python installation we
provide/control, with a fairly deeply nested tree:

apt
apt\library
apt\library\corba.py
apt\library\database\*
apt\library\controls\*
apt\application\globalapp1\
apt\application\globalapp2\
apt\system\editorial\application\client\
apt\system\editorial\application\client2\
apt\system\editorial\library\elementgrid.py

... etc. You get the idea, I hope. We have apt.library where we store all of
our 'global', 'shared' libraries that everything can make use of, and
apt.application with some global, shared apps. Then we have 'systems', which
are a collection of applications and libraries in a single domain. These
mostly get sold as a single product, really. Etc, etc.

I end up doing a lot of:

from apt.library.corba import *
from apt.library.database import Connection
from apt.library import machine
from apt.system.editorial.library.elementgrid import ElementGrid

... And such. When PEP328 gets in, this may become more attractive, but
occasionally I'm alarmed by the depth of the whole thing. But I also want to
keep stuff logical and separated.

Well you don't neccessarily have to have everything under apt... You
just need to have unique package names.
Also the cleaner your system design the less you should have to refer to
bits inside other packages.

And how about distribution? Freezing and such won't work, because of the
shared nature of most of this code. Currently I'm probably planning on
looking into finding the build scripts for python, win32all, and wxPython
and seeing if I can combine them into one single install. Don't know if
that's the best thing to do or not, but expecting my customers to install
this stuff... is expecting too much of them.

On the wxPython users list we are planning something called wxPRE which
would be an environment pre-setup with Python and wxPython. You may want
to help set this up as it is very similar to your requirements.
See http://wiki.wxpython.org/index.cgi/wxPRE

David