code packaging

[Paul Rubin]

I've been through this kind of thing a few times in the past and
received excellent advice here on clpy about how to deal with specific
technical aspects (e.g. how to use setuptools, InnoSetup, etc).

I'm now wondering where this type of thing is best addressed in more
general terms. What I usually see happening, in projects I've worked
on and in others, is that developers get the code working on their own
computers, using source control tools and (if we're lucky) tests
developed in parallel with the code, but without much attention to
final packaging until the very end when the code is about to be
shipped.

Is this appropriate? Inappropriate? Do your projects start using
serious packaging and distribution tools very early in development,
before the code is anywhere near usable? Should they?

I'm involved in something that I wish was that way, mainly because I
really don't have any way to put a complete installation on my own
computers; I have to develop my subsystem more or less independently
of the overall app, then toss stuff over the wall to the more central
application guys for integration, which gets slow sometimes because
we're distributed all over. This is a fairly complex server app that
depends on a lot of external packages and programs running across
multiple machines, so I could see the packaging problem being messy.
It would cause a significant interruption in development to try to
package things so that the non-central developers could install and
hack it without a lot of assistance. But I think it would be
worthwhile given how much easier it would become to test changes. We
do plan to bundle up and release the code sometime (maybe as a Debian
package) but for now that's deferred until the code is more complete
and stable.

I'm wondering how other projects go about this.

 

[Miki]

Hello Paul,

I'm now wondering where this type of thing is best addressed in more
general terms. What I usually see happening, in projects I've worked
on and in others, is that developers get the code working on their own
computers, using source control tools and (if we're lucky) tests
developed in parallel with the code, but without much attention to
final packaging until the very end when the code is about to be
shipped.

[snipped]

IMO you should invest the time a build automated packaging and some
smoke tests for the product.
After this is done, start using some "continuous automation" tools
(like http://cruisecontrol.sourceforge.net/, http://buildbot.net/trac
and others).

HTH,

 

[Ben Finney]

Is this appropriate? Inappropriate? Do your projects start using
serious packaging and distribution tools very early in development,
before the code is anywhere near usable? Should they?

Yes. Test-driven development has taught me that putting off important
parts of the development process until the end is rarely a good
idea. So, I try to make a point of writing the first features inside a
complete unit test environment, and inside a complete
distribution/build environment. The tests are run continuously while
developing the code, and the distribution/build process is tested
manually.

I'm looking into the "build-bot" approach used to ensure that the
build process is also run automatically and continually during
development, just like the unit tests <URL:http://buildbot.net/>; but
so far I'm running the build manually.

 

[Ryan Ginstrom]

On Behalf Of Paul Rubin

I'm wondering how other projects go about this.

I develop an automated build system from the very beginning. Running the
build script:
* Creates the API documentation (epydoc)
* Creates the help files (extracting some information from the source)
* Builds a windows executable (py2exe)
* Builds an installer (Inno Setup)
* Runs the installer (AutoIt)
* Runs a smoke test on the installed application (AutoIt)

The program must at least install and pass the smoke tests at any given
point in development.

Another script uploads the installer to my server, and sends an email to the
testers stating that a new version is ready, with a list of
additions/improvements.

Regards,
Ryan Ginstrom

 

[John J. Lee]

I'm now wondering where this type of thing is best addressed in more
general terms. What I usually see happening, in projects I've worked
on and in others, is that developers get the code working on their own
computers, using source control tools and (if we're lucky) tests
developed in parallel with the code, but without much attention to
final packaging until the very end when the code is about to be
shipped.

Is this appropriate? Inappropriate? Do your projects start using
serious packaging and distribution tools very early in development,
before the code is anywhere near usable? Should they?

I'm involved in something that I wish was that way, mainly because I
really don't have any way to put a complete installation on my own
computers; I have to develop my subsystem more or less independently
of the overall app, then toss stuff over the wall to the more central
application guys for integration, which gets slow sometimes because
we're distributed all over. This is a fairly complex server app that
depends on a lot of external packages and programs running across
multiple machines, so I could see the packaging problem being messy.
It would cause a significant interruption in development to try to
package things so that the non-central developers could install and
hack it without a lot of assistance. But I think it would be
worthwhile given how much easier it would become to test changes. We
do plan to bundle up and release the code sometime (maybe as a Debian
package) but for now that's deferred until the code is more complete
and stable.

I'm wondering how other projects go about this.

You can go further: in addition to developing a Debian package (or a
few of them) for your own project, build your OS image using
debootstrap (you'll probably want your own local repository(ies), for
reproducibility). Then you do your development in a virtual machine
or chroot environment. That way you develop in the same (or very
similar) environment to the one in which your code will eventually be
deployed (one difference being the use of an version control checkout
rather a .deb install for the code you're actually working on, unless
troubleshooting problems that are somehow tied up with that particular
aspect of deployment). I've seen this working in a real-world
project.

Of course, you can set up a server that does a complete build-and-test
every night, and a separate continuous integration server that just
does an "svn up" (rather than a full rebuild) every time somebody
checks in, so that test failures can be detected quickly and the
guilty party suitably humiliated


John

 

[Alex Popescu]

I develop an automated build system from the very beginning. Running
the build script:
* Creates the API documentation (epydoc)
* Creates the help files (extracting some information from the source)
* Builds a windows executable (py2exe)
* Builds an installer (Inno Setup)
* Runs the installer (AutoIt)
* Runs a smoke test on the installed application (AutoIt)

The program must at least install and pass the smoke tests at any
given point in development.

Another script uploads the installer to my server, and sends an email
to the testers stating that a new version is ready, with a list of
additions/improvements.

Regards,
Ryan Ginstrom

What "build" tools are available in Python? Any initial hints are highly
appreciated.

bests,
../alex

 

[Alex Popescu]

Hi Alex:

Do you develop for Windows? Are you looking to automate a build
process?

The standard library's build module is distutils:
http://docs.python.org/lib/module-distutils.html

As I mentioned in my post, I use a variety of third-party modules
(epydoc, py2exe, Inno Setup, AutoIt), which I glue together with my
own brew of python scripts. Since I develop mostly for Windows (except
for Web stuff, which I use Linux for), my build process is tailored to
that platform.

Regards,
Ryan Ginstrom

Thanks for following up on this. I am mostly used with a world where
platform dependent builds are not very used (Java).

My current attempt to learn Python is by creating a Python equivalent of
the advanced Java testing framework TestNG (http://testng.org) -- but I
will give more details about this when I'll be starting to have
something more solid .

However, due to my background where builds are required and there are
some de facto build standards (Ant, Maven, etc.) I am starting to think
I will be needing something similar while working on the tool I've
mentioned. At this point I think that what I am interested in are how
distros are build in Python world (no platform specific distros, but
rather generic ones) and ways to automate the process of creating the
distros (and probably running the framework internal tests etc.)

My research lead me to distutils and/or setuptools, respectively SCon
and/or buildutils. So I am wondering if I am looking in the right
direction or do I need to do some more research .

tia,
../alex

 

[Paul Rubin]

You can go further: in addition to developing a Debian package (or a
few of them) for your own project, build your OS image using
debootstrap (you'll probably want your own local repository(ies), for
reproducibility). Then you do your development in a virtual machine
or chroot environment. That way you develop in the same (or very
similar) environment to the one in which your code will eventually be
deployed (one difference being the use of an version control checkout
rather a .deb install for the code you're actually working on, unless
troubleshooting problems that are somehow tied up with that particular
aspect of deployment). I've seen this working in a real-world
project.

Thanks for these suggestions. Buildbot (per Ben Finney) looks
interesting as does the above, and Ryan's notes about experience with
Windows installers is also appreciated. Whether lot of build
automation would have been worth it at the beginning of the thing I'm
working on or not, we probably need to move towards it now that more
of the system is running.