I am wondering what is the best way of organizing python source code
in a large projects. There are package code, testing code. I'm
wondering if there has been any summary on previous practices.
(Sorry for the late reply.)
My advice: Don't write big projects. Not that Python can't do it, but
because it's a bad idea in general. Instead, write lots of small,
independent projects that have well-defined, simple interfaces and
work together.
First step is to map out what components you have and what their
interfaces are. (The interface is what functions and classes exist and
what their behavior is.) Nail this down early on.
Try to reuse components from other projects. If something exists but
it isn't up to par with what you need, adapt your interface
requirements or consider contributing to the project.
Next step is to build your scaffolding. For each component you'll be
building, build up the interface and leave out the implementation. If
possible, put in something reasonable but incorrect. For instance:
def twice(x): return 1
If that's not possible:
def twice(x): raise NotImplementedError
(If you think of 'twice' as something complicated like
'standard_deviation', then you'll appreciate this more.)
Next, get the stack "working". Although it will be incorrect, it
should do something visible.
Finally, flesh out the scaffolding by replacing the wrong
implementations with correct ones, working from basic components to
higher level components. Keep the stack working at all times.
When writing unit tests, test only the code in the component and not
code from another component.
The structure of individual components or modules should be standard,
not just for your project but for the vast majority of projects out
there. Use paster to get your code setup. This will give you a good
standard template to work from. Create a virtualenv to get your
working environment. Then run "python setup.py develop" from each of
the modules to install them in the working environment. If you have
the dependencies mapped out, it should install all the stuff you need.
In the end, you should get something that looks more like Pylons and
less like Django. Projects built in this way tend to have more
replaceable components and the components tend to be more useful
outside of the project. That means you write less code and get more
work done.
