Development tools and practices for Pythonistas

[rusi]

I actually use rcs in Windows. Needs a little setup, but works great,
from Emacs VC-mode too.

Where do you get it?
[What google is showing seems to be about 10-15 years old]

 

[Anssi Saari]

I actually use rcs in Windows. Needs a little setup, but works great,
from Emacs VC-mode too.

Where do you get it?
[What google is showing seems to be about 10-15 years old]

As far as I know, RCS hasn't been updated since 5.7 which is about 10
years old now. Linux distributions also package the same version. I
use the stuff from rcs57pc1.zip, at ftp://ftp.cs.purdue.edu/pub/RCS/

The package includes also comparison tools cmp, diff, diff3, sdiff as
win32 versions. I suppose one would need to recompile if 64-bit
versions were needed.

The setup I mentioned was just setting RCSINIT to -x,v although I
don't remember now why I needed that.

 

[Jonathan Hartley]

I appreciate any advice or guidance anyone has to offer.

The 'Python Project HOWTO' gives good advice in terms of setting up a
new project, what files and directories to create, what to put in
version control, etc:

http://infinitemonkeycorps.net/docs/pph/

Also, where I work we have tried many IDEs, but happily and
productively use GVim and very little else, so don't feel you *have*
to use an IDE.

Best regards,

Jonathan Hartley

 

[Tim Golden]

The 'Python Project HOWTO' gives good advice in terms of setting up a
new project, what files and directories to create, what to put in
version control, etc:

http://infinitemonkeycorps.net/docs/pph/

I'd forgotten about that. Great resource! Thanks for reminding me...

TJG

 

[rusi]

I'd forgotten about that. Great resource! Thanks for reminding me...

TJG

Thanks for that link.

There is one question in this regard that is not covered: package-use
Of course this http://infinitemonkeycorps.net/docs/pph/#id10 is there.
But I am talking of setting up one's python environment.

For example on a linux box how to best optimize using the native
package manager (eg apt/rpm) along with packages from pypi. And when
one needs to run with multiple pythons how to use virtualenv etc

 

[rusi]

I'm not a Pythonista, but I aspire to be.

My current tools:

Python, gvim, OS file system

My current practices:

When I write a Python app, I have several unorganized scripts in a
directory (usually with several named test1.py, test2.py, etc., from
random ideas I have tested), and maybe a todo.txt file. Then I hack
away, adding features in a semi-random order. Then I get busy with
other things. Maybe one week I spend 20 hours on development. The next
week, no time on development. A few weeks later when I have some time,
I'm excited to get back to making progress, only to find that I have
to spend 30-60 minutes figuring out where I left off. The code is
usually out of sync with todo.txt. I see people who release new
versions and bug fixes, so I sometimes will create a new directory and
continue working from that copy, because it seems like the thing to
do. But if I ever made something worth releasing, and got a request
like, "I have problems with the 2.0 version. Can you send me the old
1.1 version?" I'd be like, "uhhh... let me hunt through my files by
hand and get back to you in a month". I'm thinking I can do a lot
better than this.

I am aware of tools like version control systems, bug trackers, and
things like these, but I'm not really sure if I need them, or how to
use them properly. I think having some organization to all of this
would help me to make more consistent progress, and spend less time
bringing myself up to speed after some time off.

I really like the idea of having a list of features, and tackling
those features one at a time. I read about people who do this, and
each new features gets a new minor version number. It sounds very
organized and clean. But I'm not really sure of the best way to
achieve this. Mainly I think I just need some recommendations to help
create a good mental map of what needs to happen, and mapping jargon
to concepts. Like, "each feature gets its own directory". Or with a
version control tool, I don't know if a feature maps to a branch, or a
commit?

I appreciate any advice or guidance anyone has to offer.

Recently saw this: [Disclaimer: Not tried]

http://labs.creativecommons.org/2010/11/10/bridging-public-bugtrackers-and-local-tasklists/

 

[Roy Smith]

[standard tale of chaotic software development elided]

I am aware of tools like version control systems, bug trackers, and
things like these, but I'm not really sure if I need them, or how to
use them properly.

None of this has anything to do with python. It's all standard software
engineering stuff that's the same with any language or technology.

Bug trackers are essential for large projects. For a one-man project,
it's probably more effort than it's worth. For the project I'm on now
(4 developers co-located with the customer), we're using a shared google
docs spreadsheet for bug tracking. It's free, low-overhead, and works
well enough. For a single person, even that may be more than you need.

On the other hand version control is essential from day zero, even on a
one-man project. There's plenty of perfectly good, free, systems out
there. The obvious choices are hg, git, and bzr. Pick one and use it.
The nice thing about them all is there's no lock-in. If you decide you
don't like the one you're using, you can easily convert your entire code
repository to any of the others without losing anything.

I really like the idea of having a list of features, and tackling
those features one at a time.

Yup, that's the core concept of all the agile development processes that
are all the rage these days. Sometimes called "release early and often".

I read about people who do this, and
each new features gets a new minor version number.

For the project I'm on, we don't even bother with version numbers. We
have a main branch which we (try to) keep stable and shippable at all
times, and push that to production whenever we've completed a feature
(or bug fix) that the customer needs. I see we're currently running:

9be3fc6a0e01cf128f63d1af2b19c180fb4eaacb (tip)

Version numbers make more sense with a traditional ("waterfall") type
process, where you design a bunch of stuff, write the code, go through a
testing and bug-fixing cycle, and the finally push it out the door.

I think I just need some recommendations to help
create a good mental map of what needs to happen, and mapping jargon
to concepts. Like, "each feature gets its own directory". Or with a
version control tool, I don't know if a feature maps to a branch, or a
commit?

Well, start from the premise that you have a main branch which is always
shippable. That means the code runs and passes all the tests after
every single commit to that branch.

Now, when you need to make a change (add a feature or fix a bug), ask
yourself if the change is small enough that you can do all the work
required in one day. If so, then doing it in a single commit on your
main branch may make sense. If not, then spin up a development branch,
do the work there, and when you're satisfied it's shippable, merge it
onto your main branch.

 

[rusi]

Sorry for a silly subject change: A better one will be welcome -- cant
think of a name myself.

There is this whole area of python that may be called the non-
programming side of programming:

Is there some central site where all such is put up?
What if any should such a bundle of things be called?

-------------------------------------------------

| Area | Tool(s) |
|------------------+------------------------|
| packaging | distutils, setuptools, |
| | distutils2, distribute |
| | Native tools (eg apt) |
| versioning | hg, git, bzr |
| multiple pythons | virtualenv |
| ?? | tox |
| testing | unittest, nose, pytest |
| build | scons, make... |
| deployment | fabric |

------------------------------
* Primary Development tools/aids

1. Help
2. Completion ('intellisense')
3. Tags (Jumping)
4. Refactoring
5. Integration with 'non-programming' above (eg VCSes, packagers
etc)

* Other Development Tools
- Debugger
- Profiler
- Heap Profiler
- Coverage

 

[Terry Reedy]

Sorry for a silly subject change: A better one will be welcome -- cant
think of a name myself.

Associated tools. I might separate them into development tools (up to
the production of python.exe) and usage tools (everything thereafter).
On Windows, this is a pretty clean separation. On Linux, less so since
users sometimes build their own binaries and therefore use some of the
development tools.

Assuming that there is not one already, this could be the beginning of a
useful overview wiki page with links to existing pages on the specific
topics ('areas') listed below.

There is this whole area of python that may be called the non-
programming side of programming:

Is there some central site where all such is put up?
What if any should such a bundle of things be called?

-------------------------------------------------

| Area | Tool(s) |
|------------------+------------------------|
| packaging | distutils, setuptools, |
| | distutils2, distribute |
| | Native tools (eg apt) |
| versioning | hg, git, bzr |
| multiple pythons | virtualenv |
| ?? | tox |
| testing | unittest, nose, pytest |
| build | scons, make... |
| deployment | fabric |

------------------------------

I would reorder this list in the typical order used, starting with editors.

 

[Steven D'Aprano]

Sorry for a silly subject change: A better one will be welcome -- cant
think of a name myself.

There is this whole area of python that may be called the non-
programming side of programming:

Is there some central site where all such is put up? What if any should
such a bundle of things be called?

Documentation.

Check the Python wiki.

 

[rusi]

Associated tools. I might separate them into development tools (up to
the production of python.exe) and usage tools (everything thereafter).
On Windows, this is a pretty clean separation. On Linux, less so since
users sometimes build their own binaries and therefore use some of the
development tools.

Can you elaborate? I dont understand

Assuming that there is not one already, this could be the beginning of a
useful overview wiki page with links to existing pages on the specific
topics ('areas') listed below.






I would reorder this list in the typical order used, starting with editors.

Some more 'areas':
1. Which python 'form' does one use?
At the least python vs pythonw on windows.

But more generally scripting vs REPL. In REPL python vs ipython
Note 1. I am often unnerved by how even experienced python programmers
think that the only way to 'do' python is like C -- write a main, not
appreciating the luxury of an unstructured, exploratory mode that an
REPL makes possible.
Note 2. ruby makes this distinction more obvious by distinguishing the
scripting engine -- ruby -- form the interactive interpreter (REPL) --
irb.

2. Literate Programming: When the primary purpose of the program is
not the program but (some form of) discussion around it

3. Program namespace lookup and structuring: sys.path is the interior
program view but there is also the 'exterior' view -- PYTHONPATH, .pth
files etc.

Finally some thoughts on how to name this list of areas:
a. Software Engineering? : Inasmuch as real program development is
programming + 'something-else' and SE is that 'something else'
b. Python Development Environment? Similar to above

[Cannot say I like these names too much but at least its more specific
than Steven's vanilla 'documentation' ]