how do "real" python programmers work?

[bblais]

Hello,

Let me start by saying that I am coming from a background using Matlab
(or Octave), and C++. I am going to outline the basic nuts-and-bolts
of how I work in these languages, and ask for some help to find out how
the same thing is done in Python. I am not sure what the standard is.

In C++, I open up an editor in one window, a Unix shell in another. I
write the code in the editor, then switch to the shell window for
compile and run. I then go back to the editor for modifications, and
then compile and run in the shell window.

In Matlab, I do much the same thing, except there is no compile phase.
I have the editor on one window, the Matlab interactive shell in the
other. I often make a bunch of small scripts for exploration of a
problem, before writing any larger apps. I go back and forth editing
the current file, and then running it directly (Matlab looks at the
time stamp, and automagically reloads the script when I modify it).

In Python, there seems to be a couple ways of doing things. I could
write it in one window, and from a Unix shell call
python myscript.py
and be like C++, but then I lose the interactiveness which makes
prototyping easier. If I use the python shell, I can use import (and
reload), or execfile perhaps.

How do experienced python programmers usually do it? Is there a
"usually" about it, or is it up to personal taste? Are there any
convenient ways of doing these things?

I realize this is a pretty newbie question, but it could possibly save
me hours of time if there is a better way to work.


thanks,

Brian Blais

 

[Barbier de Reuille Pierre]

Hello,

Let me start by saying that I am coming from a background using Matlab
(or Octave), and C++. I am going to outline the basic nuts-and-bolts
of how I work in these languages, and ask for some help to find out
how the same thing is done in Python. I am not sure what the
standard is.

In C++, I open up an editor in one window, a Unix shell in another. I
write the code in the editor, then switch to the shell window for
compile and run. I then go back to the editor for modifications, and
then compile and run in the shell window.

In Matlab, I do much the same thing, except there is no compile phase.
I have the editor on one window, the Matlab interactive shell in the
other. I often make a bunch of small scripts for exploration of a
problem, before writing any larger apps. I go back and forth editing
the current file, and then running it directly (Matlab looks at the
time stamp, and automagically reloads the script when I modify it).

In Python, there seems to be a couple ways of doing things. I could
write it in one window, and from a Unix shell call
python myscript.py
and be like C++, but then I lose the interactiveness which makes
prototyping easier. If I use the python shell, I can use import (and
reload), or execfile perhaps.

How do experienced python programmers usually do it? Is there a
"usually" about it, or is it up to personal taste? Are there any
convenient ways of doing these things?

I realize this is a pretty newbie question, but it could possibly save
me hours of time if there is a better way to work.


thanks,

Brian Blais

Well, I think it will depend on your project ...
If you're developing GUI application, you will have trouble using the
python shell. At least you will need a special, customized shell that
will take care of launching the main loop of your GUI and let you enter
commands. But I don't think this is a big help to build GUI ...
Now, what I do for other stuffs (or when I have access to such a
modified shell). First, when I develop an object, I quit and
reload my script after each modification, mainly because problems
arises quickly with different variables with difference version of the
same class and you have no way to exactly know what variable hold what
version of the class. However, when I'm developing algorithms I reload
the modules while keeping the shell launched. Only at the very end do I
quit and relaunch to be 100% sure my algorithm is working with the
new versions of the modules ...

Pierre

 

[sjdevnull]

In Python, there seems to be a couple ways of doing things. I could
write it in one window, and from a Unix shell call
python myscript.py
and be like C++, but then I lose the interactiveness which makes
prototyping easier. If I use the python shell, I can use import (and
reload), or execfile perhaps.

or "python -i myscript.py" which runs the script then leaves you in the
interpreter.

 

[Tom Anderson]

In Matlab, I do much the same thing, except there is no compile phase. I
have the editor on one window, the Matlab interactive shell in the
other. I often make a bunch of small scripts for exploration of a
problem, before writing any larger apps. I go back and forth editing
the current file, and then running it directly (Matlab looks at the time
stamp, and automagically reloads the script when I modify it).

I wouldn't describe myself as an experienced programmer, but this is
definitely how i work - editor plus interactive interpreter, using
import/reload to bring in and play with bits of of code.

Towards the end of coding a program, when i'm done with the inner
functions and am working on the main function, which does stuff like
command line parsing, setting up input and output, etc, i'll often leave
the interpreter and work from the OS shell, since that's the proper
environment for a whole program.

Often, i'll actually have more than one shell open - generally three: one
with an interpreter without my code loaded, for doing general exploratory
programming, testing code fragments, doing sums, etc; one with an
interpreter with my code loaded, for testing individual components of the
code, and one at the OS shell, for doing whole-program tests, firing up
editors, general shell work, etc.

Another trick is to write lightweight tests as functions in the
interpreter-with-code-loaded that reload my module and then do something
with it. For example, for testing my (entirely fictional) video
compressor, i might write:

def testcompressor():
reload(vidzip)
seq = vidzip.ImageSequence((640, 480))
for i in xrange(200):
frameName = "testmovie.%02i.png" % i
frame = Image.open(frameName)
seq.append(frame)
codec = vidzip.Compressor(vidzip.DIRAC, 9)
codec.compress(seq, file("testmovie.bbc", "w"))

Then, after editing and saving my code, i can just enter
"testcompressor()" (or, in most cases, hit up-arrow and return) to reload
and test. You can obviously extend this a bit to make the test routine
take parameters which control the nature of the test, so you can easily
test a range of things, and you can have multiple different test on the go
at once.

tom

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

In Python, there seems to be a couple ways of doing things. I could
write it in one window, and from a Unix shell call
python myscript.py
and be like C++, but then I lose the interactiveness which makes
prototyping easier. If I use the python shell, I can use import (and
reload), or execfile perhaps.

For programs, I usually have two shells: one interactive shell where
I test fragments (usually to find out how some library call needs to
be spelled); and another shell to run the program in.

For libraries, I typically reload the library module in a single
interactive shell.

I often want to do sequences of actions. I first type them one
by one, e.g.

py> import httplib
py> httplib=httplib.HTTP("localhost")
py> import httplib
py> h=httplib.HTTP("localhost")
py> h.connect()
py>

Then, I get tired of fetching all the lines from the history
again and again, and rephrase it (through cut-n-paste) as

py> import httplib;h=httplib.HTTP("localhost");h.connect()

Then I only need to fetch a single line from the history
to redo all the initialization.

I set-up readline so the history survives the end of
the interpreter. I quit the interpreter, restart it,
and immediately have the last command I typed available, see
my attached .pythonrc.

Regards,
Martin

# -*- python -*-
#from __future__ import division
import os, sys
sys.ps1 = 'py> '
histfile = os.path.join(os.environ["HOME"], ".pyhist")
try:
import readline, rlcompleter
if os.path.exists(histfile):
readline.read_history_file(histfile)
readline.parse_and_bind("tab: complete")

import atexit
atexit.register(readline.write_history_file, histfile)

del readline, rlcompleter, atexit

except ImportError:
pass

del os,histfile


try:
help
except NameError:
try:
from pydoc import help
except ImportError:
pass

 

[Dave Hansen]

Hello,

Let me start by saying that I am coming from a background using Matlab
(or Octave), and C++. I am going to outline the basic nuts-and-bolts

I generally write C code for embedded controllers.

of how I work in these languages, and ask for some help to find out how
the same thing is done in Python. I am not sure what the standard is.

[...]

I have an editor I use for writing C code with a button I've
configured to run "lint". When the code lints clean I drop to a shell
and run "make" when I can, fire up the vile IDE for the target
platform when I can't, and build the code. Testing means firing up a
simulator or downloading the code to the target.

How do experienced python programmers usually do it? Is there a
"usually" about it, or is it up to personal taste? Are there any
convenient ways of doing these things?

I've been writing Python for about 6 years, but mostly small utilities
for personal use, so I don't think I'm typical. I think most of my
code would run fine in 1.5.2, because that's what I started learning
on...

I use Idle. I try things out in the interactive shell, open an editor
window to write code. When a script is ready to test, I hit F5 to run
it. I build my systems bit-by-bit, so it's rarely worth my while to
fire up my C editor (though it does support Python syntax coloring and
auto-indenting. It may not be the shiniest toy in the box, but Idle
does what I need it to do, and it's actually a very powerful and
effective tool for writing simple scripts.

Huge multi-module projects are probably another animal entirely.

Regards,
-=Dave

 

[Mike Meyer]

In Python, there seems to be a couple ways of doing things. I could
write it in one window, and from a Unix shell call
python myscript.py
and be like C++, but then I lose the interactiveness which makes
prototyping easier. If I use the python shell, I can use import (and
reload), or execfile perhaps.

How do experienced python programmers usually do it?

I do it both ways - and at least one other. If I'm working on code
that's largely independent of the module, I'll use python-mode's
"execute" commands to send the code to the Python interpreter. If the
code needs the rest of the module, I'll use reload in the interactive
interpreter. In the final stages of module development, I'll have a
test at the end of the module, and switch to the shell to run the
module as a script for testing.

Of course, a lot of what I do is web apps, so the later stages of
testing involve activating a browser and hitting "reload".

Is there a "usually" about it, or is it up to personal taste? Are
there any convenient ways of doing these things?

I think you've found some of the most convenient ones already. Maybe
some of the people who IDEs (instead of - well, we need a term for
development environment built out of Unix tools....) will let us know
what they do.

<mike

 

[Scott David Daniels]

How do experienced python programmers usually do it? Is there a
"usually" about it, or is it up to personal taste? Are there any
convenient ways of doing these things?

There are a lot of us who use a test-first process:
Write a unit test, watch it fail, fix the code til the test passes.
If the project is smallish, I'll have Idle to experiment a bit (as
others have said, for checking boundary cases on standard module
functions).

So:
window A: test file editor
window B: module editor
window C: command line for running tests.
window D: idle for checking boundary conditions on Python modules.

My typical cycle is change file (either window A or B), save that,
re-run the unit test in window C. Every now and then (when test are
succeeding), I check in the sources (and the tests). Before I was
doing this, I explored my module with Idle, but that often left me
with too few test cases.


--Scott David Daniels
(e-mail address removed)

 

[sandravandale]

I'm not an experienced python programmer, but I come from a C++
background as well. I like to code in Komodo ($29 for the personal
edition) and that lets me have multiple python files opened in tabs,
and multiple interpreters opened below, since the interpreter is
command based, it doesn't have to be very tall, and I find this works
great (the debugging windows also appear down at the bottom, which is
convienient. The debugger alone is worth the $29, within a week or two
of serious development and you've paid for the software in terms of
saved time.) Komodo runs on windows and linux.

This leaves my second screen free, where I keep all my web browsers,
with comp.lang.python, gmail, and all the documentation I need.

I also keep a macros.py file in a place where it's in the python path,
and keep it open in Komodo. Anything I find myself typing over and over
in the interpreters, I simply make into a function in the macros file
and then reload the macros module and call it. I work with mod_python a
lot, so I have a macro that reads in the last 10 lines or so of the
apache error log, removes extraneous information and prints them, huge
timesaver.

-Sandra

 

[Ben Finney]

There are a lot of us who use a test-first process:
Write a unit test, watch it fail, fix the code til the test passes.

I'm in this camp, but because I use Emacs I have an IDE:

- multiple edit buffers for the Python code (any good editor can do
this)

- from any Python code buffer, C-c C-c to execute

- output shown in a new buffer, automatically jump to errors in code

- separate buffer with a live Python prompt to experiment

- separate buffer browsing documentation if needed

- edit buffer for version control commit message

That's all a single Emacs session in one window of GNU screen; in a
separate window is a regular shell session where I can run other
things outside Emacs if I choose.

I'm currently using a version control system that doesn't integrate
with Emacs, so I have another shell buffer for that, plus an Emacs
buffer for the commit message. Ideally, I'd use Emacs native features
for version control too.

Another shell session constantly runs the project's unit tests (with a
'nice' setting so it doesn't get in the way of anything
process-intensive I might do elsewhere). I can quickly check the
pass/fail state of all the tests just by switching to that window.

All of the above could be done using many more screen sessions, but
Emacs buffers are more accessible when you're already editing.

 

[Travis E. Oliphant]

In Python, there seems to be a couple ways of doing things. I could
write it in one window, and from a Unix shell call
python myscript.py
and be like C++, but then I lose the interactiveness which makes
prototyping easier. If I use the python shell, I can use import (and
reload), or execfile perhaps.

Try IPython. It makes the process of executing live code very productive.

http://ipython.scipy.org

-Travis

 

[Linsong]

bblais wrote:



Try IPython. It makes the process of executing live code very productive.

http://ipython.scipy.org

-Travis

I also strongly recommend IPython, it provide a lots of very amazing
functionality and it is worth spending 1h- to learn it. Go to the
website and have a look about the key features and I am very sure it
will help you a lot!

BR
Linsong

 

[bruno at modulix]

Hello,
(snip)

In C++, I open up an editor in one window, a Unix shell in another. (snip)
In Matlab, I do much the same thing, except there is no compile phase. (snip)
In Python, there seems to be a couple ways of doing things. I could
write it in one window, and from a Unix shell call
python myscript.py
and be like C++, but then I lose the interactiveness which makes
prototyping easier. If I use the python shell, I can use import (and
reload), or execfile perhaps.

How do experienced python programmers usually do it?

I'm not sure I qualify as a "experienced", but...

Is there a
"usually" about it, or is it up to personal taste?

Mostly. There's no shortage of Python IDE/editors/...

Are there any
convenient ways of doing these things?

<my-2-cents>
Try emacs + python-mode. It offers the best editor/interactive shell
I've ever seen.
</my-2-cents>

 

[bruno at modulix]

(snip)


Well, I think it will depend on your project ...
If you're developing GUI application, you will have trouble using the
python shell. At least you will need a special, customized shell that
will take care of launching the main loop of your GUI and let you enter
commands.

Well, most of your code should be runnable/testable without the GUI
part, so that may not be such a big problem.

(snip)

 

[Alan J. Salmoni]

Hi Brian,

I'm sure I don't qualify as an "experienced Python programmer", but I
write lots of code usually for statistical analysis (via numarray) or
for GUI work.

The way I work is to use an editor and Idle for interactive testing of
small routines.

My choice of editor is SciTE. I'll have different modules (both working
and testing modules) in a range of different tabs and use the built-in
execute code function (F5) to run and check errors. Idle is then used
to test small functions and explore when I need to. The editor gets
more of the GUI work and Idle gets more of the statistics work - the
ability to test routines on the fly is enormously helpful and quite a
productive way to work.

Say I was looking at some linear algebra and I want an SSCP matrix.
Just enter the data into variable 'X' then:

and there's my matrix. However, the useful thing is that the transposed
X matrix is still there so if I go on to a multiple regression, I can
use it there too. For statistics, it's tremendously useful,
particularly if the analysis doesn't go as planned (eg, presence of
outliers) and further exploration is needed. Perhaps later I need the
variances of each variable:

And out they appear without having to go through an entire program.
Even though Python is slower than well-written Fortran, the interactive
nature can make statistical analysis quicker.

All the best!

Alan.

 

[Juho Schultz]

In Python, there seems to be a couple ways of doing things. I could
write it in one window, and from a Unix shell call
python myscript.py
and be like C++, but then I lose the interactiveness which makes
prototyping easier. If I use the python shell, I can use import (and
reload), or execfile perhaps.

How do experienced python programmers usually do it? Is there a
"usually" about it, or is it up to personal taste? Are there any
convenient ways of doing these things?

Disclaimer: I do not consider myself "an experienced python programmer".

For scripts and small programs (one file, one developer) I use emacs and
python shell. For larger projects I use Eclipse and pydev.

 

[Michele Simionato]

As many others, I use emacs for programming and ipython for interactive
experiments.

Michele Simionato

 

[bruno at modulix]

Mike Meyer wrote:
(snip)

Maybe
some of the people who IDEs (instead of - well, we need a term for
development environment built out of Unix tools....)

"Extegrated Development environment" ?-)

 

[Fuzzyman]

bblais wrote:
[snip..]

In Python, there seems to be a couple ways of doing things. I could
write it in one window, and from a Unix shell call
python myscript.py
and be like C++, but then I lose the interactiveness which makes
prototyping easier. If I use the python shell, I can use import (and
reload), or execfile perhaps.

How do experienced python programmers usually do it? Is there a
"usually" about it, or is it up to personal taste? Are there any
convenient ways of doing these things?

I realize this is a pretty newbie question, but it could possibly save
me hours of time if there is a better way to work.

SPE (the IDE) has several nice ways of running code - including
launching it as an external file.

This is what I use for almost all my development.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

 

[Roy Smith]

we need a term for development environment built out of Unix tools....

We already have one. The term is "emacs".