How to use Python well?

[snorble]

I use Python a lot, but not well. I usually start by writing a small
script, no classes or modules. Then I add more content to the loops,
and repeat. It's a bit of a trial and error learning phase, making
sure I'm using the third party modules correctly, and so on. I end up
with a working script, but by the end it looks messy, unorganized, and
feels hacked together. I feel like in order to reuse it or expand it
in the future, I need to take what I learned and rewrite it from
scratch.

If I peeked over a Python expert's shoulder while they developed
something new, how would their habits differ? Do they start with
classes from the start?

I guess I'm looking for something similar to "Large Scale C++ Software
Design" for Python. Or even just a walkthrough of someone competent
writing something from scratch. I'm not necessarily looking for a
finished product that is well written. I'm more interested in, "I have
an idea for a script/program, and here is how I get from point A to
point B."

Or maybe I'm looking for is best practices for how to organize the
structure of a Python program. I love Python and I just want to be
able to use it well.

 

[Kurt Smith]

I use Python a lot, but not well. I usually start by writing a small
script, no classes or modules. Then I add more content to the loops,
and repeat. It's a bit of a trial and error learning phase, making
sure I'm using the third party modules correctly, and so on. I end up
with a working script, but by the end it looks messy, unorganized, and
feels hacked together. I feel like in order to reuse it or expand it
in the future, I need to take what I learned and rewrite it from
scratch.

If I peeked over a Python expert's shoulder while they developed
something new, how would their habits differ? Do they start with
classes from the start?

I guess I'm looking for something similar to "Large Scale C++ Software
Design" for Python. Or even just a walkthrough of someone competent
writing something from scratch. I'm not necessarily looking for a
finished product that is well written. I'm more interested in, "I have
an idea for a script/program, and here is how I get from point A to
point B."

Or maybe I'm looking for is best practices for how to organize the
structure of a Python program. I love Python and I just want to be
able to use it well.

Try this:

http://www.refactoring.com/

Not a silver bullet, but a good place to start.

 

[Dan Stromberg]

I use Python a lot, but not well. I usually start by writing a small
script, no classes or modules. Then I add more content to the loops,
and repeat. It's a bit of a trial and error learning phase, making
sure I'm using the third party modules correctly, and so on. I end up
with a working script, but by the end it looks messy, unorganized, and
feels hacked together. I feel like in order to reuse it or expand it
in the future, I need to take what I learned and rewrite it from
scratch.

To some extent, writing code well just comes from practice with
programming, and practice with a language.

Exploratory programming is pretty normal (though some still insist on
having a complete design before starting on coding), but I find that
having lots of automated tests helps make exploratory programming more
practical. You may or may not want to read a bit about agile
programming: http://en.wikipedia.org/wiki/Agile_software_development

A rewrite once in a while is not the end of the world, unless your
management decides it is (then it's just a pain to live without the
rewrite). ^_^ Oh, and if you use modules and classes and even just
functions to limit the impact of one detail on another (each design
decision probably should be wrapped up into its own scope somehow),
you'll find that rewrites of Portions of your code are pretty viable -
without having changes need to cascade through one's codebase.

Just saying "I want this to read clearly, not run marginally faster"
helps, as does using tools like pylint, pychecker, pyflakes and/or
pep8 (pylint probably obviates the pep8 script, but pychecker or
pyflakes likely work well in combination with pep8 - so far I've only
used pylint).

Also, pymetrics is nice for its McCabe Complexity statistic - if the #
gets too high, simplify - this often means subdividing large functions
or methods into a larger number of smaller functions or methods. But
pylint and perhaps others have a warning if your code blocks get too
long - that almost gives the same benefit as McCabe.

You may find that http://rope.sourceforge.net/ helps with your
refactoring, though I have yet to try rope - I just use vim and n.n.n.
I hear that some IDE's support refactoring well - pycharm might be a
good example.

IOW, using some automated tools should give you lots of nearly
immediate feedback and assistance in your goal.

Yes, some people do start with classes at the outset. Just think of a
class as a jack in the box - something with an external view, and a
different, hidden, internal view. When you see a need for something
like a jack in a box in your code (two different views of what's going
on, to limit detail getting scattered more broadly than necessary),
consider using a class or perhaps a generator. And yeah, sometimes
functions are enough - hey, some functional programming languages have
no other means of limiting the impact of details, and there is still
some really good functional code out there.

Finally, look over someone else's code now and then for ideas; that's
a great way to learn. You don't necessarily have to hover over
someone's shoulder to learn from them - fortunately we live in a world
with symbolic language . Make sure the copyright on the code won't
bite you though.

HTH

 

[Terry Reedy]

I use Python a lot, but not well. I usually start by writing a small
script, no classes or modules. Then I add more content to the loops,
and repeat. It's a bit of a trial and error learning phase, making
sure I'm using the third party modules correctly, and so on. I end up
with a working script, but by the end it looks messy, unorganized, and
feels hacked together. I feel like in order to reuse it or expand it
in the future, I need to take what I learned and rewrite it from
scratch.

Not a completely bad idea, except for the 'from scratch' part. Parts of
code that work may just need reorganizing.

The most import thing is automated tests. They should grow with the
code. Tests are like having a safety net.

If I peeked over a Python expert's shoulder while they developed
something new, how would their habits differ? Do they start with
classes from the start?

Depends on whether the particular purpose needs user-defined classes or
is fine with functions using built-in classes.

 

[Steven D'Aprano]

I use Python a lot, but not well. I usually start by writing a small
script, no classes or modules. Then I add more content to the loops, and
repeat. It's a bit of a trial and error learning phase, making sure I'm
using the third party modules correctly, and so on. I end up with a
working script, but by the end it looks messy, unorganized, and feels
hacked together. I feel like in order to reuse it or expand it in the
future, I need to take what I learned and rewrite it from scratch. [...]
Or maybe I'm looking for is best practices for how to organize the
structure of a Python program. I love Python and I just want to be able
to use it well.

I don't think best practice for writing Python is that much different
from best practice for other languages. The main difference is that
Python is multi-paradigm: you can mix procedural, functional and object-
oriented code in the one program.

You should read about bottom-up and top-down programming. You'll probably
end up doing some of both, but mostly top-down.

The most important thing is structured programming and modularization.
The debate over structured programming was won so decisively that people
have forgotten that there was ever an argument to be made for spaghetti
code!

You can learn a lot (and lose a lot of time!) reading the c2.com wiki at
http://c2.com/cgi/wiki. These may be useful:

http://c2.com/cgi/wiki?StructuredProgramming
http://c2.com/cgi/wiki?WhatIsRefactoring


Break your code up into small pieces, whether you use functions or
classes doesn't really matter, although for small scripts functions are
simpler and have less mental overhead. Instead of writing one giant
monolithic block of code that does twenty things, write one function for
each thing, and then one extra main function to call them. This
encourages code reuse, ease of testing, and simplifies maintenance.


I find that I've learned more from "things to avoid" than from "things to
do". Something about reading about disasters makes it really clear why
you shouldn't do it that way

Avoid:

* GOTO, but Python doesn't have that

* Copy-and-paste programming. If you want to do almost the same thing
twice, don't copy the relevant code, paste it, and make a small
modification to it. Write one or more functions that handle the common
code, and call the function.

* Functions that do unrelated things. Functions should do one thing,
although that "thing" can get quite complicated.

* Don't sweep errors under the rug. Don't catch exceptions unless you can
do something about them. "Just ignore it, I'm sure it's not important" is
rarely appropriate.


See also:
http://c2.com/cgi/fullSearch?search=CategoryDevelopmentAntiPattern

 

[Roy Smith]

I use Python a lot, but not well. I usually start by writing a small
script, no classes or modules.

One anti-pattern that I see in my own code is starting out thinking,
"this is just a little script, I doesn't need any real structure". That
almost always turns out to be wrong, but by the time I start to realize
I'm writing spaghetti code, it's so temping to say, "I don't have the
time to refactor this now, I'll just hack on it a bit more". Which, of
course, makes it even harder to unravel later.

The first step is to break up a monolithic script into a few functions.
I encourage myself to do that from the git-go by keeping a template
around:

#!/usr/bin/env python

def main():
pass

if __name__ == '__main__':
main()

and I use that whenever I start a new script. That at least gets me off
on the right foot.

The next step is to turn my collection of functions (with the inevitable
collection of global variables that lets them communicate) into a class
with methods and instance variables. I can't tell you how many times
I've started out saying, "this isn't going to be complicated enough to
justify making it a class". Inevitably, I'm wrong.

Finally, the next layer of stupid mistake I often make is to start out
saying, "This isn't going to be complicated enough to justify writing
unit tests". Inevitably, I'm wrong about that too.

So far, none of the above is at all specific to Python, It's equally
true in any language.

Now, for some Python-specific advice; you can write Fortran in any
language. What that means is it's one thing to translate some existing
script into Python and make it work, but it's another to actually take
advantage of some of Python's biggest strengths. Learn to be
comfortable with list comprehensions, generator expressions, and
iterators in general. Learn about Python's advanced data structures
such as sets, defaultdicts, and named tuples.

 

[Steven D'Aprano]

Steel-cage death match. FIGHT!

To the death? No, to the pain!

http://en.wikiquote.org/wiki/The_Princess_Bride

 

[flebber]

To the death? No, to the pain!

http://en.wikiquote.org/wiki/The_Princess_Bride

I really liked Abstraction Chapter 6 & 7 In Magnus Lie Hetlands book
novice to professional. It really show the how to "think it out" which
seems to be what your after. The first sub heading in Chapter 6 is
"Laziness is a virtue" can't beat that.

 

[Chris Rebert]

I really liked Abstraction Chapter 6 & 7 In Magnus Lie Hetlands book
novice to professional. It really show the how to "think it out" which
seems to be what your after. The first sub heading in Chapter 6 is
"Laziness is a virtue" can't beat that.

Related:
http://c2.com/cgi/wiki?LazinessImpatienceHubris

Cheers,
Chris

 

[Jean-Michel Pichavant]

I use Python a lot, but not well. I usually start by writing a small
script, no classes or modules. Then I add more content to the loops,
and repeat. It's a bit of a trial and error learning phase, making
sure I'm using the third party modules correctly, and so on. I end up
with a working script, but by the end it looks messy, unorganized, and
feels hacked together. I feel like in order to reuse it or expand it
in the future, I need to take what I learned and rewrite it from
scratch.

If I peeked over a Python expert's shoulder while they developed
something new, how would their habits differ? Do they start with
classes from the start?

I guess I'm looking for something similar to "Large Scale C++ Software
Design" for Python. Or even just a walkthrough of someone competent
writing something from scratch. I'm not necessarily looking for a
finished product that is well written. I'm more interested in, "I have
an idea for a script/program, and here is how I get from point A to
point B."

Or maybe I'm looking for is best practices for how to organize the
structure of a Python program. I love Python and I just want to be
able to use it well.

I guess this is happening to anyone using python for their everyday
scripting. Writing tools to make life easier.
You start writing some quick and dirty code just to do the trick ASAP.
Then after adding some features your realize it became a little more
than a script but don't want to share it because the code is just a
freaking mess that no one but you can understand.

Here are the things I do to avoid getting to that point:
1 - accept the fact that I will lose time trying to write it correcly
from the begining
2 - the script has to be "importable" and usable by a 3rd python program.
3 - regulary test my code with the python shell importing the script
4 - use a linter (pylint) with all warnings activated.
5 - document the public class/function. This "force" me to expose the
minimum interface, because like everyone, I dislike writing doc
6 - always include a argument parser.
7 - always use the logging module.
8 - if the project starts to become medium, write unitary tests.

And maybe the most important one, I always start from the upper level
interface down to primitives (top-down programming ?). This helps me
making suff really easy to use, with a clean interface. Sometimes the
implementation can become tricky that way, I guess it's a matter of
preference. In pratice, that just means that I write the method call
before writing the method definition.

JM

 

[Jorgen Grahn]

I use Python a lot, but not well. I usually start by writing a small
script, no classes or modules. Then I add more content to the loops,
and repeat. It's a bit of a trial and error learning phase, making
sure I'm using the third party modules correctly, and so on. I end up
with a working script, but by the end it looks messy, unorganized, and
feels hacked together. I feel like in order to reuse it or expand it
in the future, I need to take what I learned and rewrite it from
scratch.

If I peeked over a Python expert's shoulder while they developed
something new, how would their habits differ? Do they start with
classes from the start?

I guess I'm looking for something similar to "Large Scale C++ Software
Design" for Python. Or even just a walkthrough of someone competent
writing something from scratch. I'm not necessarily looking for a
finished product that is well written. I'm more interested in, "I have
an idea for a script/program, and here is how I get from point A to
point B."

Or maybe I'm looking for is best practices for how to organize the
structure of a Python program. I love Python and I just want to be
able to use it well.

Good questions -- and you got some really good answers already!

What I always do when starting a program is:

- Split it into a 'if __name__ == "__main__":' which does the
command-line parsing, usage message and so on; and a function
which contains the logic, i.e. works like the program would have
if the OS had fed it its arguments as Python types

- Document functions and classes.

- Avoid having functions use 'print' and 'sys.std*', in case I need to
use them with other files. I pass file-like objects as arguments
instead.

- Write user documentation and build/installation scripts. Since I'm
on Unix, that means man pages and a Makefile.

And that's all in the normal case. No need to do anything more fancy
if it turns out I'll never have to touch that program again.

I use classes when I see a use for them. The "see" part comes from
quite a few years' worth of experience with object-oriented design in
Python and C++ ... not sure how to learn that without getting lost in
Design with a capital 'D' for a few years ...

Anyway, I don't feel bad if I don't find any classes at first.

/Jorgen

 

[Roy Smith]

- Write user documentation and build/installation scripts. Since I'm
on Unix, that means man pages and a Makefile.

Wow, I haven't built a man page in eons. These days, user documentation
for me means good help text for argparse to use. If I need something
more than that, I'll write it up in our wiki.

Anyway, I don't feel bad if I don't find any classes at first.

Same here. I don't usually find a reason to refactor things into
classes until I've written the second or third line of code

Somewhat more seriously, the big clue for me that I've got a class
hiding in there is when I start having all sorts of globals. That's
usually a sign you've done something wrong.

 

[Michael Torrie]

You should read about bottom-up and top-down programming. You'll probably
end up doing some of both, but mostly top-down.

Most of my development is done by designing top-down and then coding
bottom-up. Coding top down is fine, but I'd expect to refactor the code
frequently as I try to spin code off into standalone modules.

 

[Jorgen Grahn]

Wow, I haven't built a man page in eons. These days, user documentation
for me means good help text for argparse to use.

Perhaps I'm old-fashioned, but all other software I use (on Unix) has
man pages. I /expect/ there to be one. (It's not hard to write a man
page either, if you have a decent one as a template.)

Help texts are better than nothing though (and unlike man pages they
work on Windows too).

If I need something
more than that, I'll write it up in our wiki.

I guess you're working within an organization? Local rules override
personal preferences -- if everyone else is using the wiki, I guess
you must do too.

I have to say though that *not* handling the documentation together
with the source code is harmful. If source code and documentation
aren't in version control together, they *will* go out of sync.

Same here. I don't usually find a reason to refactor things into
classes until I've written the second or third line of code

Somewhat more seriously, the big clue for me that I've got a class
hiding in there is when I start having all sorts of globals. That's
usually a sign you've done something wrong.

Or a whole bunch of related arguments to a function, and/or the same
arguments being passed to many functions.

/Jorgen

 

[Roy Smith]

Perhaps I'm old-fashioned, but all other software I use (on Unix) has
man pages. I /expect/ there to be one. (It's not hard to write a man
page either, if you have a decent one as a template.)

The nice thing about help text is that it keeps the documentation and
the code in one place, which makes it a little more likely people will
actually update the docs as they update the code.

I guess you're working within an organization?

FSVO "organization", but yes.

Local rules override personal preferences -- if everyone else is
using the wiki, I guess you must do too.

I've become very enamored of wikis because of the lost activation energy
barrier and instant feedback. To update a man page (info node, etc),
you need to find the source document, perhaps check it out, edit it,
submit it back to version control, install the new version in /usr/man,
and so on. People tend not to bother. Wikis are so much more
lightweight, they're that much more likely to get kept current.

I have to say though that *not* handling the documentation together
with the source code is harmful. If source code and documentation
aren't in version control together, they *will* go out of sync.

That is a valid argument against wikis.

 

[Westley Martínez]

Yes, that's nice for the programmer. But isn't the point of the man page
to be nice for the users? The man pages document many more things than
help text output from the program.

--
\ “Very few things happen at the right time, and the rest do not |
`\ happen at all. The conscientious historian will correct these |
_o__) defects.†—Mark Twain, _A Horse's Tale_ |
Ben Finney

From what I've seen, the man pages are supposed to be in depth

information that covers every nook and cranny of every option while the
--help option is supposed to simply print a summary in case one forgets
the syntax, but nowadays they've kind of been blended together.

 

[Roy Smith]

This collection of a great deal of documentation for the operating
system into a single “manual†is one reason why users like man pages so
much: we want to find anything installed on the system documented in
that one place.

What made man pages such a great technology back in the 70's was exactly
what Ben is saying. Everything was on-line and instantly available for
quick reference. Not to mention that you could use man as just another
cog in the unix toolset and do things like grep all of /usr/man for a
term (or an error message which appeared and you didn't know what had
produced it). These were astonishing advances in usability vs. having
printed manuals (which may or may not have been available to you).

But, today we have such better tools available. HTML, for example.
Whether it's a wiki or the generated output of sphinx/doxygen/etc, HTML
provides for a much richer presentation. Which is more convenient:
having the signal(3) man page reference "sigaction(2)" textually, or
having it be a clickable link that can take me right there? HTML also
gives you much greater formatting flexibility than what's still
basically 35-year old nroff.

If, for whatever reason, you're still wed to plain text, even info gives
you much better capabilities than man. At least you get basic stuff
like menus, document hierarchy, cross-linking, and browsing history.

I'm not saying that help text is the be-all and end-all for
documentation. I'm just saying that if you're going to do more than
help text, it's hard to imagine putting any effort into producing man
pages. Except possibly as the automated output of some multi-target
documentation system which produces them as a by-product of producing
other, richer, formats.

 

[Jorgen Grahn]

Full agreement there.

Some disagreement here. There are typographical features in
nroff/troff today which you don't get in web browsers: ligatures and
hyphenation for example.

Then of course there's the argument that "formatting flexibility"
isn't a good thing for reference manuals -- you want them to look
similar no matter who wrote them. (Not that all man pages look similar
in reality, but there are some pretty decent conventions which most
follow).

/Jorgen

 

[Roy Smith]

Some disagreement here. There are typographical features in
nroff/troff today which you don't get in web browsers: ligatures and
hyphenation for example.

Saying that HTML doesn't have ligatures and hyphenation is kind of like
saying Python is a bad programming language because it doesn't come in
purple.

Yes, n/troff does ligatures and hyphenation. Are such things really
essential for an on-line reference manual? The ligatures, clearly not,
since what most of us are talking about here are the plain-text
renditions you get with the on-line "man" output.

Hyphenation? I suppose it has some value for this kind of stuff, but it
can also be a pain. What happens when you're grepping the man page for
"egregious" and can't find it because it got hyphenated?

No, those are things you want for typesetting documents, not for
browsing on-line reference material. And I can't imagine anybody using
troff for typesetting today. I'm sure there are a few people who will
pop out of the woodwork insisting they do, but when it comes to
typesetting, "I want a tool, not a hobby".

 

[Westley Martínez]

Saying that HTML doesn't have ligatures and hyphenation is kind of like
saying Python is a bad programming language because it doesn't come in
purple.

Yes, n/troff does ligatures and hyphenation. Are such things really
essential for an on-line reference manual? The ligatures, clearly not,
since what most of us are talking about here are the plain-text
renditions you get with the on-line "man" output.

Hyphenation? I suppose it has some value for this kind of stuff, but it
can also be a pain. What happens when you're grepping the man page for
"egregious" and can't find it because it got hyphenated?

No, those are things you want for typesetting documents, not for
browsing on-line reference material. And I can't imagine anybody using
troff for typesetting today. I'm sure there are a few people who will
pop out of the woodwork insisting they do, but when it comes to
typesetting, "I want a tool, not a hobby".

But you can't seriously say that authoring HTML is effective. Sure,
outputting HTML is fine, but as for writing the source, troff, docbook,
sphinx, even TeX, etc, is superior to HTML simply because HTML was
designed for web pages and those others were designed specifically for
documentation (not TeX, but that's another story). I hate writing HTML,
it's a pain in the neck.

But anyways, I find it easier to simply type "man sigaction" than to
search around on google all day for a handful of facts and a truckload
of opinions, as do I find it easier to type "help(os.path)" than have to
open my browser, click on the python doc bookmark, search for os.path,
wait for the search results, and click on it. This is just what I find
easier though, and I think using tools that output HTML, groff, LaTeX,
etc should continued to be used.