Use the Source Luke

[Raymond Hettinger]

I hoping a new trend will start with dev's putting direct
source code links in their documentation:

http://rhettinger.wordpress.com/2011/01/28/open-your-source-more/

I'm looking for more examples of projects that routinely
link their docs back into relavant sections of code.
Have any of you all seen other examples besides
the Go language docs and the Python docs?


Raymond

 

[Paul Rubin]

http://rhettinger.wordpress.com/2011/01/28/open-your-source-more/

I'm looking for more examples of projects that routinely
link their docs back into relavant sections of code.
Have any of you all seen other examples besides
the Go language docs and the Python docs?

That is a very good post, and just about 2 days ago I happened to be
looking at the source of heapq for something I was doing, and I think I
got to it through the doc link that you added. So the link has already
been useful.

Haddock (Haskell's equivalent to Pydoc or Javadoc) can automatically
generate source links in Haskell documentation. For example, here's the
docs (including source links) for Haskell's standard library for dealing
with lists:

http://www.haskell.org/ghc/docs/7.0-latest/html/libraries/base-4.3.0.0/Data-List.html

I've wanted for a long time for developer-oriented Linux distributions
to include full source code of everything as an integral part of the
distro rather than as a separate distribution. For example, you could
examine any application and instantly see its source. All programs
would be compiled with debugging enabled and a way to attach a debugger
to the running process, so you could at any time interrupt the program
and use gdb to see what it was doing, single step it, etc.

 

[Giampaolo Rodolà]

I hoping a new trend will start with dev's putting direct
source code links in their documentation:

 http://rhettinger.wordpress.com/2011/01/28/open-your-source-more/

I'm looking for more examples of projects that routinely
link their docs back into relavant sections of code.
Have any of you all seen other examples besides
the Go language docs and the Python docs?


Raymond

Thanks, I think this is a great idea.
I think this definitively should be done for 2.7 documentation as well.

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/

 

[Rui Maciel]

Have any of you all seen other examples besides
the Go language docs and the Python docs?

Wasn't doxygen developed with that in mind?


Rui Maciel

 

[Jack Diederich]

I hoping a new trend will start with dev's putting direct
source code links in their documentation:

 http://rhettinger.wordpress.com/2011/01/28/open-your-source-more/

I'm looking for more examples of projects that routinely
link their docs back into relavant sections of code.
Have any of you all seen other examples besides
the Go language docs and the Python docs?

I think you overestimate how common it used to be to carry around the
sourcecode for the software you use compared to now; In the past it
wasn't even always possible - if the Sun cc compiler core dumps you
have no recourse to code. Promoting the idea of doing it is good
because it /is/ a novel idea to many people. Promoting the idea of
making it extremely easy via documentation links is good because it is
new as well.

Modern tools are making this easier than it used to be so your call
for making it easier still is well timed. Github/bitbucket/launchpad
have combined the source with documentation; github especially because
the README on github is the canonical documentation and the source is
only one mouse click away. ack-grep has changed my life. Sure, I
could always do the same thing in the past with find+grep but ack-grep
makes it so easy (switches for language file types!) that I use it
much more; I have "ag" aliased to "ack-grep --python" and I use it
all the f'ing time because it costs me near zero to do so. Likewise I
have an alias "cdp" that "cd"s me into the directory where any given
python module lives. "cdp collections" puts me straight into
"/usr/local/lib/python2.6" - again, it makes it so easy to look at
sourcecode that I do it all the time. It is usually quicker to do
cdp/python/import module_name/help(module_name) than to look up the
docs. Worst case the docstrings suck and I just read the code.

* Anecdote. I was in a room with Tim Peters and has some questions
about the interface to code he wrote so I thought "Hey, I'll just ask
Tim!" I asked him and he replied "I'm not sure what you're asking -
do you want me to read the code aloud to you?" So I just went and
read it.

-Jack

 

[Raymond Hettinger]

[Jack Diedrich]

I think you overestimate how common it used to be to carry around the
sourcecode for the software you use compared to now;  In the past it
wasn't even always possible - if the Sun cc compiler core dumps you
have no recourse to code.

You're right of course. For the Python world though, there does seem
to have been a change. A decade ago in this newsgroup, there were
frequent references to standard library source. I don't see that
much anymore.

 Promoting the idea of doing it is good
because it /is/ a novel idea to many people.  Promoting the idea of
making it extremely easy via documentation links is good because it is
new as well.

Judging from the comments so far, it looks like everyone here agrees.

Modern tools are making this easier than it used to be so your call
for making it easier still is well timed.  Github/bitbucket/launchpad
have combined the source with documentation;

Do they typically feature multiple links from documentation specifics
to corresponding code specifics?

Part of my thesis is that it is not enough to make docs and source
available, they need to be linked in a way that helps people
answer specific problems without having to invest a week in learning
the gestalt of a foreign code base.

 Worst case the docstrings suck and I just read the code.

That's a good habit to have. I find that my willingness to do it
varies across projects -- I'm happy to look at the Mercurial source
but could never being myself to look at the innards of Git or CouchDB.

* Anecdote.  I was in a room with Tim Peters and has some questions
about the interface to code he wrote so I thought "Hey, I'll just ask
Tim!"  I asked him and he replied "I'm not sure what you're asking -
do you want me to read the code aloud to you?"  So I just went and
read it.

Thanks for the anecdote. I love that story
Uncle Timmy's style is both clever and pointed.


Raymond

 

[rusi]

Note that Raymond is speaking specifically in the context of free
software, where the license is by definition permitting free
redistribution of the source code.

It is an obvious necessary condition that for code to be opened it
should be open (source).
However the necessary condition is not sufficient.

I have a quibble with the framing:

Consider this example:
The emacs source if compiled from source will give you help for each
lisp or even builtin (C) function out of the box from inside emacs.
However if you get the emacs package from debian/ubuntu you will get
neither unless you install el files -- which is fine -- just install
the el package. About the C sources? I dont believe that one can get
that linkage from within apt; one just hast to compile oneself. (I
would be happy to be surprised on this).

Isn't this an instance of the problem that Raymond is talking of?

[Personal note: Ive been a python user and teacher for nearly 10
years and would eagerly welcome more easy code-open-ness]

 

[TP]

I hoping a new trend will start with dev's putting direct
source code links in their documentation:

 http://rhettinger.wordpress.com/2011/01/28/open-your-source-more/

I'm looking for more examples of projects that routinely
link their docs back into relavant sections of code.
Have any of you all seen other examples besides
the Go language docs and the Python docs?


Raymond

The Sphinx Python Documentation Generator
(http://sphinx.pocoo.org/index.html), used for documenting lots of
things other than Python, has an extension called "sphinx.ext.viewcode
– Add links to highlighted source code"
(http://sphinx.pocoo.org/ext/viewcode.html).

You can see my use of it here on a small PyQt project:
http://tpgit.github.com/MDIImageViewer/imageviewer.html. You can even
view the Sphinx documentation "code" by clicking the "Show Source"
link on the left.

-- TP

 

[Raymond Hettinger]

The Sphinx Python Documentation Generator
(http://sphinx.pocoo.org/index.html), used for documenting lots of
things other than Python, has an extension called "sphinx.ext.viewcode
– Add links to highlighted source code"
(http://sphinx.pocoo.org/ext/viewcode.html).

Thanks, I didn't know about that extension.

To support my effort to add source links to the Python docs,
Georg Brandl added a new directive :source: so that we could
write :source:`Lib/heapq.py` and the set the prefix somewhere else
(i.e. pointing at the py3k branch on subversion or on mercurial).


Raymond

 

[rusi]

The “problem”, which I don't consider to be a problem per se, is one of
OS-wide policy, not “installers”. The policy is a matter of tradeoffs
across the system, and isn't “tucking the code away in a dark corner”..

Earlier mail:

If you want to blame anything for this (though I think it’s inaccurate
to frame it as a problem), the correct target of your accusation is the
fact that a filesystem path is the identifier for these modules that
will be used by programs to find them.

I think this is a fairly accurate description of (one aspect of) the
problem.
If you dont see it as a problem how do you explain that google can
search the World Wide Web better than we can search our individual
hard disks?

 

[Steven D'Aprano]

Earlier mail:


I think this is a fairly accurate description of (one aspect of) the
problem.
If you dont see it as a problem how do you explain that google can
search the World Wide Web better than we can search our individual hard
disks?

I fail to see any connection between the location that operating systems
store files, and the ability of Google to index publicly available
websites. It sounds to me like the equivalent of "If you don't think
Python programmers are a problem, how do you explain that it takes me 45
minutes to drive to work in the morning but nearly an hour to drive home
in the evening?"

Google has approximately one million servers indexing the web, and
they're willing to use hundreds of terabytes of disk space to store the
indexes. My desktop is *one* PC with little free space, and I'd rather
trade off time for storage, so I don't keep any indexes of file content
on my system. If I *wanted* to index my files, I could do so, although in
fairness I'm not aware of any Linux tools which do this -- I know of
`locate`, which indexes file *names* but not content, and `grep`, which
searches file content but doesn't index what it finds.

On Windows and Mac, though, I believe there are standard utilities which
will index file content if you allow them.

So... Google can search the web better than we can search our local hard
drives because Google has invested tens or hundreds of millions into
building a world-class search engine, and we haven't.

 

[rusi]

I fail to see any connection between the location that operating systems
store files, and the ability of Google to index publicly available
websites.

http://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._Location-addressed

 

[Raymond Hettinger]

That’s hardly a “blame” of installers. The modules are placed in such
locations because they need to be accessible in a hierarchy at a
location that is known to not conflict with anything else, and be
predictable for the Python interpreter on the system.

Sure. Installers are just installing where they're supposed to.
And good people have given a lot of thought to the preferred
target directories. I'm just observing that the source files
are often ending-up out-of-sight and out-of-mind so that fewer
users ever see the source.

It's not deep a problem -- it would only take a symlink to
provide quick access.

My thesis is that we can do even better than that by adding
direct links from the docs to the relevant code with nice
syntax highlighting.


Raymond


P.S. Making it easy to get to relevant source is only half of
the solution. The remaining problem is cultural. Perhaps every
project should have a recommended reading list.

As a start, I think the following are instructive and make for a good
read:

http://svn.python.org/view/python/branches/py3k/Lib/ftplib.py?view=markup
http://svn.python.org/view/python/branches/py3k/Lib/heapq.py?view=markup
http://svn.python.org/view/python/branches/py3k/Lib/collections.py?view=markup
http://svn.python.org/view/python/branches/py3k/Lib/queue.py?view=markup
http://svn.python.org/view/python/branches/py3k/Lib/functools.py?view=markup

 

[Steven D'Aprano]

http://en.wikipedia.org/wiki/Content-addressable_storage#Content-

addressed_vs._Location-addressed


Nope, sorry, doesn't help. Both local files on your hard drive, and most
remote websites on the Internet, are location addressed. Google indexes
the content, but they don't provide content-addresses. Indeed, they
*can't* do so (except, possibly, for content they control such as Google
Books), since they can't prevent content owners from modifying either the
location address or the content. And as I've mentioned, there are desktop
utilities that index content for Windows and Macintosh. In fact, Google
themselves offer a desktop app that does just that:

http://desktop.google.com/features.html

 

[David Boddie]

If I *wanted* to index my files, I could do so, although in
fairness I'm not aware of any Linux tools which do this -- I know of
`locate`, which indexes file *names* but not content, and `grep`, which
searches file content but doesn't index what it finds.

You might find this page useful:

http://www.wikinfo.org/index.php/Comparison_of_desktop_search_software

David

 

[Tim Wintle]

My thesis is that we can do even better than that by adding
direct links from the docs to the relevant code with nice
syntax highlighting.

+1 - I think the source links are very useful (and thanks for pushing
them).


However I think the biggest changes that have probably happened with
python itself are:

(1) More users for whom this is their first language.
(2) CS courses / training not teaching C (or pointer-based languages).

(2) is especially important IMO - under half of the python developers I
have regularly worked with would feel comfortable reading C - so for the
other half reading C source code probably isn't going to help them
understand exactly what's going on (although in the long run it might
help them a lot)


Tim Wintle

 

[rantingrick]

In fact, Google
themselves offer a desktop app that does just that:

http://desktop.google.com/features.html

Yes, but at the expense of your privacy! How much private information
is being sent back to Google plex an used to flash more click ads at
you? Well i guess we could say, google does spam us, but at least we
are more likely to be *slightly* interested in the spam.

 

[rusi]

You might find this page useful:

http://www.wikinfo.org/index.php/Comparison_of_desktop_search_software

David

Thanks for that link David

I note particularly the disclaimer that it was removed from wikipedia
[Like when <your-unfavorite-TV-channel> censors stuff you know it
deserves a second look ;-) ]

 

[Raymond Hettinger]

+1 - I think the source links are very useful (and thanks for pushing
them).

Happy to do it.

However I think the biggest changes that have probably happened with
python itself are:

 (1) More users for whom this is their first language.
 (2) CS courses / training not teaching C (or pointer-based languages).

(2) is especially important IMO - under half of the python developers I
have regularly worked with would feel comfortable reading C - so for the
other half reading C source code probably isn't going to help them
understand exactly what's going on (although in the long run it might
help them a lot)

That would explain why fewer people look at the C source code.

However, even the parts of the standard library written in pure Python
don't seem to be getting read anymore, so I'm still inclined to
attribute the issue to 1) inconvenient placement of source code,
2) a largish code base, and 3) possibly a cultural shift.

I'm thinking that all of those can be addressed by efforts
to lower to intellectual investment required to find the
relevant source code.


Raymond

 

[rantingrick]

I note particularly the disclaimer that it was removed from wikipedia
[Like when <your-unfavorite-TV-channel> censors stuff you know it
deserves a second look ;-) ]

Oh you mean that channel that *claims* to provide a specific type of
"programming" however they really provide *anything* but that specific
type of programming! Yes, thank god for you tube.