Michael Hudson said:
it annoys me no end if I need to start an expensive computation from
scratch because some trivial and easily fixable problem occured towards the
end of the computation (sometimes it is possible to salvage stuff by hand
by pickling things from the appropriate post-mortem frame, but I'd *much*
prefer being able to say: foo=some_value; resume).
I'd like this too. It might be quite hard to implement
non-disruptively but I haven't thought about it too hard. Would make
an excellent project for a master's thesis, IMHO.
Footnotes:
[1] Number 2 would be the stupid try: finally: idiom which also seems to
screw up tracebacks
?
I verified that this doesn't happen with plain python -- ipython's traceback
pretty printing code however doesn't display the right line for frames where
the exception occured within try: finally:. I've now tracked it down to the
use of inspect.getinnerframes, the line numbers here are subtly different from
what traceback.extract_tb returns (last executed expression in frame vs. where
the error occured). Since I couldn't find something readymade, I'll submit
some patch to ipython that merges the functionality of the two functions.
What do you mean here, specifically?
In no particular order:
- Interactively redefining modules or classes in a way that propogates
to other modules/preexisting instances is not exactly fun. One can often
get by by judicious use of reload, mutating classes [1] (got to be careful
to to avoid pickle failing with something along the lines of
"myModule.myClass is not of type myModule.myClass")
- no images, i.e. you can't freeze and dump the state of the whole system
- it's not particularly easy or convenient to get an useful overview of the
variables in your "workspace", e.g. memory consumption, narrowing down to
"interesting" categories etc (I know that there is of course no general way
of doing that, but compare e.g. for matlab)
- pdb is uhm, well... suboptimal (also crashes on me from time to time, not
sure why)
- global variables in python are a painful in a number of ways that does
affect interactive development
- there is not much of a culture to make modules work properly for interactive
users -- things are often not reload-safe, export all sorts of crap, not
just their interface (so that 'from foo import *' likely will hose things up
-- this is slightly aggravated 'helpful' naming conventions such as
datetime.datetime or StringIO.StringIO). Finally I think some modules,
notably GUI toolkits won't work at all.
- the available IDEs I know of are clearly far from ideal. I'd venture the
uneducated guess that ipython+emacs is amongst the best offerings for
interactive development with python and it's really not that great -- e.g.
if you use the py-execute commands source level debugging no longer will
work since the temp file created by python-mode for execution will be gone
(you can of course hang on to it, which is actually what I wound up doing
but that's very risky -- you end up inadvertenly fixing temp-files rather
than the real thing. I guess it might help if there were some easy way to
exec(file) a string, but lie about were it came from (i.e. ``exec foo,
filename="bla.py", startline=10)). In case anyone wonders that given my
misgivings about pdb I'm bothered about this -- well, one can easily set up
emacs/ipython to jump to right file and line when an error in your
interactive session occurs (and than walk up and down the traceback). This
is **very** useful, I have it activated pretty much all the time.
I find I can do interactive development in Python most of the time (I
do wish it was more possible with PyObjC, though).
I'm not saying it's impossible (I *always* run ipython in emacs, never python
in the commandline on a file, with the sole exception of regression test) but
kludging a workable interactive enviroment together needs, I think, a fair
amount of expertise and additional software (something like ipython+emacs) --
so I'd guess that most people primarily treat python as a really fast C
compiler (but I might be wrong), which is a pitty.
Also, whlilst python interactive offerings might be great when compared to
Java and C++( thankfully I haven't tried), it clearly falls far short of what
is in principle achievable and indeed has been often been achieved many, many
years ago (squeak, cmucl/sbcl+slime, matlab, j and plt scheme, to name just a
few all have things to offer for interactive work that a python user can only
dream of).
'as
Footnotes:
[1] Here are a few of the hacks I'm using, in case anyone might find them
useful -- or even better tell me about better alternatives (If someone has
cooked something reasoable for reloading modules, I'd love to hear about
it).
# to update all existing class instances
def updateClass(oldClass, newClass):
"""Destrucitively modify the ``__dict__`` and ``__bases__`` contents of
`oldClass` to be the same as of `newClass`. This will have the effect that
`oldClass` will exhibit the same behavior as `newClass`.
Won't work for classes with ``__slots__`` (which are an abomination
anyway).
"""
assert type(oldClass) is type(newClass) is type #FIXME
#FIXME redefinition of magic methods
for name in dir(oldClass):
if not name.startswith('__') or not name.endswith('__'):
delattr(oldClass, name)
for name in dir(newClass):
if not name.startswith('__') or not name.endswith('__'):
setattr(oldClass, name, newClass.__dict__[name])
# XXX should check that this is absolutely correct
oldClass.__bases__ = newClass.__bases__
## easy pickling and unpickling for interactive use
def magicGlobals(level=1):
r"""Return the globals of the *caller*'s caller (default), or `level`
callers up."""
return inspect.getouterframes(inspect.currentframe())[1+level][0].f_globals
def __saveVarsHelper(filename, varNamesStr, outOf,extension='.bpickle',**opts):
filename = os.path.expanduser(filename)
if outOf is None: outOf = magicGlobals(2)
if not varNamesStr or not isString(varNamesStr):
raise ValueError, "varNamesStr must be a string!"
varnames = varNamesStr.split()
if not splitext(filename)[1]: filename += extension
if opts.get("overwrite") == 0 and os.path.exists(filename):
raise RuntimeError("File already exists")
return filename, varnames, outOf
def saveVars(filename, varNamesStr, outOf=None, **opts):
r"""Pickle name and value of all those variables in `outOf` (default: all
global variables (as seen from the caller)) that are named in
`varNamesStr` into a file called `filename` (if no extension is given,
'.bpickle' is appended). Overwrites file without asking, unless you
specify `overwrite=0`. Load again with `loadVars`.
Thus, to save the global variables ``bar``, ``foo`` and ``baz`` in the
file 'savedVars' do::
saveVars('savedVars', 'bar foo baz')
"""
filename, varnames, outOf = __saveVarsHelper(
filename, varNamesStr, outOf, **opts)
print "pickling:\n", "\n".join(isort(varnames))
try:
f = None
f = open(filename, "wb")
cPickle.dump(dict(zip(varnames, [outOf, varnames])),
f, 1) # UGH: cPickle, unlike pickle doesn't accept bin=1
finally:
if f: f.close()
def loadVars(filename, ask=True, into=None, only=None):
r"""Load variables pickled with `saveVars`.
Parameters:
- `ask`: If `True` then don't overwrite existing variables without
asking.
- `only`: A list to limit the variables to or `None`.
- `into`: The dictionary the variables should be loaded into (defaults
to global dictionary).
"""
filename = os.path.expanduser(filename)
if into is None: into = magicGlobals()
varH = loadDict(filename)
toUnpickle = only or varH.keys()
alreadyDefined = filter(into.has_key, toUnpickle)
if alreadyDefined and ask:
print "The following vars already exist; overwrite (yes/NO)?\n",\
"\n".join(alreadyDefined)
if raw_input() != "yes":
toUnpickle = without(toUnpickle, alreadyDefined)
if not toUnpickle:
print "nothing to unpickle"
return None
print "unpickling:\n",\
"\n".join(isort(list(toUnpickle)))
for k in varH.keys():
if k not in toUnpickle:
del varH[k]
into.update(varH)