Online Modification of Python Code

[malv]

I am involved in a major scientific algorithm design problem in which
simulation of the underlying physics and visualization play an
important role. Algorithm adaptation from run to run often involves few
lines of code. The setup of the system's state requires quite a bit of
processing time and rapidly becomes a real drag on the experimentation
sessions.

It would be very nice if Python would allow 'online' code modification
from the IDE, roughly analogous to VisualBasic (this is the only thing
I like in VB). Some reply that such a feature is unnescessary because
of unittest. This does not hold at all for a project like this wherby
the problem is not really to debug and veryfy program code.

Can anybody recommend a practical ad hoc solution to such problem?
Thanks all.

 

[Dennis Lee Bieber]

I am involved in a major scientific algorithm design problem in which
simulation of the underlying physics and visualization play an
important role. Algorithm adaptation from run to run often involves few
lines of code. The setup of the system's state requires quite a bit of
processing time and rapidly becomes a real drag on the experimentation
sessions.

It would be very nice if Python would allow 'online' code modification
from the IDE, roughly analogous to VisualBasic (this is the only thing
I like in VB). Some reply that such a feature is unnescessary because
of unittest. This does not hold at all for a project like this wherby
the problem is not really to debug and veryfy program code.

Can anybody recommend a practical ad hoc solution to such problem?
Thanks all.

Presuming most of the set-up code is fixed, and it is just some
function that is being changed...

Stuff the function itself into a separate file, and use reload() on
the imported file after changing it...

Might need to code a bit of a driver for this -- run, evaluate, ask
for re-run (edit first, respond yes), reload changed function, repeat ad
nauseum


This is, of course, just an off-the-wall idea before I drop off to
bed...

--

 

[Robert Kern]

I am involved in a major scientific algorithm design problem in which
simulation of the underlying physics and visualization play an
important role. Algorithm adaptation from run to run often involves few
lines of code. The setup of the system's state requires quite a bit of
processing time and rapidly becomes a real drag on the experimentation
sessions.

If the only part that's changing is after the setup, then why don't you
save the results of the setup to a file and read it back in for each
experiment?

Do the experiments modify the objects which are created by the setup? If
not, then you can just run the different experiments over and over
without problem. If they do modify those objects, then you might be able
to copy the objects and run an experiment on the copies, then make fresh
copies for each later experiment.

It would be very nice if Python would allow 'online' code modification
from the IDE, roughly analogous to VisualBasic (this is the only thing
I like in VB).

I'm not familiar with this feature. Could you please describe it more fully?

If I'm following you correctly, it seems that you might be satisfied
with an interactive session. I use ipython as my shell, and it makes
these kind of tasks very convenient.

http://ipython.scipy.org

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[malv]

I wrote a module testalgo.py with some functions to be modified.
After modification I typed to the interactive shell:.... and things seem to workout as you suggested. At least I modified
some quickie print statements to check. I use eric3 and nothing in this
IDE seems to interfere with the process.
I hadn't thought about doing it this way.
(I have in fact an extensive driver/command infrastructure in place)

I seem to recall a post by Diez Roggish that reload() doesn't always
work as it should. Any news on this? At least im my preliminary test it
works.
Thank you Dennis!

 

[malv]

Hi Robert:
(1) Things are not that simple. In fact in setting up a run, extensive
database backup is already involved. Subsequently, data structures are
built at runtime of typically 0.5Gbytes. Although saving this should
work, it would require quite some debug/verification to check out the
validity. Many data structures intervene. Dennis' suggestion worked
fine so I'm happy to be able to leave everything else 'untouched'.

(2) As to your VB question. I have been 'forced' to do a lot of VB
related to M$ Access db work. VB is 100% integrated in M$'s Visual IDE
and allows you to change 'on the fly' almost anything in the code, even
in the midst of a debug breakpoint stop. This is very, very handy. It
would be a very valuable addition to Python, though it's not very clear
to me how this could be implemented without a specific IDE. (Don't
switch to VB for this though. Python is much cooler).

 

[Diez B. Roggisch]

I seem to recall a post by Diez Roggish that reload() doesn't always
work as it should. Any news on this? At least im my preliminary test it
works.

Read the docs on reload:

http://www.python.org/doc/current/lib/built-in-funcs.html

"""

If a module is syntactically correct but its initialization fails, the
first import statement for it does not bind its name locally, but does
store a (partially initialized) module object in sys.modules. To reload
the module you must first import it again (this will bind the name to
the partially initialized module object) before you can reload() it.

When a module is reloaded, its dictionary (containing the module's
global variables) is retained. Redefinitions of names will override the
old definitions, so this is generally not a problem. If the new version
of a module does not define a name that was defined by the old version,
the old definition remains. This feature can be used to the module's
advantage if it maintains a global table or cache of objects -- with a
try statement it can test for the table's presence and skip its
initialization if desired:


try:
cache
except NameError:
cache = {}


It is legal though generally not very useful to reload built-in or
dynamically loaded modules, except for sys, __main__ and __builtin__.
In many cases, however, extension modules are not designed to be
initialized more than once, and may fail in arbitrary ways when reloaded.

If a module imports objects from another module using from ... import
...., calling reload() for the other module does not redefine the
objects imported from it -- one way around this is to re-execute the
from statement, another is to use import and qualified names
(module.name) instead.

If a module instantiates instances of a class, reloading the module
that defines the class does not affect the method definitions of the
instances -- they continue to use the old class definition. The same is
true for derived classes.
"""

So - expect strange things to happen, sometimes.

I personally _never_ felt the need for reload - after all, writing more
than totally trivial crap on the commandline justifies a small
testscript - which I then just execute on the commandline.

Diez

 

[malv]

Diez,

You are quite right on using module.name when importing module objects.
This happened to be the way I ran my initial successful tests. Later I
modified things by using 'import * from algotest' and my modifications
stopped from coming through.

In my case reload comes in very handy. I can see your point about
testscripts in software development. However this particular project is
in a stage where code writing became now somewhat insignificant
relative to mathematical algorithm development and simultion. Very
quick turnaroud is of prime importance. A testscript wouldn't test
anything without going through a complete setup cycle involving several
stage chains. So online localized modification comes in very handy (of
course you could always goof up in syntax in a modifying a statement -
too bad!)

Diez, thank you very much!

 

[Lisandro Dalcin]

can you send a small example of thing you are trying to do? As Python
is *really* dynamic, you can adapt algorithms in many ways... in
general, you can change methods, or even class! of almost any object...

 

[Bengt Richter]

Read the docs on reload:

http://www.python.org/doc/current/lib/built-in-funcs.html

"""

If a module is syntactically correct but its initialization fails, the
first import statement for it does not bind its name locally, but does
store a (partially initialized) module object in sys.modules. To reload
the module you must first import it again (this will bind the name to
the partially initialized module object) before you can reload() it.

When a module is reloaded, its dictionary (containing the module's
global variables) is retained. Redefinitions of names will override the
old definitions, so this is generally not a problem. If the new version
of a module does not define a name that was defined by the old version,
the old definition remains. This feature can be used to the module's
advantage if it maintains a global table or cache of objects -- with a
try statement it can test for the table's presence and skip its
initialization if desired:


try:
cache
except NameError:
cache = {}


It is legal though generally not very useful to reload built-in or
dynamically loaded modules, except for sys, __main__ and __builtin__.
In many cases, however, extension modules are not designed to be
initialized more than once, and may fail in arbitrary ways when reloaded.

If a module imports objects from another module using from ... import
..., calling reload() for the other module does not redefine the
objects imported from it -- one way around this is to re-execute the
from statement, another is to use import and qualified names
(module.name) instead.

If a module instantiates instances of a class, reloading the module
that defines the class does not affect the method definitions of the
instances -- they continue to use the old class definition. The same is
true for derived classes.
"""

So - expect strange things to happen, sometimes.

I personally _never_ felt the need for reload - after all, writing more
than totally trivial crap on the commandline justifies a small
testscript - which I then just execute on the commandline.

Diez

What about something on the pattern of (untested!)

import algomodule # first time

def algoreload():
algosource = algomodule.__file__.replace('.pyc','py')
algomodule.__dict__.clear()
try:
execfile(algosource, algomodule.__dict__)
algomodule.__file__ = algosource
return 'ok'
except Exception, e:
return '%s: %s -- couldn't execfile %r' %(e.__class__.__name__, e, algosource)

while True:
cmd = raw_input('cmd > ').strip()
if cmd == 'reload': print algoreload()
elif cmd == 'exit': raise SystemExit('ok, exiting ...')
elif cmd == 'edit': print os.popen('notepad.exe '+algomodule.__file__.replace('.pyc','py')).read() or 'ok'
else:
cmd = cmd.split()
args = map(float, cmd[1:])
print getattr(algomodule, cmd[0], (lambda name, *ign: 'No such function: %r'%name).__get__(cmd[0], str))(*args)

this would (theoretically ;-) let you type commands like
sqrt 9
and have alogomodule.sqrt called with float('9'), and then
edit
and edit the module source in notepad, and then
sqrt 16
and have the new function called, etc.

The cmd module will let you set up something fancier than above, and obviously you don't have to run notepad ;-)

Regards,
Bengt Richter

 

[Robert Kern]

What about something on the pattern of (untested!)

import algomodule # first time

def algoreload():
algosource = algomodule.__file__.replace('.pyc','py')
algomodule.__dict__.clear()
try:
execfile(algosource, algomodule.__dict__)
algomodule.__file__ = algosource
return 'ok'
except Exception, e:
return '%s: %s -- couldn't execfile %r' %(e.__class__.__name__, e, algosource)

while True:
cmd = raw_input('cmd > ').strip()
if cmd == 'reload': print algoreload()
elif cmd == 'exit': raise SystemExit('ok, exiting ...')
elif cmd == 'edit': print os.popen('notepad.exe '+algomodule.__file__.replace('.pyc','py')).read() or 'ok'
else:
cmd = cmd.split()
args = map(float, cmd[1:])
print getattr(algomodule, cmd[0], (lambda name, *ign: 'No such function: %r'%name).__get__(cmd[0], str))(*args)

this would (theoretically ;-) let you type commands like
sqrt 9
and have alogomodule.sqrt called with float('9'), and then
edit
and edit the module source in notepad, and then
sqrt 16
and have the new function called, etc.

The cmd module will let you set up something fancier than above, and obviously you don't have to run notepad ;-)

http://ipython.scipy.org

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[malv]

Your algoreload() is of the hand of a master.
I'll give it a try.
Thank you Bengt!