Dynamically defined functions via exec in imported module

[Nadeem]

Hello all,
I'm trying to write a function that will dynamically generate other
functions via exec. I then want to be able to import the file (module)
containing this function and use it in other modules, but for some
reason it only works using the "import <mod>" syntax, and not "from
<mod> import *" syntax... i.e. in the latter case, the function is
dynamically generated, but not accessible from the importing module.
Any ideas on what I can do to be able to retain the second form of
import and still have the exec'd functions visible?

Here's the code... I have three files:

###################################
# modA.py

def dynamicdef(name, amt):
'''Dynamically defines a new function with the given name that
adds
the given amt to its argument and returns the result.'''
stm = "def %s(x):\n\treturn x + %d" % (name, amt)
print stm
# exec stm # --- with this, 'name' is only accessible within
this fn
exec stm in globals() # --- this makes it global within this
module...

print eval(name)


dynamicdef('plus5', 5)

print plus5(7)


###################################
# modB.py
# This uses the dynamicdef to dynamically define a new function, and
it
# works fine, with the newly defined function being accessible thru
the modA
# module...

import modA

modA.dynamicdef('plus10', 10)

print help(modA.plus5)
print help(modA.plus10)

print modA.plus5(20)
print modA.plus10(20)



###################################
# modC.py
# This uses the dynamicdef to dynamically define a new function, but
it
# doesn't work; seems like knowledge of the newly defined function in
the
# modA module is not propagated back to this context or something...?

from modA import *

dynamicdef('plus10', 10)

print help(plus5)
#print help(plus10) # !!! Fails: NameError: name 'plus10' is not
defined

print plus5(20)
print plus10(20)
###################################


Thanks for any help/suggestions,
---nadeem

 

[George Sakkis]

Hello all,
I'm trying to write a function that will dynamically generate other
functions via exec.

General tip: whenever you think you need to use exec (or eval), 99% of
the time you don't; most of the time there is a better (meaning, less
fragile and obscure) solution.

I then want to be able to import the file (module)
containing this function and use it in other modules, but for some
reason it only works using the "import <mod>" syntax, and not "from
<mod> import *" syntax... i.e. in the latter case, the function is
dynamically generated, but not accessible from the importing module.
Any ideas on what I can do to be able to retain the second form of
import and still have the exec'd functions visible?

Here's the code... I have three files:

###################################
# modA.py

def dynamicdef(name, amt):
    '''Dynamically defines a new function with the given name that
adds
    the given amt to its argument and returns the result.'''
    stm = "def %s(x):\n\treturn x + %d" % (name, amt)
    print stm
    # exec stm      # --- with this, 'name' is only accessible within
this fn
    exec stm in globals()   # --- this makes it global within this
module...

    print eval(name)

dynamicdef('plus5', 5)

print plus5(7)

Unsurprisingly, there is indeed a better way, a closure:

def adder(amt):
def closure(x):
return x + amt
return closure
12

HTH,
George

 

[Nadeem]

I understand the 99% rule... the example I gave was to simplify the
issue. The full thing I'm working on is a library for an introductory
CS class I'm teaching. I'm trying, essentially, to build a library of
macros for students to use allowing them to define records (like
structs in C) with selector functions. In particular, I'm trying to
replicate some of the Scheme stuff from the HtDP project in Python
(http://www.htdp.org/2003-09-26/Book/curriculum-Z-
H-9.html#node_sec_6.3). I want to provide a function, called
defineStruct that is called like this:

defineStruct('pos', 'x', 'y')

The effect of this function will be to dynamically define several new
functions for working with structures:

makePos(x, y)
posX(p)
posY(p)
isPos(p)

I understand that all this can be done with classes and OO
programming, but the whole point of the HtDP curriculum is to
introduce students to programming in a pedagogically-effective way
using a functional approach instead of OO-first. They do it in Scheme,
which is primarily a f.p. language, and I'm trying to replicate a
similar approach in Python. The defineStruct thing is basically meant
to be a macro that introduces a set of functions for whatever
structure definition is needed.

So, for these reasons, I don't believe the closure example above is
helpful. I don't want to have to tell students anything about
closures, and certainly have them worrying about functions returning
functions, and function pointers, etc. I'm trying to bundle all that
up behind the scenes.

So, thinking about my problem again, an alternate question may be: Is
it possible, in a function called in a module, to access and update
the global definitions (dictionary or whatever) in the caller module.

--- nadeem

 

[Nadeem]

Well, I found one hack that seems to achieve this by accessing the
globals() dictionary of the outermost stack frame and adding an entry
to it for the newly created functions:

import inspect

def dynamicdef(name, amt):
'''Dynamically defines a new function with the given name that
adds
the given amt to its argument and returns the result.'''
stm = "def %s(x):\n\treturn x + %d" % (name, amt)
print stm
exec stm in globals()

## ADDED THIS TO EXPORT THE NEW FUNCTION NAME TO THE TOP LEVEL...
inspect.stack()[-1][0].f_globals[name] = eval(name)


I guess I'll go with this unless someone suggests an alternate. Thanks
anyway,
--- nadeem

 

[Calvin Spealman]

I understand the 99% rule... the example I gave was to simplify the
issue. The full thing I'm working on is a library for an introductory
CS class I'm teaching. I'm trying, essentially, to build a library of
macros for students to use allowing them to define records (like
structs in C) with selector functions. In particular, I'm trying to
replicate some of the Scheme stuff from the HtDP project in Python
(http://www.htdp.org/2003-09-26/Book/curriculum-Z-
H-9.html#node_sec_6.3). I want to provide a function, called
defineStruct that is called like this:

defineStruct('pos', 'x', 'y')

The effect of this function will be to dynamically define several new
functions for working with structures:

makePos(x, y)
posX(p)
posY(p)
isPos(p)

def defineStruct(name, *parameters):
class _struct:
def __init__(self, *init_parameters):
for pname, pvalue in zip(parameters, init_parameters):
setattr(self, pname, pvalue)
globals()["make" + name] = _struct
for parameter in parameters:
def getter(o, parameter=parameter):
return getattr(o, parameter)
globals()[name + parameter] = getter
globals()["is" + name] = lambda o: isinstance(o, _struct)

You might do other things, of course, like stepping up the frames from
sys._getframe() to inject the functions into the callers global scope.
There are some obvious optimizations you could make, too, as well as
other "methods" you might want to add.

 

[Nadeem]

That's a really neat way of doing it, thanks a lot! I hadn't realized
how accessible all those globals() dictionaries were. Guess my example
still falls in the 99%...
--- nadeem

def defineStruct(name, *parameters):
  class _struct:
    def __init__(self, *init_parameters):
      for pname, pvalue in zip(parameters, init_parameters):
        setattr(self, pname, pvalue)
  globals()["make" + name] = _struct
  for parameter in parameters:
    def getter(o, parameter=parameter):
      return getattr(o, parameter)
    globals()[name + parameter] = getter
  globals()["is" + name] = lambda o: isinstance(o, _struct)

You might do other things, of course, like stepping up the frames from
sys._getframe() to inject the functions into the callers global scope.
There are some obvious optimizations you could make, too, as well as
other "methods" you might want to add.

I understand that all this can be done with classes and OO
programming, but the whole point of the HtDP curriculum is to
introduce students to programming in a pedagogically-effective way
using a functional approach instead of OO-first. They do it in Scheme,
which is primarily a f.p. language, and I'm trying to replicate a
similar approach in Python. The defineStruct thing is basically meant
to be a macro that introduces a set of functions for whatever
structure definition is needed.

So, for these reasons, I don't believe the closure example above is
helpful. I don't want to have to tell students anything about
closures, and certainly have them worrying about functions returning
functions, and function pointers, etc. I'm trying to bundle all that
up behind the scenes.

So, thinking about my problem again, an alternate question may be: Is
it possible, in a function called in a module, to access and update
the global definitions (dictionary or whatever) in the caller module.

--- nadeem

 

[Michele Simionato]

I understand the 99% rule... the example I gave was to simplify the
issue. The full thing I'm working on is a library for an introductory
CS class I'm teaching. I'm trying, essentially, to build a library of
macros for students to use allowing them to define records (like
structs in C) with selector functions.

The namedtuple recipe by Raymond Hetting does
exactly that and, guess what, it uses exec!
Also the doctest module in the standard library
uses exec at good effect. So, I would say it is
not a 99% rule, let's say it is a 98% rule
BTW, I also write a paper on namedtuple which I
am going to translate in English soon or later
(for the moment there is Google Translator):
http://stacktrace.it/articoli/2008/05/gestione-dei-record-python-1/
The paper, just as the namedtuple recipe,
has a strong functional bias so it may be of interest
to people coming from Scheme.

Michele Simionato

 

[George Sakkis]

The namedtuple recipe by Raymond Hetting does
exactly that and, guess what, it uses exec!

I might be wrong, but the reason namedtuple uses exec is performance.
IIRC earlier versions of the recipe used a metaclass instead, so it's
not that it *has* to use exec, it's just an optimization, totally
justified in this case since namedtuples should be almost as fast as
plain tuples.

George

 

[Fredrik Lundh]

I understand that all this can be done with classes and OO
programming, but the whole point of the HtDP curriculum is to
introduce students to programming in a pedagogically-effective way
using a functional approach instead of OO-first.

And yet, one of the HtDP authors just posted a paper that argues that
it's time to abandon the whole idea of "programming language paradigms"
in teaching, and focus on language features instead.

Most books rigorously adhere to the sacred division of languages
into "functional", "imperative", "object-oriented", and "logic"
camps. I conjecture that this desire for taxonomy is an artifact
of our science-envy from the early days of our discipline: a
misguided attempt to follow the practice of science rather than
its spirit. We are, however, a science of the artiï¬cial. What
else to make of a language like Python, Ruby, or Perl? Their
designers have no patience for the niceties of these Linnaean
hierarchies; they borrow features as they wish, creating melanges
that utterly defy characterization. How do we teach PL in this
post-Linnaean era?

http://www.cs.brown.edu/~sk/Publications/Papers/Published/sk-teach-pl-post-linnaean/

Alright, his emphasis is on teaching programming language *design*
features (and there's still the usual tone of "Scheme is right all the
time and the others make mistakes all the time", despite the recent RS6S
brouhaha ;-), but I'd say you should think twice before exposing your
students to an absurdly artificial rendering of one language's model in
another language's syntax. It's trivial to show how Python's method
*syntax* is used to associate ordinary functions with "attribute
containers" without having to first introduce OO as a formal concept.

</F>

 

[Maric Michaud]

Le Saturday 16 August 2008 06:50:02 Michele Simionato, vous avez écrit :

The namedtuple recipe by Raymond Hetting does
exactly that and, guess what, it uses exec!

It uses exec, but could not, and IMO, should not, all of the cooking could be
done in the closure. I join version of it without exec. Of course the only
drawback is that you need to do the job of argument checking in __new__ and
can't rely on a dynamically compiled arglist. But it is a common idiom I
think.

Also the doctest module in the standard library
uses exec at good effect. So, I would say it is
not a 99% rule, let's say it is a 98% rule

Yes, but in doctest, exec is used for what it is intended to be use, execute
arbitrary code provided by the user, this is legal and perfect use of exec
because here the user is a trusted one (the programer himself).

I'd say that everywhere exec/eval are used in a application/function/lib that
doesn't mean to interpret arbitrary and user provided python code, it is a
bad usage.

 

[Michele Simionato]

On Aug 16, 12:50 am, Michele Simionato <[email protected]>
wrote:

I might be wrong, but the reason namedtuple uses exec is performance.
IIRC earlier versions of the recipe used a metaclass instead, so it's
not that it *has* to use exec, it's just an optimization, totally
justified in this case since namedtuples should be almost as fast as
plain tuples.

I always thought the reason for "exec" was having a good signature in
the namedtuple constructor. I have not
measure performance, though.

 

[Michele Simionato]

I'd say that everywhere exec/eval are used in a application/function/lib that
doesn't mean to interpret arbitrary and user provided python code, it is a
bad usage

Problem is, there are things you cannot do without
exec/eval: for instance look at the implementation of
my own decorator module, which uses eval to
guarantee a good signature for the decorated function.
Probably it could be done without exec, with
bytecode hacks, but I regard fiddling with bytecodes
as something worse than exec itself. I am not
sure if in Python 3.0 the signature object is
powerful enough to do the job, I never had to
time to check how it works.

Michele Simionato