Executing bytecode from a string.

[Benjamin Scherrey]

I'm curious as to how difficult it would be to take a string that contains
compiled bytecode, load it into memory, give it a function name then
execute that function. I'm thinking of a database that contains compiled
objects that I can load and execute. I'm also curious as to what level of
grainularity this would work - module, class, class method, function?
Anyone tried to do this before? Obviously dependencies are a consideration
but I'm more interested in the mechanics of this. Appreciate ideas &
pointers you might have...

Ben Scherrey

 

[Michael Hudson]

I'm curious as to how difficult it would be to take a string that contains
compiled bytecode, load it into memory, give it a function name then
execute that function.

Fairly hard: you'd need to guess or work out all the other parameters
to new.code such as argcount, stacksize,...

I'm thinking of a database that contains compiled objects that I can
load and execute.

You'd likely be much better off storing marshalled code objects than
raw code strings. Whether that would suffice is hard to say from your
description, but it's probably a good start.

Or just Python source...

I'm also curious as to what level of grainularity this would work -
module, class, class method, function?

I think "it depends".

Anyone tried to do this before?

Almost certainly

Cheers,
mwh

 

[Peter Hansen]

Fairly hard: you'd need to guess or work out all the other parameters
to new.code such as argcount, stacksize,...

I seem to recall doing this, and it wasn't hard for my particular
case because it wasn't an arbitrary function. It took no arguments,
returned nothing, etc...

I was doing this as an experiment to see whether it was feasible
to use evolutionary programming to evolve Python bytecode that
would solve some problem.

The idea worked in principle (I was able to get the bytecode to
execute) but unfortunately there were numerous instances where the
effectively random bytecode would lead to the interpreter crashing
fatally.

I still think it's a neat idea, but it won't work well with Python
in its current form.

Might be useful for doing some kind of stress testing on the
interpreter though, if someone was interested in making it
bullet-proof even for psychotic code (as part of a new security
initiative, perhaps?).

And sorry, but this was a quickie experiment and I don't have the
code any more. Just wanted to say it was doable for the simplest
case.

-Peter

 

[John Roth]

I'm curious as to how difficult it would be to take a string that contains
compiled bytecode, load it into memory, give it a function name then
execute that function. I'm thinking of a database that contains compiled
objects that I can load and execute. I'm also curious as to what level of
grainularity this would work - module, class, class method, function?
Anyone tried to do this before? Obviously dependencies are a consideration
but I'm more interested in the mechanics of this. Appreciate ideas &
pointers you might have...

Ben Scherrey

Look at the 'new' module. It's in the Library Reference under
Python Runtime Services. You'll need to wrap the byte string
with 'code', and then wrap that with 'function'.

As someone else pointed out, there are a number of other
parameters you'll need to supply, and some of them aren't
at all obvious. You may find the disassembler (documented
under Python language services) to be helpful in figuring
some of them out.

John Roth

 

[Fredrik Lundh]

Look at the 'new' module. It's in the Library Reference under
Python Runtime Services. You'll need to wrap the byte string
with 'code', and then wrap that with 'function'.

that's insane, though. use marshal on code objects, and keep your
sanity.

see page 4-10 in this document for a small sample:

http://www.effbot.org/zone/librarybook/data-representation.pdf

to avoid chaos when you upgrade Python, it's a good idea to attach
imp.get_magic() to the marshalled string, and verify it before you un-
marshall the code; see:

http://mail.python.org/pipermail/python-list/2003-November/196230.html

</F>

 

[Peter Otten]

I'm curious as to how difficult it would be to take a string that contains
compiled bytecode, load it into memory, give it a function name then
execute that function. I'm thinking of a database that contains compiled
objects that I can load and execute. I'm also curious as to what level of
grainularity this would work - module, class, class method, function?
Anyone tried to do this before? Obviously dependencies are a consideration
but I'm more interested in the mechanics of this. Appreciate ideas &
pointers you might have...

Ben Scherrey

The following was written just for fun with no pretension that it will work
in the "real world". I've only manipulated co_consts and co_code of my
custom code object - you can investigate the other parameters one after
another and replace them with something at your choice.
The question that remains - how do you want to create the bytecode strings?
If from actual functions, why not store their source code, if from
codeobjects, why not marshal.dump()/load() them?

Peter

<code>
# get hold of the function and code types
def f(): pass
function = type(f)
del f
code = type(compile("def f(): pass", "noname", "exec"))

template = compile("def f(): pass", "noname", "exec")

def makeFunc(bytecode, consts):
""" create a new function from a bytecode string and a consts tuple """

# instead of inventing all code-attributes ourselves
# copy those we are not interested in from an existing template
co = template
codeobj = code(co.co_argcount, co.co_nlocals, co.co_stacksize,
co.co_flags, bytecode, consts, co.co_names,
co.co_varnames, co.co_filename, co.co_name,
co.co_firstlineno, co.co_lnotab, co.co_freevars,
co.co_cellvars)

return function(codeobj, globals(), "f", None, None)

# get hold of a valid bytecode string
# (I could have pasted the string literal from the interpreter,
# but I chose not to cheat
def prototype():
print "so what"
bytecode = prototype.func_code.co_code

# test what we have so far
g = makeFunc(bytecode, consts=(None, "it takes more than the byte-code",))
h = makeFunc(bytecode, consts=(None, "to make a function"))
g()
h()
</code>

 

[Markus von Ehr]

I'm curious as to how difficult it would be to take a string that contains
compiled bytecode, load it into memory, give it a function name then
execute that function. I'm thinking of a database that contains compiled
objects that I can load and execute. I'm also curious as to what level of
grainularity this would work - module, class, class method, function?
Anyone tried to do this before? Obviously dependencies are a consideration
but I'm more interested in the mechanics of this. Appreciate ideas &
pointers you might have...

Ben Scherrey

I am just writing my Macro/Script in python sourcecode.
In the program I'm doing:

f = open('segment.py','r')
str = f.read()
f.close()

segment_code = compile(str, '<string>', 'exec')

exec(segment_code)

Markus