Executing python script stored as a string

[Ecir Hana]

Hello,

please, how to execute a python script stored as a string? But let me
impose several limitations, so simple "exec" wont work:

- if I understood it correctly defining a function in the string and
exec-ing it created the function in current scope. This is something I
really don't want

- simple exec also blocks the rest of the program

- I also would like the string to be able to use and return some parts
of the caller

So to give an example what I try to achieve:

result = []
def up(s):
result.append(s.upper())

code = '''
up("abc")
print 'hello'
i = i + 3
def x(s):
up(s)
x('def')
print i
'''

somehow_execute(code)

Couple of points:

- the script in string should behave just like any other ordinary
python script executed in separate process, except it should also know
about a function caller "up". Nothing else. (I read that something
similar is possible while embedding python into your C project - that
you could invoke the VM and provide some default "imports")

- if the other script runs in separate process how should it call the
remote function? And how to pass its arguments? I really hope I don't
have to serialize every communication, maybe I should use threading
instead of process? All I want is that running it wont block the
caller and that it cannot modify callers code/variables/scope (apart
from calling the predefined callers' functions). Or maybe even better,
let it block the caller but provide a way to stop its execution?

- how to know that the script finished? I was thinking about atexit()
- could it work here?

Think of it as a text editor with a special ability to execute its
content, while providing access of some of its functionality to the
script.

The reason I *think* I cannot just simple import the "editor" module
into the script is that the"editor" is GUI application and script
should have access to just this instance of editor.

Anyway, I hope I was not too confusing. Thanks for any help!

 

[Steven D'Aprano]

Hello,

please, how to execute a python script stored as a string? But let me
impose several limitations, so simple "exec" wont work:

- if I understood it correctly defining a function in the string and
exec-ing it created the function in current scope. This is something I
really don't want

You can pass in a global and local namespaces to exec as arguments:

x = 4
ns = {'x': 4}
exec "x += 1" in ns
x 4
ns['x']

5


See the docs for details.

- simple exec also blocks the rest of the program

Run it in a thread.

- I also would like the string to be able to use and return some parts
of the caller

You can copy the parts of the current scope into the namespace you pass
to exec, then later copy the revised values out again.

But are you sure you really want to take this approach? exec is up to ten
times slower than just executing the code directly. And if the string is
coming from an untrusted source, it is a *huge* security risk.

Couple of points:

- the script in string should behave just like any other ordinary python
script executed in separate process, except it should also know about a
function caller "up". Nothing else. (I read that something similar is
possible while embedding python into your C project - that you could
invoke the VM and provide some default "imports")

If you want it to execute in a separate *process*, that's a whole
different question. If you do that, you get separation of code for free,
as well as separate namespaces. My approach would be to have a special
module "common" which subprocesses can import, to get access to the
shared functions. You will probably need to create some sort of message
passing infrastructure to get results out of the subprocess into the
parent process.

- if the other script runs in separate process how should it call the
remote function? And how to pass its arguments? I really hope I don't
have to serialize every communication, maybe I should use threading
instead of process?

If you want separate processes, they're *separate*. Threads are not.

All I want is that running it wont block the caller
and that it cannot modify callers code/variables/scope (apart from
calling the predefined callers' functions). Or maybe even better, let it
block the caller but provide a way to stop its execution?

As far as I know, you can't kill threads, you can only ask them to kill
themselves.

- how to know that the script finished? I was thinking about atexit() -
could it work here?

I doubt it. You would need to poll each thread to see if it has completed.

Think of it as a text editor with a special ability to execute its
content, while providing access of some of its functionality to the
script.

Something like this?

In the text editor, you have contents:

text goes here
and more text
# Python script starts here
x = 'a'
up(x)
print "foo"
# Python script stops here
more text again


and the user selects lines 4 and 5 and chooses the command "Execute". The
script executes, and its output (foo) is appended to the end of the file:

text goes here
and more text
# Python script starts here
x = 'a'
up(x)
print "foo"
# Python script stops here
more text again
foo

Is this what you mean?

If so, I think you are making this much too complicated for such a simple
use-case. Just publish an API which the script can use, and have the main
text editor application specify a "script" namespace containing only that
API. That could be a module:
42



Then execute the text using exec, but don't bother about putting it into
a thread or subprocess. That just makes it harder to implement, and you
have to worry about concurrency issues.

 

[Ecir Hana]

You can pass in a global and local namespaces to exec as arguments:

x = 4
ns = {'x': 4}
exec "x += 1" in ns
x 4
ns['x']

5

See the docs for details.

Thanks! This is very useful!

You can copy the parts of the current scope into the namespace you pass
to exec, then later copy the revised values out again.

But are you sure you really want to take this approach? exec is up to ten
times slower than just executing the code directly. And if the string is
coming from an untrusted source, it is a *huge* security risk.

I don't know if I should use exec. I don't really mind that it's slow
(btw., why is it so?). But I don't quite understand why is it security
risk. How is it different to run:
exec 'format(your_hdd)'
than:
/bin/python format.py
?

As far as I know, you can't kill threads, you can only ask them to kill
themselves.

Also, I'm not sure if I follow. What does this mean? If a thread runs:

while True:
pass

it is not possible to kill it from another thread? (Bacause it doesn't
check whether some other thread asks to stop it..?)

Something like this?

Well, something more like:

data = [1, 2, 3]
map(lambda x: x * 2, data)
display_data_in_editor_viewport(data) #this renders into part of main
editor window (may take some time)

If so, I think you are making this much too complicated for such a simple
use-case. Just publish an API which the script can use, and have the main
text editor application specify a "script" namespace containing only that
API. That could be a module:


42

Then execute the text using exec, but don't bother about putting it into
a thread or subprocess. That just makes it harder to implement, and you
have to worry about concurrency issues.

Ok, I could try exec, thanks for the explanation. But what about those
security concerns you mentioned above?

Thanks a lot, very informative!

 

[Ecir Hana]

But are you sure you really want to take this approach? exec is up to ten
times slower than just executing the code directly.

Oh, you mean because of parsing and compiling? But otherwise it's as
fast as regular python? That's perfectly ok.

Or maybe I misunderstood you - what do you mean by "executing the code
directly"?

 

[Steven D'Aprano]

I don't know if I should use exec. I don't really mind that it's slow
(btw., why is it so?).

Because it has to parse and compile the string into a code object before
it can run it.

But I don't quite understand why is it security
risk. How is it different to run:
exec 'format(your_hdd)'
than:
/bin/python format.py
?

It's not different. But read what I said -- "if the string is coming from
an UNTRUSTED source" -- presumably you trust yourself. If you run 'exec
"format(your_hdd)"' it is because *you* want to format your hard disk.

Now imagine you have a web-app which gets a string from the user and
calls exec on it. Then you might have this:

exec "search('%d')" % user_input

and the user, who is halfway across the world, enters the following
search string:

places to eat'); import os; os.system('#rm -rf /

Your web app will go right ahead and erase itself. That's why you need to
keep untrusted strings away from exec, execfile, and eval.

Also, I'm not sure if I follow. What does this mean? If a thread runs:

while True:
pass

it is not possible to kill it from another thread? (Bacause it doesn't
check whether some other thread asks to stop it..?)

No, I believe that the only way to halt that is to halt the entire
process.

Possibly there is a way to have a thread halt itself after a certain
amount of time? I'm not an expert on threads, I've hardly ever used them.

 

[Ecir Hana]

It's not different. But read what I said -- "if the string is coming from
an UNTRUSTED source" -- presumably you trust yourself. If you run 'exec
"format(your_hdd)"' it is because *you* want to format your hard disk.

Now imagine you have a web-app which gets a string from the user and
calls exec on it. Then you might have this:

exec "search('%d')" % user_input

and the user, who is halfway across the world, enters the following
search string:

places to eat'); import os; os.system('#rm -rf /

Your web app will go right ahead and erase itself. That's why you need to
keep untrusted strings away from exec, execfile, and eval.

Ah, I see! Ok.

No, I believe that the only way to halt that is to halt the entire
process.

Possibly there is a way to have a thread halt itself after a certain
amount of time? I'm not an expert on threads, I've hardly ever used them.

Thank you once again!

 

[Hendrik van Rooyen]

Possibly there is a way to have a thread halt itself after a certain
amount of time? I'm not an expert on threads, I've hardly ever used them.

Not automagically, as far as I can see.
You are on your own if you want to somehow kill a thread.

What I do is to wrap what I want to do in a while clause:

while runbool:
do_what_you_want()

where runbool is a global that I set to false elsewhere when I want to stop.

There is of course nothing to stop you writing a thread with something like:

import time
start_time = time.time()
while time.time() - start_time < some_parameter:
do_what_you_want()

Which will have the effect of running for a while and then stop.
I cannot see much use use for that as it is, as it will be bloody-minded about
the time, and might chop off before the whole job is done, but one could make
it more intelligent, such that it keeps track of idle time, and aborts after
say a second (or a minute) of not doing anything useful.

Unlike you, I am a thread and queue fanatic - use them all the time.

- Hendrik