reference or pointer to some object?

[Torsten Mohr]

Hi,

i'd like to pass a reference or a pointer to an object
to a function. The function should then change the
object and the changes should be visible in the calling
function.

In perl this would be something like:

sub func {
$ref = shift;

$$ref += 123; # change
}

$a = 1;
func(\$a);

is something like this possible in python?

The keyword "global" does NOT fit this purpose to
my understanding as it only makes the variables of
the UPPERMOST level visible, not the ones of ONE
calling level above.

Is this somehow possible with weakref?

I don't want to pass the parameter to a function and
then return a changed value.

Is there some other mechanism in python available to
achieve a behaviour like this?


Thanks for any hints,
Torsten.

 

[Paul Rubin]

i'd like to pass a reference or a pointer to an object
to a function. The function should then change the
object and the changes should be visible in the calling
function.

Normally you would pass a class instance or boxed object, and let the
function change the instance or object:

def bump(b):
b[0] += 123 # change

x = [5]
bump(x)
print x # prints [128]

 

[Peter Maas]

i'd like to pass a reference or a pointer to an object
to a function. The function should then change the
object and the changes should be visible in the calling
function. [..]
is something like this possible in python?

Yes, wrap it in a container, e.g. a list or an object.
Change the containers content in the called function.

The keyword "global" does NOT fit this purpose to
my understanding as it only makes the variables of
the UPPERMOST level visible, not the ones of ONE
calling level above.

There are three namespaces in python, sorted according to
priority:

- local
variables of the current scope (function or method),
highest priority, show with locals()

- global
variables of the containing module, show with globals()

- builtin
builtin variables, show with __builtins__.__dict__

Since Python 2.1 the local namespace can be nested e.g. if
a function is defined inside a function. Example:
.... def fi(v):
.... v2 = 2*v
.... print locals()
.... return v2
.... u2 = fi(u)
.... print locals()
.... return u2
....{'v2': 8, 'v': 4} <-- inner local namespace
{'fi': <function fi at 0x011AEBB0>, 'u': 4, 'u2': 8} <-- outer local namespace
8

 

[Jeff Shannon]

Hi,

i'd like to pass a reference or a pointer to an object
to a function. The function should then change the
object and the changes should be visible in the calling
function.

There are two possible meanings of "change the object" in Python. One
of them will "just work" for your purposes, the other won't work at all.

Python can re-bind a name, or it can mutate an object. Remember,
names are just convenient labels that are attached to an object in
memory. You can easily move the label from one object to another, and
the label isn't affected if the object it's attached to undergoes some
sort of change.

Passing a parameter to a function just creates a new label on that
object, which can only be seen within that function. The object is
the same, though. You can't change what the caller's original label
is bound to, but you *can* modify (mutate) the object in place.
.... somedict['foo'] = 'bar'
.... .... somedict = {'foo':'bar'}
....
In mutate(), we take the object (which is d in the caller, and
somedict in the function) and mutate it. Since it's the same object,
it doesn't matter where the mutation happened. But in rebind(), we're
moving the somedict label to a *new* dict object. Now d and somedict
no longer point to the same object, and when the function ends the
object pointed to by somedict is garbage-collected, while the object
pointed to by d has never changed.

So, to do what you want to do, you simply need to arrange things so
that your parameter is an object that can be mutated in-place.

Jeff Shannon
Technician/Programmer
Credit International

 

[Torsten Mohr]

Hi,

thank you all for your explanations.

I still wonder why a concept like "references" was not
implemented in Python. I think it is (even if small)
an overhead to wrap an object in a list or a dictionary.

Isn't it possible to extend Python in a way to use
real references? Or isn't that regarded as necessary?


Thanks for any hints,
Torsten.

 

[JCM]

I still wonder why a concept like "references" was not
implemented in Python. I think it is (even if small)
an overhead to wrap an object in a list or a dictionary.

Isn't it possible to extend Python in a way to use
real references? Or isn't that regarded as necessary?

Some think it's unwise. The ability to modify local variables in your
caller's scope can lead to messiness, especially if the modifiability
isn't evident to the caller (eg. via C's "&").

 

[Steven Bethard]

I still wonder why a concept like "references" was not
implemented in Python. I think it is (even if small)
an overhead to wrap an object in a list or a dictionary.

Isn't it possible to extend Python in a way to use
real references? Or isn't that regarded as necessary?

IMHO it's not really necessary. As long as you're passing a mutable
object around, when you call one of its methods, the object will be
changed. So the only time you'd want a "reference" is if you were
passing around an immutable object (e.g. an int, float, str or tuple)
and you wanted to change it. The thought of changing an immutable
object seems kind of silly to me.

Could you give us a more concrete use case? My suspicion is that
anything complicated enough to be passed to a method to be modified will
probably be more than a simple int, float, str or tuple... In which
case, it will probably have methods to allow you to update it...

In my case, rather than your original example, which you want to look
something like:

def func(x):
x += 123

x = 5
func(x)

I'd just write:

x = 5
x += 123

=)

Steve

 

[Jeff Shannon]

I still wonder why a concept like "references" was not
implemented in Python. I think it is (even if small)
an overhead to wrap an object in a list or a dictionary.

Because Python uses a fundamentally different concept for variable
names than C/C++/Java (and most other static languages). In those
languages, variables can be passed by value or by reference; neither
term really applies in Python. (Or, if you prefer, Python always
passes by value, but those values *are* references.) Python doesn't
have lvalues that contain rvalues; Python has names that are bound to
objects. Passing a parameter just binds a new name (in the called
function's namespace) to the same object.

It's also rather less necessary to use references in Python than it is
in C et. al. The most essential use of references is to be able to
get multiple values out of a function that can only return a single
value. Where a C/C++ function would use the return value to indicate
error status and reference (or pointer) parameters to communicate
data, a Python program will return multiple values (made quick & easy
by lightweight tuples and tuple unpacking) and use exceptions to
indicate error status. Changing the value of a parameter is a
side-effect that complicates reading and debugging code, so Python
provides (and encourages) more straightforward ways of doing things.

Jeff Shannon
Technician/Programmer
Credit International

 

[Steven Bethard]

Because Python uses a fundamentally different concept for variable names
than C/C++/Java (and most other static languages). In those languages,
variables can be passed by value or by reference; neither term really
applies in Python. (Or, if you prefer, Python always passes by value,
but those values *are* references.) Python doesn't have lvalues that
contain rvalues; Python has names that are bound to objects. Passing a
parameter just binds a new name (in the called function's namespace) to
the same object.

Point of clarification: Java objects (but not Java primitive types like
ints or doubles) work almost identically to Python objects -- you can
basically think of them as references being passed by value. There's no
way to pass by reference in Java (even for Java primitive types).

Steve

 

[Antoon Pardon]

Because Python uses a fundamentally different concept for variable
names than C/C++/Java (and most other static languages). In those
languages, variables can be passed by value or by reference; neither
term really applies in Python. (Or, if you prefer, Python always
passes by value, but those values *are* references.)

I would think the reference was the id. As such python always
passes by reference, as the id of the parameter is the id
of the argument.

Python doesn't
have lvalues that contain rvalues; Python has names that are bound to
objects. Passing a parameter just binds a new name (in the called
function's namespace) to the same object.

It's also rather less necessary to use references in Python than it is
in C et. al.

You use nothing but references in Python, that is the reason why
if you assign a mutable to a new name and modify the object through
either name, you see the change through both names.

 

[Jeff Shannon]

You use nothing but references in Python, that is the reason why
if you assign a mutable to a new name and modify the object through
either name, you see the change through both names.

Perhaps it would've been better for me to say that the sorts of
problems that are solved by (pointers and) references in C/C++ can be
better solved in other ways in Python...

One can take the position that every variable in Python is a
reference; the semantics work out the same. But I find it clearer to
view the Python model as conceptually distinct from the "classic"
value/reference model. Re-using the old terms is likely to lead to
making mistakes based on inapplicable presumptions.

Jeff Shannon
Technician/Programmer
Credit International

 

[Alex Martelli]

Because Python uses a fundamentally different concept for variable
names than C/C++/Java (and most other static languages). In those
languages, variables can be passed by value or by reference; neither
term really applies in Python. (Or, if you prefer, Python always

Java's model is quite similar to Python's, except that Java makes some
exception for very low-level types such as int or float. But there is
no "passing by reference" in Java, anyway: the semantics of assignment
and argument passing (same thing) are VERY close in Java and Python.

I hope this is enough to show that static or dynamic is quite a
different issue -- Java's more or less statically typed, Python
dynamically, but their model of variables is VERY similar anyway.


Alex

 

[Torsten Mohr]

Hi,

Could you give us a more concrete use case? My suspicion is that
anything complicated enough to be passed to a method to be modified will
probably be more than a simple int, float, str or tuple... In which
case, it will probably have methods to allow you to update it...

yes, to be more explicit: I'm quite new to python and i wrote
a small function that does a hexdump of a string. That string
can be quite large, so i suspected a large overhead when the
string would be copied and handed over to the function.

But i think my understanding was wrong (though it is not yet
clear). If i hand over a large string to a function and the
function had the possibility to change it, wouldn't that mean
that it is necessary to hand over a _copy_ of the string?
Else, how could it be immutable?

Thinking about all this i came to the idea "How would i write
a function that changes a string with not much overhead?".

def func(s):
change s in some way, remove all newlines, replace some
charaters by others, ...
return s

s = func(s)

This seems to be a way to go, but it becomes messy if i hand over
lots of parameters and expect some more return functions.

Maybe it is because i did lots of perl programming, but

func(\$s) looks easier to me.

In my case, rather than your original example, which you want to look
something like:

def func(x):
x += 123

x = 5
func(x)

I'd just write:

x = 5
x += 123

You're right, of course. I'm sorry the second example is still
a bit constructed, but i came across it by writing the hexdump
utility and wanted to reduce overhead.


Best regards,
Torsten.

 

[Jeff Shannon]

But i think my understanding was wrong (though it is not yet
clear). If i hand over a large string to a function and the
function had the possibility to change it, wouldn't that mean
that it is necessary to hand over a _copy_ of the string?
Else, how could it be immutable?

Anything which might change the string, can only do so by returning a
*new* string.

Saying that strings are immutable means that, when 'a' is pointing to
a string, the (string) object that 'a' points to will always be the
same. (Unless 'a' is re-bound, or someone uses some deep black magic
to change things "behind the scenes"...) No method that I call on
'a', or function that I pass 'a' to, can alter that object -- it can
only create a new object based off of the original. (You can
demonstrate this by checking the id() of the objects.)

Mutable objects, on the other hand, can change in place. In the case
of lists, for example, it will stay the same list object, but the list
contents can change.

Note, also, that passing a string into a function does not copy the
string; it creates a new name binding (i.e. reference) to the same object.
.... print "Initial ID:", id(item)
.... item = item.replace('first', 'second')
.... print "Resulting ID:", id(item)
.... Initial ID: 26278416
Resulting ID: 26322672

Thinking about all this i came to the idea "How would i write
a function that changes a string with not much overhead?".

Since strings cannot really be changed, you simply try to minimize the
number of new strings created. For example, appending to a string
inside of a for-loop creates a new string object each time, so it's
generally more efficient to convert the string to a list, append to
the list (which doesn't create a new object), and then join the list
together into a string.

def func(s):
change s in some way, remove all newlines, replace some
charaters by others, ...
return s

s = func(s)

This seems to be a way to go, but it becomes messy if i hand over
lots of parameters and expect some more return functions.

This has the advantage of being explicit about s being (potentially)
changed. References, in the way that you mean them, are even messier
in the case of numerous parameters because *any* of those parameters
might change. By simply returning (new) objects for all changes, the
function makes it very clear what's affected and what isn't.

Jeff Shannon
Technician/Programmer
Credit International

 

[Reinhold Birkenfeld]

Hi,


yes, to be more explicit: I'm quite new to python and i wrote
a small function that does a hexdump of a string. That string
can be quite large, so i suspected a large overhead when the
string would be copied and handed over to the function.

It isn't.

But i think my understanding was wrong (though it is not yet
clear). If i hand over a large string to a function and the
function had the possibility to change it, wouldn't that mean
that it is necessary to hand over a _copy_ of the string?
Else, how could it be immutable?

You cannot modify a string. Notice that there are no in-place string
methods -- str.strip() for example returns a new string.

Thinking about all this i came to the idea "How would i write
a function that changes a string with not much overhead?".

Basically, working with very large strings is costly (it saves overhead
otherwise). So do make smaller parts and operate on these chunks.

def func(s):
change s in some way, remove all newlines, replace some
charaters by others, ...
return s

s = func(s)

This seems to be a way to go, but it becomes messy if i hand over
lots of parameters and expect some more return functions.

You can use tuples for that. Automatic tuple packing helps:

def func(x, y):
# change x, y and generate z

return x, y, z

x, y, z = func(x, y)

Maybe it is because i did lots of perl programming, but

func(\$s) looks easier to me.

It does, but in fact the problem is not that the string is passed by
value, but that the string is not modifiable...

Reinhold

 

[Torsten Mohr]

Hi,

thank you all for your explanations.

That's really great and helps me a lot.


Thanks,
Torsten.