"Can't return a temporary from lvalue subroutine..."

[J Krugman]

I'm OK with coding objects, but I find tied variables a bit
terrifying...

Anyway, it is common to implement objects as hashes (or rather as
references to hashes). My question is, is it possible to use a
(reference to a) tied hash instead of a (reference to a) regular
hash for this? I would have thought yes, but if I try to use a
tied hash instead of a regular hash I get this strange error

Can't return a temporary from lvalue subroutine...

This happens when I attempt to assign to an lvalue method (which
works fine if I don't use the tied hash).

What "temporary" is the error message referring to? Is there a
work around?

I give the relevant code below.

TIA,

jill

package Helper;
use Tie::Hash;
our @ISA = 'Tie::StdHash';

sub STORE {
my ($self, $key, $value) = @_;
warn "storing value $value under key $key\n";
$self->{$key} = $value;
}

1;
__END__

package My_Class;
use Helper;

sub new {
my $class = shift;
my %self = ();
tie %self, 'Helper'; # OK if this line is commented out
return bless \%self, $class;
}

sub var : lvalue {
my $self = shift;
$self->{_var_} = shift if @_;
$self->{_var_};
}

1;
__END__

use My_Class;

my $v = My_Class->new();
$v->var = 42; # fatal error

 

[Mr P]

I'm OK with coding objects, but I find tied variables a bit
terrifying...

 

[Anno Siegel]

I'm OK with coding objects, but I find tied variables a bit
terrifying...

It's quite ordinary magic.

Anyway, it is common to implement objects as hashes (or rather as
references to hashes). My question is, is it possible to use a
(reference to a) tied hash instead of a (reference to a) regular
hash for this? I would have thought yes, but if I try to use a
tied hash instead of a regular hash I get this strange error

Can't return a temporary from lvalue subroutine...

This happens when I attempt to assign to an lvalue method (which
works fine if I don't use the tied hash).

What "temporary" is the error message referring to? Is there a
work around?

It looks like "temporary" refers to the state of a scalar of being
"mortal". Mortal SVs are all over the Perl source, but normally
only concern XS programmers. This is the first time I see the
distinction appear at Perl level.

I give the relevant code below.

The code is reasonable, I would have written it similarly, except for

sub var : lvalue {
my $self = shift;
$self->{_var_} = shift if @_;

This line is now redundant, or it would be if the code worked as expected.
You're assigning through the lvalue now. It isn't exactly wrong to have
it, but it distracts from the purpose of the lvalue-method.

$self->{_var_};
}

I, too would have expected the construction to work. I'm not sure what's
wrong, but then lvalue subs are still (permanently?) experimental, so
irregularities must be expected.

For a workaround, instead of tying the whole hash, you can tie each
hash value to a scalar. That works without the error. Like a good
workaround, it has serious disadvantages.

For one, you must either pre-tie all fields in the ->new method of
My_Class, in which case you can't add fields dynamically. If you must
do that, the ->var method(s) must do the tying on the fly when a new
key is generated.

Another problem is that the STORE method of a tied scalar doesn't know
about the hash key the value is stored under. Since each tied scalar
belongs to a fixed key, it would be possible to store the hash key in
the tying object (the one in Tie::Scalar...). But that's extra work,
and it means you can't use Tie::StdScalar as is, because it has no
provisions for extra values.

I suppose your motivation is that you like the idea of lvalue methods,
but need more control over what gets stored by careless users in your
sensitive objects. A similar problem exists with objects that expose
(parts of) their interior though overloading a dereference operator,
say %{}. Like lvalues, it gives you pretty syntax, but leaves your
objects wide open. I have used tied hashes to correct this (that is,
returned a reference to a tied hash in response to %{ $obj}, and haven't
encountered the particular difficulty you're seeing. That may be
an alternative approach.

Anno

 

[Anno Siegel]

I'm OK with coding objects, but I find tied variables a bit
terrifying...

.
.
.

terrifying?? Dude if a programming language is terrifying do NOT watch
the news tonight! Indimidating I think is what you meant?
[...]

Tying is a way to tell the O/S "store THIS variable result on disk, NOT
in memory". That's pretty much it as long as your variable construct is
simple. If not, its more more involved since "tying" only supports
simple structures. To get beyond that you may need data dumper or
something else.

You speak with authority, MisterPerl, but you are wrong. Tying as such
has nothing to do with storage on disk, or complexity of data structures.

What you say is more or less true of tying to one of the *DBM_File
modules, but you seem to be unaware of other applications of tie().

Jill's question is about one of those other applications.

Anno

 

[J Krugman]

The code is reasonable, I would have written it similarly, except for

This line is now redundant, or it would be if the code worked as expected.
You're assigning through the lvalue now. It isn't exactly wrong to have
it, but it distracts from the purpose of the lvalue-method.

My thought was to give users the option setting through the lvalue
*or* through the more conventional

$self->var(1);

....but maybe this is not such a hot idea...

For a workaround, instead of tying the whole hash, you can tie each
hash value to a scalar.

That's good to know. I was under the mistaken impression that only
"simple" scalar variables (i.e. like $foo, as opposed to $foo{bar})
could be tied to scalars.

For one, you must either pre-tie all fields in the ->new method of
My_Class, in which case you can't add fields dynamically. If you must
do that, the ->var method(s) must do the tying on the fly when a new
key is generated.

You lost me there, Anno. I don't see how, in general, an lvalue
method could do any on-the-fly tying, since all but the last
statement of the method are ignored when it is used as an lvalue.
(Yes, it could do it if it is being used as a non-lvalue method,
but that imposes a strange API).

Another problem is that the STORE method of a tied scalar doesn't know
about the hash key the value is stored under. Since each tied scalar
belongs to a fixed key, it would be possible to store the hash key in
the tying object (the one in Tie::Scalar...). But that's extra work,
and it means you can't use Tie::StdScalar as is, because it has no
provisions for extra values.

(A bit over my head, but that's OK: I know have a lot to learn.)

I suppose your motivation is that you like the idea of lvalue methods,
but need more control over what gets stored by careless users in your
sensitive objects.

Exactly! Very clairvoyant of you.

A similar problem exists with objects that expose
(parts of) their interior though overloading a dereference operator,
say %{}. Like lvalues, it gives you pretty syntax, but leaves your
objects wide open. I have used tied hashes to correct this (that is,
returned a reference to a tied hash in response to %{ $obj}, and haven't
encountered the particular difficulty you're seeing. That may be
an alternative approach.

I'm having a hard time picturing what you describe here. It seems
to imply that $obj somehow knows when it's being dereferenced (and
responds accordingly by "returning" a tied hash???), which I find
hard to understand. Is this something you do in any published code
that I could study?

Thank you very much for your post.

jill

 

[attn.steven.kuo]

(snipped)


It looks like "temporary" refers to the state of a scalar of being
"mortal". Mortal SVs are all over the Perl source, but normally
only concern XS programmers. This is the first time I see the
distinction appear at Perl level.

Found in the perl source code:

http://search.cpan.org/src/NWCLARK/perl-5.8.7/pp_hot.c

It appears values on the Perl stack are being checked for
their SvFLAGS to determine whether the subroutine is returning
a "temporary".


(snipped)

For a workaround, instead of tying the whole hash, you can tie each
hash value to a scalar. That works without the error. Like a good
workaround, it has serious disadvantages.

For one, you must either pre-tie all fields in the ->new method of
My_Class, in which case you can't add fields dynamically. If you must
do that, the ->var method(s) must do the tying on the fly when a new
key is generated.

Another problem is that the STORE method of a tied scalar doesn't know
about the hash key the value is stored under. Since each tied scalar
belongs to a fixed key, it would be possible to store the hash key in
the tying object (the one in Tie::Scalar...). But that's extra work,
and it means you can't use Tie::StdScalar as is, because it has no
provisions for extra values.

It seems the Class::Accessor::Lvalue module on CPAN uses
this approach to implement lvalue methods.

 

[Tassilo v. Parseval]

Also sprach J Krugman:

That's good to know. I was under the mistaken impression that only
"simple" scalar variables (i.e. like $foo, as opposed to $foo{bar})
could be tied to scalars.

Not at all. If that was the case, you couldn't store a tied scalar in
an array or hash thusly:

tie my $scalar => 'Class';
$hash{ key } = $scalar;

You lost me there, Anno. I don't see how, in general, an lvalue
method could do any on-the-fly tying, since all but the last
statement of the method are ignored when it is used as an lvalue.
(Yes, it could do it if it is being used as a non-lvalue method,
but that imposes a strange API).

It can be done quite easily:

sub var : lvalue {
my $self = shift;
tie $self->{_var_} => 'Class';
$self->{_var_};
}

Also, it's not at all true that only the last statement in a
lvalue-function is executed. It's just so that the last statement has to
be the (non-temporary) scalar you want to assign to.

(A bit over my head, but that's OK: I know have a lot to learn.)

I've read this paragraph a few times now. I am not quite sure either
what Anno is referring to.

I'm having a hard time picturing what you describe here. It seems
to imply that $obj somehow knows when it's being dereferenced (and
responds accordingly by "returning" a tied hash???), which I find
hard to understand. Is this something you do in any published code
that I could study?

You can overload the hash-dereference operator for a class:

package Class;

use overload '%{}' => sub {
my $self = shift;
$self->[0];
};

sub new {
my ($class) = @_;
bless [
{
key1 => 'value1',
key2 => 'value2',
},
] => $class;
}

package main;

use Data:umper;
my $obj = Class->new;
print Dumper $obj;
%$obj = (key3 => 'value3');
print Dumper $obj;

__END__
$VAR1 = bless( [
{
'key2' => 'value2',
'key1' => 'value1'
}
], 'Class' );
$VAR1 = bless( [
{
'key3' => 'value3'
}
], 'Class' );

By returning a reference to a tied-hash in the overload-handler of '%{}'
one could then control what is stored in $self->[0].

Tassilo

 

[Tassilo v. Parseval]

Also sprach (e-mail address removed):

Found in the perl source code:

http://search.cpan.org/src/NWCLARK/perl-5.8.7/pp_hot.c

It appears values on the Perl stack are being checked for
their SvFLAGS to determine whether the subroutine is returning
a "temporary".

The temporary here is a red-herring, I think. Some time ago I was making
a patch that would allow perl to assign plain GLOBs to tied scalars
which was eventually applied. I remember I had a hard time understanding
the mechanisms behind tying and so I put them down for later reference
at <http://use.perl.org/~ethan/journal/16385>.

The problem with tying and lvalue functions is that both require to
return a special kind of Perl scalar, namely an SvPVLV. This happens to
be a temporary because it is no longer needed after the actual
assignment happens.

However, if you try to assign to a tied-scalar via an lvalue-function,
you're in big trouble because now you have two PVLV-scalars that somehow
have to be crammed into one.

I could be wrong here, though, because perl might be smart enough to
notice that only one of these two lvalue-scalars is needed (namely the
one belonging to the tied scalar). In this case the check of temporarity
you mentioned above could be extended to also check for tiedness of the
variable in which case a temporary might be ok.

There'd only need to be some provisions that this temporary is not
destroyed too early (that is: not before the deferred assignment
happens).

Tassilo

 

[Joe Smith]

I often had trouble getting back the same data I put into the
variable. I was told by users here things like

"Oh don't use "SBMD_File, it's broken!"

So I guess that's the most intimidating part- the apparent fact that
some techniques are "broken". Read on here to find the "unbroken" ones.
If you're using a small dataset, they all will probably work. As you
get into larger ones, that may no longer be true. Beware. Choose one
that seems to work for most datasets. "NDBM_File" or "GDBM_File" are
probably OK choices.

No, NDBM_File is not OK. To quote from 'perldoc AnyDBM_File':

Here's a partial table of features the different packages offer:

odbm ndbm sdbm gdbm bsd-db
---- ---- ---- ---- ------
Linkage comes w/ perl yes yes yes yes yes
Src comes w/ perl no no yes no no
Comes w/ many unix os yes yes no no no
Builds ok on !unix ? ? yes yes ?
Code Size ? ? small big big
Database Size ? ? small big? ok
Speed ? ? slow ok fast
FTPable no no yes yes yes
Easy to build N/A N/A yes yes ok
Size limits 1k 4k 1k none none
Byte-order independent no no no no yes
Licensing restrictions ? ? no yes no

I got burned by the 4K limit in ndbm. What I understand is that all
keys that hash to the same value are stored in the same hash bucket.
If one bucket fills up before it is time to split the buckets, you're
screwed. The number of entries you can put into an ndbm hash before
it crashes is nondeterministic: it is significantly affected by the
order in which items are added. It's OK to use ndbm for a few thousand
entries with short keys (like "username" or "192.168.1.2"), but not
for the big stuff.
-Joe

 

[Brian McCauley]

I, too would have expected the construction to work. I'm not sure what's
wrong, but then lvalue subs are still (permanently?) experimental, so
irregularities must be expected.

For a workaround, instead of tying the whole hash, you can tie each
hash value to a scalar. That works without the error. Like a good
workaround, it has serious disadvantages.

Another work-round is not to have any persitant ties and to have the
accessor create a tied scalar on the fly. This approach can put the
validation inside the accessor method which I think aids redability.

use Tie::OneOff;

sub var : lvalue {
my $self = shift;
Tie::OneOff->lvalue({
FETCH => sub { $self->{_var_} },
STORE => sub {
my $newval = shift;
# validate $newval
$self->{_var_} = $newval;
},
});
}

 

[Anno Siegel]

In <[email protected]>




My thought was to give users the option setting through the lvalue
*or* through the more conventional

$self->var(1);

...but maybe this is not such a hot idea...

Oh, okay. I was wondering why it's there because the purpose of
the exercise was to replace that kind of assignment.

That's good to know. I was under the mistaken impression that only
"simple" scalar variables (i.e. like $foo, as opposed to $foo{bar})
could be tied to scalars.


You lost me there, Anno. I don't see how, in general, an lvalue
method could do any on-the-fly tying, since all but the last
statement of the method are ignored when it is used as an lvalue.
(Yes, it could do it if it is being used as a non-lvalue method,
but that imposes a strange API).

The statements of an lvalue sub aren't ignored, it's just that the
return value *must* be specified as the last statement of the code.
So you could just write

sub var : lvalue {
my $self = shift;
tie $self->{ _var_}, 'Tie::StdScalar' unless tied $self->{ _var_};
$self->[ _var_];
}

(A bit over my head, but that's OK: I know have a lot to learn.)

What I mean is this: Your original STORE method (for a tied hash)

sub STORE {
my ($self, $key, $value) = @_;
warn "storing value $value under key $key\n";
$self->{$key} = $value;
}

makes use of (prints out) the hash key the value is stored under.
The STORE method of a scalar doesn't have the $key parameter (what
for?), so you won't have it when you do the extra stuff the print()
stands for.

If you really need the hash key at that time, you could use an
extra field in the tie object to store the key at creation time:

tie $self->{ _var_}, 'Tie::NonStdScalar', '_var_';

would tie the variable as usual, but also squirrel away the value
"_var_" somewhere in the object. An extra method in the
Tie::NonStdScalar class, say ->key would retrieve it. Then the
STORE method could go

sub STORE {
my ($self, $value) = @_;
my $key = $self->key;
warn "storing value $value under key $key\n";
$self->{$key} = $value;
}

and function as before.

I suppose your motivation is that you like the idea of lvalue methods,

Exactly! Very clairvoyant of you.

No clairvoyance involved. I know the place, I've been there. Many
people have, look at all the footprints and litter

I'm having a hard time picturing what you describe here. It seems
to imply that $obj somehow knows when it's being dereferenced (and
responds accordingly by "returning" a tied hash???), which I find
hard to understand.

You have described it correctly and succinctly.

Is this something you do in any published code
that I could study?

No. In the particular case, the support code went to to overgrow
the actual content of the class, and I threw it all out again.

However, there is a module on CPAN that does exactly this combination
of dereference overloading and returning tied values. Unfortunately I've
lost the reference. The module author (a well known name I have also
forgotten) has published an article on the module in either TPJ or TPR
(I forget) under a title I don't remember.

The module itself would be of little use for study, it is expressly
*not* written with readability in mind, but the article would. Maybe
someone with a better memory can help out.

Anno

 

[Anno Siegel]

Another work-round is not to have any persitant ties and to have the
accessor create a tied scalar on the fly. This approach can put the
validation inside the accessor method which I think aids redability.

use Tie::OneOff;

sub var : lvalue {
my $self = shift;
Tie::OneOff->lvalue({
FETCH => sub { $self->{_var_} },
STORE => sub {
my $newval = shift;
# validate $newval
$self->{_var_} = $newval;
},
});
}

Ah, you're handing in the FETCH and STORE methods as run time parameters
to tie(). They could even be closured. That's crazy, and I like it.

Anno

 

[Brian McCauley]

Ah, you're handing in the FETCH and STORE methods as run time parameters
to tie(). They could even be closured. That's crazy, and I like it.

Thankyou. It's nice to be appreciated. I have no idea if anyone else
(except one person who left a CPAN review) has ever used my Tie::OneOff.

 

[Brian McCauley]

I suppose your motivation is that you like the idea of lvalue methods,
but need more control over what gets stored by careless users in your
sensitive objects. A similar problem exists with objects that expose
(parts of) their interior though overloading a dereference operator,
say %{}. Like lvalues, it gives you pretty syntax, but leaves your
objects wide open. I have used tied hashes to correct this (that is,
returned a reference to a tied hash in response to %{ $obj}, and haven't
encountered the particular difficulty you're seeing. That may be
an alternative approach.

Here be dragons.

There are bugs in Perl. In particular if the reference returned by the
%{} overload is the only reference to the tied hash some very nasty
things happen very soon. Even if it's not, nasty things seem to happen
after a while.

I wrote whole load of in-house classes that used this API technique and
then had to labourously go through all our programs to use a direct
method-call API rather than the overload-tie API.

 

[Anno Siegel]

Thankyou. It's nice to be appreciated. I have no idea if anyone else
(except one person who left a CPAN review) has ever used my Tie::OneOff.

Since only a small percentage of users write reviews, it is likely that
there were many more than that one.

The better the module, the less likely an author is to hear about it.

Otherwise, yes, it is hard to make a module findable for potential
users. I'm not sure how CPAN search builds its indexes (or whatever
it does), nor even if the two(?) major search sites give the same results
in all cases. So it's hard to plan for that.

Also, the (set of) situation(s) that Tie::OneOff would be of advantage in
is hard to describe in the first place. What are good keywords for that,
even if one knew where to place them?

Anno

 

[Anno Siegel]

Here be dragons.

There are bugs in Perl. In particular if the reference returned by the
%{} overload is the only reference to the tied hash some very nasty
things happen very soon. Even if it's not, nasty things seem to happen
after a while.

Even without that, de-reference overloading is a major pain for its
possible heirs. Overloading propagates up the ISA-chain just like
methods do, so if you are, say, a hash class, and you decide to inherit
from a class that overloads %{}, you'll start seeing "Not a XYZ
reference at ..." errors, or even deep recursions and segfaults.
You must "counter-overload" '%{}' => sub { shift } to be able to
access your own objects.

So you can't *be* a hash, and inherit from a class that overloads
hash dereferencing, and similar for arrays, scalars and whathaveyou.
Not all that surprising, really, but I only noticed this recently.

It's like real life and soap operas. People inherit things, along
with the good stuff, whose consequences are less than favorable.

I wrote whole load of in-house classes that used this API technique and
then had to labourously go through all our programs to use a direct
method-call API rather than the overload-tie API.

Oh yeah!

Anno

 

[J Krugman]

Anno Siegel wrote:

Thankyou. It's nice to be appreciated. I have no idea if anyone else
(except one person who left a CPAN review) has ever used my Tie::OneOff.

I've seen references to it, though at first I thought it was only
meant as a joke around the phrase "tie one on" (i.e. "get drunk");
only when I saw you mention of it did I realize that the original
reference was serious. The name may be too clever for its own
good.

jill

 

[J Krugman]

FWIW, I found a bug in bugs.perl.org that seems related to the
error message in the subject line:

http://rt.perl.org/rt3/Ticket/Display.html?id=30582

In particular, one of the follow ups shows a one-liner that elicits
the bug:

$ perl -MTie::Hash -we'tie %x,"Tie::StdHash"; sub foo:lvalue {$x{foo}} foo=1'
Can't return a temporary from lvalue subroutine at -e line 1.

As far as I can tell, there doesn't seem to be much momentum to
get this bug fixed (but I know little of the process, so I could
be way off-base on this).


jill

 

[Tassilo v. Parseval]

Also sprach J Krugman:

FWIW, I found a bug in bugs.perl.org that seems related to the
error message in the subject line:

http://rt.perl.org/rt3/Ticket/Display.html?id=30582

In particular, one of the follow ups shows a one-liner that elicits
the bug:

$ perl -MTie::Hash -we'tie %x,"Tie::StdHash"; sub foo:lvalue {$x{foo}} foo=1'
Can't return a temporary from lvalue subroutine at -e line 1.

As far as I can tell, there doesn't seem to be much momentum to
get this bug fixed (but I know little of the process, so I could
be way off-base on this).

Your post here added enough momentum.

The problem was indeed as I described in
<[email protected]>.
I submitted a patch for it which got applied roughly eight minutes ago.
Applied means it's now in the development branch 5.9.x. But I assume it
will also be in 5.8.8 whenever it's due at which point you can start to
make use of it to your heart's desire.

Tassilo

 

[Anno Siegel]

FWIW, I found a bug in bugs.perl.org that seems related to the
error message in the subject line:

http://rt.perl.org/rt3/Ticket/Display.html?id=30582

In particular, one of the follow ups shows a one-liner that elicits
the bug:

$ perl -MTie::Hash -we'tie %x,"Tie::StdHash"; sub foo:lvalue {$x{foo}} foo=1'
Can't return a temporary from lvalue subroutine at -e line 1.

Well, that's exactly your problem, isn't it?

As far as I can tell, there doesn't seem to be much momentum to
get this bug fixed (but I know little of the process, so I could
be way off-base on this).

I don't know the process either, but

http://rt.perl.org/rt3/NoAuth/perl5/Overview.html

has some statistics and other info about bug resolution.

I would imagine that many bugs are resolved not by individual solutions
but by general cleanup actions (sorry, refactoring), perhaps triggered
by the hunt for a different, superficially unrelated bug.

It may be that the frequency of the event "lvalue sub meets tied hash"
is underestimated at first view. Lvalue subs lack control over what
gets stored, and tying offers that control, so "lvalue sub searches
control, finds tied hash" may describe the event better. But that is
not immediately obvious.

Anno