SvUOK always fails on 64bit platform

[cyl]

I have a dll exporting a function for perl which accepts an unsigned
interger parameter. In the exported function I use SvUOK to check what
the returned values will be by supplying different values from perl
script. Here are what I get

32bit
(in perl) (SvUOK return value)
0 - 0x7fffffff false
0x80000000 - 0xffffffff true

64bit
(in perl) (SvUOK return value)
0-0xffffffff false

I'm wondering why there's such a difference on 32 and 64 bit platform.
My thought is if SvUOK returns true then the passed parameter should
be an unsigned interger but it is true only on 32 bit platform. Do I
misuse this macro? How should I correct it? Thanks.

 

[Ben Morrow]

I have a dll exporting a function for perl which accepts an unsigned
interger parameter. In the exported function I use SvUOK to check what
the returned values will be by supplying different values from perl
script. Here are what I get

32bit
(in perl) (SvUOK return value)
0 - 0x7fffffff false
0x80000000 - 0xffffffff true

0x7fffffff here is the largest value that fits into a 32bit IV.

64bit
(in perl) (SvUOK return value)
0-0xffffffff false

0xffffffff easily fits into a 64bit IV, so that's what perl uses. If you
try numbers like 1<<63 that fit into an UV but not an IV, you'll find
they start becoming SvUOK again, so:

64bit value SvUOK
0 .. (1<<63 - 1) false
(1<<63) .. (1<<64 - 1) true

exactly equivalent to the 32bit table.

Ben

 

[sisyphus]

..
..

64bit value SvUOK
0 .. (1<<63 - 1) false
(1<<63) .. (1<<64 - 1) true

There's an interesting little catch here. On many 64-bit builds of
perl you'll find that whilst SvUOK returns true for 1<<63, it returns
false for 2**63 (because, on perls that don't also have long double
support, 2**63 is an NV). If your perl is also built with long double
support, then this issue does not arise.

I personally find that behaviour quite annoying ... but I seem to be
the only one, and it has been put to me that this would *not* be
trivial to fix .... so I reckon the behaviour is here to stay

Cheers,
Rob

 

[Ben Morrow]

There's an interesting little catch here. On many 64-bit builds of
perl you'll find that whilst SvUOK returns true for 1<<63, it returns
false for 2**63 (because, on perls that don't also have long double
support, 2**63 is an NV). If your perl is also built with long double
support, then this issue does not arise.

Yes: I noticed 2**63 was a NV, which is why I used << instead . It
seems that since ** is pow(3), it counts as a floating-point function.

This all illustrates another point, which is that depending on Perl
passing a particular representation of data to XS is usually a bad
idea, *especially* with IV/UV/NV which aren't even distinguishable from
Perl-space.

I personally find that behaviour quite annoying ... but I seem to be
the only one, and it has been put to me that this would *not* be
trivial to fix .... so I reckon the behaviour is here to stay

Well, yeah: given that ** is a floating-point operation, you no longer
have integer precision in the result:

~% perl -le'print for (1<<55)-1, sprintf "%.0f", 2**55-1'
36028797018963967
36028797018963968

though I have to say I don't really understand why this does a
floating-point subtract, when there's a whole lot of careful code in
both pp_pow and pp_subtract that tries to do arithmetic in integers where
possible... Hmmm. The logic in pp_pow goes something like

if (base or power non-integral)
use pow(3)

if (power negative)
use pow(3)

if (base is a power of 2) {
calculate by hand, in NVs

This gives a NV-only result when the result doesn't fit in a NV
with integer precision.
}
else {
if (
x**power fits in an UV, where x is the smallest power of 2
greater than base
) {
calculate by hand, in UVs

This gives a IV/UV result.
}
else
use pow(3)
}
}

but I don't really understand why the 'power of 2' branch isn't moved
inside the second 'else', so small powers of 2 to small powers are
calculated in UVs where possible.

For that matter, even if you are going to calculate in NVs, the result
is always going to be accurate (since it's a power of 2) so when the
result fits in an IV/UV at all the result should be I/UOK. That seems
like an easy bug to fix, so I must be missing something...

Ben

 

[Ben Morrow]

possible... Hmmm. The logic in pp_pow goes something like

if (base or power non-integral)
use pow(3)

if (power negative)
use pow(3)

if (base is a power of 2) {
calculate by hand, in NVs

This gives a NV-only result when the result doesn't fit in a NV
with integer precision.
}
else {
if (
x**power fits in an UV, where x is the smallest power of 2
greater than base
) {
calculate by hand, in UVs

This gives a IV/UV result.
}
else
use pow(3)
}
}

but I don't really understand why the 'power of 2' branch isn't moved
inside the second 'else', so small powers of 2 to small powers are
calculated in UVs where possible.

Sorry to be responding to myself, but having checked perlbrowse I now
think this was simply an oversight. The if(power of 2) logic is older
than the inner if, and was added to give a sane result for 2**n on
machines that have slightly insane long doubles. The latter logic was
added later, to make things like 3**33 that fit in a 64bit UV give
integer results. It looks like the possibility of doing the same for
powers of 2 was overlooked. Maybe I'll report a bug after 5.10's out.

Ben

 

[sisyphus]

Quoth Ben Morrow <[email protected]>: ..
..
Maybe I'll report a bug after 5.10's out.

Good luck with it. Somewhere, within the p5p archives there should be
buried a thread called "-Duse64bitint on Cygwin (Win32)" where I
raised the issue (without receiving any sympathy).

I also made a couple of attempts to raise the issue on perlmonks ( eg
http://www.perlmonks.org/index.pl?node_id=617840 ), where again, it
was not received with any sympathy.

Ben, I lack the expertise and self confidence to pursue the matter
with any conviction, but if you can get somewhere with this, that's
all the better.

Cheers,
Rob

 

[Dr.Ruud]

Ben Morrow schreef:

sisyphus:

Yes: I noticed 2**63 was a NV, which is why I used << instead . It
seems that since ** is pow(3), it counts as a floating-point function.

This all illustrates another point, which is that depending on Perl
passing a particular representation of data to XS is usually a bad
idea, *especially* with IV/UV/NV which aren't even distinguishable
from Perl-space.


Well, yeah: given that ** is a floating-point operation, you no longer
have integer precision in the result:

~% perl -le'print for (1<<55)-1, sprintf "%.0f", 2**55-1'
36028797018963967
36028797018963968

though I have to say I don't really understand why this does a
floating-point subtract, when there's a whole lot of careful code in
both pp_pow and pp_subtract that tries to do arithmetic in integers
where possible... Hmmm. The logic in pp_pow goes something like

if (base or power non-integral)
use pow(3)

if (power negative)
use pow(3)

if (base is a power of 2) {
calculate by hand, in NVs

This gives a NV-only result when the result doesn't fit in a
NV with integer precision.
}
else {
if (
x**power fits in an UV, where x is the smallest power of 2
greater than base
) {
calculate by hand, in UVs

This gives a IV/UV result.
}
else
use pow(3)
}
}

but I don't really understand why the 'power of 2' branch isn't moved
inside the second 'else', so small powers of 2 to small powers are
calculated in UVs where possible.

For that matter, even if you are going to calculate in NVs, the result
is always going to be accurate (since it's a power of 2) so when the
result fits in an IV/UV at all the result should be I/UOK. That seems
like an easy bug to fix, so I must be missing something...

Let's ask p5p.

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
cyl

I have a dll exporting a function for perl which accepts an unsigned
interger parameter. In the exported function I use SvUOK to check what
the returned values will be by supplying different values from perl
script. Here are what I get

32bit
(in perl) (SvUOK return value)
0 - 0x7fffffff false
0x80000000 - 0xffffffff true

64bit
(in perl) (SvUOK return value)
0-0xffffffff false

I'm wondering why there's such a difference on 32 and 64 bit platform.
My thought is if SvUOK returns true then the passed parameter should
be an unsigned interger but it is true only on 32 bit platform. Do I
misuse this macro? How should I correct it? Thanks.

One other thing to mention: SvUOK() is not Perl. You are not supposed
to TOUCH any thing not Perl unless you are capable to understand
YOURSELF what it means... Kind of self-censoreship: if you have to
ask, you are not ready to USE it for quite some time.

SvUOK() is concerned with the INTERNAL REPRESENATION of a Perl value,
not with the value itself. This says NOTHING about the value, but
says something about "the history of how it was obtained". Some
operations will ALWAYS produce SvUV; some will never; some will
differentiate on the range of values (as you observe).

Hope this helps,
Ilya

 

[cyl]

0x7fffffff here is the largest value that fits into a 32bit IV.


0xffffffff easily fits into a 64bit IV, so that's what perl uses. If you

I thought 0xffffffff should be the max value of unsigned integer on
either 32 bit or 64 bit platform. Is this correct? Actually the
problem comes from the SWIG (http://www.swig.org) implementation. I
post the generated code below.

On 64bit platform, when I pass values larger than 0x7fffffff to
TestInt, the function SWIG_AsVal_unsigned_SS_long at the bottom will
be called with parameter ST(0). Since SvUOK returns false, SvIV is
used to convert the value. In this case, values latger than 0x7fffffff
become negative and an OverflowError is thrown. I'm wondering what
changes should I make to correct this problem. Will there be any
problem if I call SvUV directly without any type check?

XS(_wrap_TestInt) {
{
unsigned int arg1 ;
unsigned int result;
unsigned int val1 ;
int ecode1 = 0 ;
int argvi = 0;
dXSARGS;

if ((items < 1) || (items > 1)) {
SWIG_croak("Usage: TestInt(unsigned int);");
}
ecode1 = SWIG_AsVal_unsigned_SS_int SWIG_PERL_CALL_ARGS_2(ST(0),
&val1);
if (!SWIG_IsOK(ecode1)) {
SWIG_exception_fail(SWIG_ArgError(ecode1), "in method '"
"TestInt" "', argument " "1"" of type '" "unsigned int""'");
}
arg1 = (unsigned int)(val1);
result = (unsigned int)TestInt(arg1);
ST(argvi) = SWIG_From_unsigned_SS_int
SWIG_PERL_CALL_ARGS_1((unsigned int)(result)); argvi++ ;

XSRETURN(argvi);
fail:

SWIG_croak_null();
}
}

SWIGINTERN int
SWIG_AsVal_unsigned_SS_int SWIG_PERL_DECL_ARGS_2(SV * obj, unsigned
int *val)
{
unsigned long v;
int res = SWIG_AsVal_unsigned_SS_long SWIG_PERL_CALL_ARGS_2(obj,
&v);
if (SWIG_IsOK(res)) {
if ((v > UINT_MAX)) {
return SWIG_OverflowError;
} else {
if (val) *val = (unsigned int)(v);
}
}
return res;
}

SWIGINTERN int
SWIG_AsVal_unsigned_SS_long SWIG_PERL_DECL_ARGS_2(SV *obj, unsigned
long *val)
{
if (SvUOK(obj)) {
if (val) *val = SvUV(obj);
return SWIG_OK;
} else if (SvIOK(obj)) {
long v = SvIV(obj);
if (v >= 0) {
if (val) *val = v;
return SWIG_OK;
} else {
return SWIG_OverflowError;
}
} else {
int dispatch = 0;
const char *nptr = SvPV(obj, PL_na);
if (nptr) {
char *endptr;
unsigned long v = strtoul(nptr, &endptr,0);
if (errno == ERANGE) {
errno = 0;
return SWIG_OverflowError;
} else {
if (*endptr == '\0') {
if (val) *val = v;
return SWIG_Str2NumCast(SWIG_OK);
}
}
}
if (!dispatch) {
double d;
int res = SWIG_AddCast(SWIG_AsVal_double
SWIG_PERL_CALL_ARGS_2(obj,&d));
if (SWIG_IsOK(res) && SWIG_CanCastAsInteger(&d, 0, ULONG_MAX)) {
if (val) *val = (unsigned long)(d);
return res;
}
}
}
return SWIG_TypeError;
}

 

[Ben Morrow]

I thought 0xffffffff should be the max value of unsigned integer on
either 32 bit or 64 bit platform. Is this correct?

Yes, 0xffffffff is 2**32-1, and the largest unsigned 32bit value.
0x7fffffff is the largest *signed* 32bit value, and perl always prefers
to store things signed when it can. When a value fits into a signed
integer, it's stored as an IV and SvUOK is false.

The largest unsigned 64bit integer is 0xffffffffffffffff (that's 16 fs),
and the largest signed half that, 0x7fffffffffffffff. On a perl with
64bit IVs, any value less than this will be stored as an IV, and will
not be SvUOK.

Actually the
problem comes from the SWIG (http://www.swig.org) implementation. I
post the generated code below.

On 64bit platform,

Stop right there . Do you mean a platform with 64bit ints, or a perl
with 64bit IVs? They are not equivalent: my perl here, for instance, is
configured for i386-freebsd-64int, so I have 64bit IVs even though my
ints are only 32bit. perl will use long long for IVs if you ask it to.

when I pass values larger than 0x7fffffff to
TestInt, the function SWIG_AsVal_unsigned_SS_long at the bottom will
be called with parameter ST(0). Since SvUOK returns false,

Because the value fits into an IV.

SvIV is used to convert the value.

*And then you cast it to (signed) long*, which is shorter than IV.
(Presumably you're running this on a 32bit platform with a 64bit perl:
you didn't say, but that's the only situation that fits.) Make the
temporary an IV, or, since you're checking SvIOK anyway, just use
SvIVX(obj) directly.

Why are you doing any of this? If you want to coerce a SV to an UV,
that's what SvUV does for you, including converting from a string or
float if necessary. I would have said the whole function would be better
written

UV uv = SvUV(obj);
*val = (unsigned long) uv;

if (*val != uv)
return SWIG_OverflowError;

return SWIG_OK;

if indeed you need that much error checking. Why aren't you just using
the typemap entry for unsigned long? That's what it's there for.

In this case, values latger than
0x7fffffff become negative and an OverflowError is thrown. I'm
wondering what changes should I make to correct this problem. Will
there be any problem if I call SvUV directly without any type check?

No. SvUV is a coercion function. SvUVX is direct access to the 'UV' slot
of the SV (SVs don't really have an UV slot, but the IV slot sometimes
holds an UV instead), and using that without checking *is* dangerous.
Have you checked perlapi?

SWIGINTERN int
SWIG_AsVal_unsigned_SS_long SWIG_PERL_DECL_ARGS_2(SV *obj, unsigned
long *val)
{
if (SvUOK(obj)) {
if (val) *val = SvUV(obj);
return SWIG_OK;
} else if (SvIOK(obj)) {
long v = SvIV(obj);
if (v >= 0) {
if (val) *val = v;
return SWIG_OK;
} else {
return SWIG_OverflowError;
}
} else {

You really didn't need to post more code than this. It illustrates the
whole of your problem.

Ben

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
cyl

SWIGINTERN int
SWIG_AsVal_unsigned_SS_int SWIG_PERL_DECL_ARGS_2(SV * obj, unsigned
int *val)
{
unsigned long v;
int res = SWIG_AsVal_unsigned_SS_long SWIG_PERL_CALL_ARGS_2(obj,
&v);
if (SWIG_IsOK(res)) {
if ((v > UINT_MAX)) {
return SWIG_OverflowError;
} else {
if (val) *val = (unsigned int)(v);
}
}
return res;
}

IMO, you should disregard the direct criticism of the other reply to
your posting (the general musing there is correct, but applications to
your code look misplaced).

I think your problems stem from misunderstanding of C, not of
Perl/XS. Your code above:

int res = function_returning_long();

makes no sense in the context.

SWIGINTERN int
SWIG_AsVal_unsigned_SS_long SWIG_PERL_DECL_ARGS_2(SV *obj, unsigned
long *val)
{
if (SvUOK(obj)) {
if (val) *val = SvUV(obj);

THis is again: you put UV into an unsigned long slot. It makes sense
ONLY IF sizeof(UV) <= sizeof(long) - which will not hold on many
builds of Perl.

[Perl defines C macros with sizeof() of most useful types (and the
corresponding MAX/MIN), so you can write conditional code. Extract
SvUV to an UV variable, then compare it with UNSIGNED_LONG_MAX
(sp?) - and remember that the way to write comparison depends on
the relative sizes of types!]

Hope this helps,
Ilya

 

[Ben Morrow]

[A complimentary Cc of this posting was sent to
cyl
<46372675-8f8c-4703-9d68-e135639c5ea0@d27g2000prf.googlegroups.com>:

IMO, you should disregard the direct criticism of the other reply to
your posting (the general musing there is correct, but applications to
your code look misplaced).

If anything I've said is incorrect or inapplicable I'll gladly accept
correction. I know you know the guts of perl better than I do .

Ben

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Ben Morrow

If anything I've said is incorrect or inapplicable I'll gladly accept
correction. I know you know the guts of perl better than I do .

I was not discussing "guts of perl". I was discussing your opinion
about "which places in the supplied C code need improvement". IMO, it
was too vague and misplaced.

Hope this helps,
Ilya

 

[cyl]

I think I'll symply remove the SvUOK check and use SvUV directly. Hope
this can work. Thanks for all the valuable informations even though I
still have lots of question marks in my head.