Numerics, NaNs, IEEE 754 and C99

[Nick Maclaren]

|>
|> > Now, can you explain why 1/0 => -Inf wouldn't work as well? I.e. why
|> > are ALL of your zeroes, INCLUDING those that arise from subtractions,
|> > are known to be positive?
|>
|> I would say that the most common reason people assume 1/0 = Inf is
|> probably because they do not make use of negative numbers or they forgot
|> they exist at all.

Usually the latter :-(

Yes, if you are working with the non-negative real numbers (or even
non-negative integers), then the best result of 1/0 is +infinity.
Thus, if you are working with events, a count of zero is a true zero,
but its inverse can reasonably be said to be +infinity.

However, it is common for non-experts to assume that they are working
in the non-negative real domain, because numbers are conventionally
measured that way, but in fact that is not so. The use of elapsed
times is one such "gotcha". I have seen quite a few programs that
have assumed that elapsed times were always positive, and which have
blown up when applied to real problems, where the measurement of an
elapsed time may be negative.

Hence, the SAFE approach is to make the inverse of all zeros a NaN.


Regards,
Nick Maclaren.

 

[Grant Edwards]

|>
|> I assume the "you" in that sentence refers to the IEEE FP
|> standards group. I just try to follow the standard, but I have
|> found that the behavior required by the IEEE standard is
|> generally what works best for my applications.

Well, it could be, but actually it was a reference to the
sentence "This makes sense since such is the limit of division
by a quantity that goes to zero."

That sentence was written by the IEEE FP standards committee
explaining why they chose the behavior they did.

The point here is that +infinity is the correct answer when the zero is
known to be a positive infinitesimal, just as -infinity is when it is
known to be a negative one. NaN is the only numerically respectable
result if the sign is not known, or it might be a true zero.

OK, you're right. I still prefer that Python follow the
standard they everything else does. Part of what I use Python
for is to simulate devices that obey the IEEE 754 floating
point standards. I need Python to follow the standard.

 

[Grant Edwards]

Hence, the SAFE approach is to make the inverse of all zeros a
NaN.

OK. You're right. I'm wrong about what my Python programs
should do.

In any case, that ship sailed.

Maybe you can argue convincingly that theoretically your
approach is better than IEEE 754. You're not going to get the
standard changed. You're not going to get all of the computers
on the planet re-worked. Making Python incompatible with IEEE
754 is a bad idea.

I might be able to come up with a convincing arguement that
driving on the right-hand side of the road is better than
driving on the left, but that doesn't make a good idea to do so
when in England, Japan, or Australia.

 

[rainbow.cougar]

...
Now, I should like to improve this, but there are two problems. The
first is political, and is whether it would be acceptable in Python to
restore the semantics that were standard up until about 1980 in the
numerical programming area. I.e. one where anything that is numerically
undefined or at a singularity which can deliver more than one value is
an error state (e.g. raises an an exception or returns a NaN). This
is heresy in the C99 and Java camps, and is none too acceptable in the
IEEE 754R one.

...

As one of the few people on this group who will admit to programming
before 1980, let alone doing numerical programming then (I presume
writing code to process gigabytes of seismic data via Fast Fourier
equations meet your criteria), the only semantics that existed then
were for the program to *CRASH* hard when a singularity or undefined
state occurred. When using the various flavors of FORTRAN and vector
processors the undefined states could take a few microseconds to
manifest, but crash and burn they would, with some difficulty to
analyze the burning wreckage.

C is not a mathematically based language as FORTRAN is, so it is funny
to criticize it for being what it never was, however IEEE754 and NaN's
and the other standardizations put into place make back analyzing what
is wrong with your program and/or data far easier then what we had
before.


Curtis

 

[Nick Maclaren]

|>
|> > Hence, the SAFE approach is to make the inverse of all zeros a
|> > NaN.
|>
|> OK. You're right. I'm wrong about what my Python programs
|> should do.
|>
|> In any case, that ship sailed.

And sank, spawning a zillion lifeboats heading in different directions

|> Maybe you can argue convincingly that theoretically your
|> approach is better than IEEE 754. You're not going to get the
|> standard changed. You're not going to get all of the computers
|> on the planet re-worked. Making Python incompatible with IEEE
|> 754 is a bad idea.

No, that is wrong, on many counts.

Firstly, a FAR more common assumption is that integers wrap in twos'
complement - Python does not do that.

Secondly, it is NOT incompatible with IEEE 754, which is a pure hardware
standard. All it does is to trap the exception and take appropriate
action (as permitted by that standard).

Thirdly, virtually no hardware sticks strictly to IEEE 754, and no
language that I know of has EVER attempted to make it the strict basis
for its arithmetic model.

Fourthly, I am not proposing to change any hardware, and could even
provide a Python option to deliver mathematically incorrect results
when you want them.


Regards,
Nick Maclaren.

 

[Nick Maclaren]

|>
|> As one of the few people on this group who will admit to programming
|> before 1980, let alone doing numerical programming then (I presume
|> writing code to process gigabytes of seismic data via Fast Fourier
|> equations meet your criteria), the only semantics that existed then
|> were for the program to *CRASH* hard when a singularity or undefined
|> state occurred. When using the various flavors of FORTRAN and vector
|> processors the undefined states could take a few microseconds to
|> manifest, but crash and burn they would, with some difficulty to
|> analyze the burning wreckage.

Yes, it does, but I had used some dozens of systems, Fortrans and other
languages by then. I did not make myself entirely clear, which partly
accounts for your response, but I am afraid that you are mistaken.

What I was referring to was that the consensus was that it was a
program error to calculate a value at a singularity or otherwise when
the result was mathematically undefined.

Yes, the languages specified that to be undefined behaviour (and usually
still do), but it was clear that a good implementation was ALOLOWED to
trap and diagnose the failures. Many did and some still do - Python is
pretty good, but not in this area.

Examples of numerically near-bulletproof compilers included Egtran, XFAT
(if I recall after 35+ years), WATFIV, Delft Algol, SPITBOL, FLACC and
many others (and some in the 1980s, too). MOST of the better Fortran
run-time systems (even IBM's Mod II library) used to detect and flag
invalid arguments to library routines.

==>> Why should Python regard that, provably achievable, level of
robustness as positevely undesirable?

|> C is not a mathematically based language as FORTRAN is, so it is funny
|> to criticize it for being what it never was, however IEEE754 and NaN's
|> and the other standardizations put into place make back analyzing what
|> is wrong with your program and/or data far easier then what we had
|> before.

As someone who has done it more-or-less continually since the 1960s,
I am afraid that is not so. Yes, NaNs COULD do that, if the (fixable)
flaws in IEEE 754 were sorted out, but Java and C99 have picked up on the
flaws and used them as justification for ignoring 90% of the principles
of IEEE 754.

==>> But the flaws in those are not the point here. I am asking
whether people would positively OBJECT (as the did in C99) to
errors being detected, possibly as an overridable option.


Regards,
Nick Maclaren.

 

[Grant Edwards]

|> Making Python incompatible with IEEE |> 754 is a bad idea.

No, that is wrong, on many counts.

Firstly, a FAR more common assumption is that integers wrap in twos'
complement - Python does not do that.

It used to, and I sure wish there was still a way to get that
behavior back. Now I have to do all sorts of bitwise ands that
I didn't use to.

Secondly, it is NOT incompatible with IEEE 754, which is a
pure hardware standard. All it does is to trap the exception
and take appropriate action (as permitted by that standard).

There are two allowed (selectable?) behaviors for 1/0: trap and
Inf.

Thirdly, virtually no hardware sticks strictly to IEEE 754,
and no language that I know of has EVER attempted to make it
the strict basis for its arithmetic model.

All the HW and libraries I've used returned Inf for 1/0.

Fourthly, I am not proposing to change any hardware,

IMO, having Python's FP results disagree with common HW FP
results would be a pretty bad thing.

and could even provide a Python option to deliver
mathematically incorrect results when you want them.

As long as I can get IEEE 754 results, that's cool. I'd prefer
if Python just let the HW do it's thing when possible.

 

[Nick Maclaren]

|> >
|> > Firstly, a FAR more common assumption is that integers wrap in twos'
|> > complement - Python does not do that.
|>
|> It used to, and I sure wish there was still a way to get that
|> behavior back. Now I have to do all sorts of bitwise ands that
|> I didn't use to.

Given that it is perhaps THE most common cause of severe failure
in large scientific codes, I don't agree that it should be the default.
A 'twos complement' class would be quite reasonable.

|> > Secondly, it is NOT incompatible with IEEE 754, which is a
|> > pure hardware standard. All it does is to trap the exception
|> > and take appropriate action (as permitted by that standard).
|>
|> There are two allowed (selectable?) behaviors for 1/0: trap and
|> Inf.

Er, no. The second is REQUIRED to set an exception flag, which IEEE 754
assumes that the code will test and take appropriate action (which can
be anything, including aborting the program and replacing it by a NaN).
See http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf

|> > Thirdly, virtually no hardware sticks strictly to IEEE 754,
|> > and no language that I know of has EVER attempted to make it
|> > the strict basis for its arithmetic model.
|>
|> All the HW and libraries I've used returned Inf for 1/0.

Well, I have clearly used a much wider range than you have, then.
But it isn't just that aspect I was referring to. You will also find
that a high proportion of hardware traps into the kernel on overflow,
and it is software that returns the infinity.

|> > Fourthly, I am not proposing to change any hardware,
|>
|> IMO, having Python's FP results disagree with common HW FP
|> results would be a pretty bad thing.

No matter WHAT results are returned, they will do that! I was triggered
into this project by attempting to write NaN-safe code and finding that
the first few versions of Python I tried all did different things.
When I saw that it just mapped what C does, all became clear.

|> > and could even provide a Python option to deliver
|> > mathematically incorrect results when you want them.
|>
|> As long as I can get IEEE 754 results, that's cool. I'd prefer
|> if Python just let the HW do it's thing when possible.

Including crash and burn?

Seriously. I don't think that you realise just how many layers of
fixup there are on a typical "IEEE 754" system, and how many variations
there are even for a single platform.


Regards,
Nick Maclaren.

 

[Fredrik Lundh]

It used to

for integers ? what version was that ?

</F>

 

[Grant Edwards]

|>> Firstly, a FAR more common assumption is that integers wrap in twos'
|>> complement - Python does not do that.
|>
|> It used to, and I sure wish there was still a way to get that
|> behavior back. Now I have to do all sorts of bitwise ands that
|> I didn't use to.

Given that it is perhaps THE most common cause of severe
failure in large scientific codes, I don't agree that it
should be the default. A 'twos complement' class would be
quite reasonable.

Since Python went to non-fixed-length integers, a couple of
those classes have been written. I tried one of them and it
was rather clumsy. There was no conversion/coercion between
different widths or between fixed-width integers and "regular"
integers. That mean you had to call a constructor even if all
you wanted to do was add 2 to a value.

|> > Secondly, it is NOT incompatible with IEEE 754, which is a
|> > pure hardware standard. All it does is to trap the exception
|> > and take appropriate action (as permitted by that standard).
|>
|> There are two allowed (selectable?) behaviors for 1/0: trap and
|> Inf.

Er, no. The second is REQUIRED to set an exception flag,

But it's not required to generate a trap according to my
reading of the spec.

which IEEE 754 assumes that the code will test and take
appropriate action (which can be anything, including aborting
the program and replacing it by a NaN). See
http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf

IEEE Std 754-1985, subclause 7.2 - Division by Zero

"If the divisor is zero and the dividend is a finite nonzero
number, then the division by zero shall be signaled. The
result, when no trap occurs, shall be a correctly signed
(infinity symbol)(6.3)."

To me it looks like the spec specifically allows for a case
where "no trap occurrs" and the result is Inf.

|> As long as I can get IEEE 754 results, that's cool. I'd prefer
|> if Python just let the HW do it's thing when possible.

Including crash and burn?

No.

Seriously. I don't think that you realise just how many
layers of fixup there are on a typical "IEEE 754" system, and
how many variations there are even for a single platform.

Probably not.

 

[Grant Edwards]

for integers ? what version was that ?

Am I remebering incorrectly? Didn't the old fixed-width
integers operate modulo-wordsize? I distinctly remember having
to rewrite a bunch of checksum code when fixed-width integers
went away.

 

[Tim Peters]

[Nick Maclaren]
[Grant Edwards]
[Fredrik Lundh]

for integers ? what version was that ?

[Grant]
Am I remebering incorrectly?

Mostly but not entirely.

Didn't the old fixed-width integers operate modulo-wordsize?

Not in Python.

I distinctly remember having to rewrite a bunch of checksum code when
fixed-width integers went away.

Best guess is that you're working on a 32-bit box, and remember this
Python <= 2.2 behavior specific to the left shift operator:
0

On a 64-bit box (except Win64, which didn't exist at the time ;-)),
those returned 2**31 and 2**32 instead, while "1 << 64" wrapped to 0
(etc).

Python 2.3 starting warning about that:
__main__:1: FutureWarning: x<<y losing bits or changing sign will
return a long in Python 2.4 and up
-2147483648

and Python 2.4 started producing platform-independent results:
4294967296L

+ - * / on short ints always complained about overflow before int-long
unification, although IIRC very early Pythons missed the overflow
error in (on a 32-bit box) int(-(2L**31))/-1.

 

[Nick Maclaren]

|>
|> > which IEEE 754 assumes that the code will test and take
|> > appropriate action (which can be anything, including aborting
|> > the program and replacing it by a NaN). See
|> > http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf
|>
|> IEEE Std 754-1985, subclause 7.2 - Division by Zero
|>
|> "If the divisor is zero and the dividend is a finite nonzero
|> number, then the division by zero shall be signaled. The
|> result, when no trap occurs, shall be a correctly signed
|> (infinity symbol)(6.3)."
|>
|> To me it looks like the spec specifically allows for a case
|> where "no trap occurrs" and the result is Inf.

That is correct. And it is a mandatory case. But it does NOT say what
the software should then do with the exception.


Regards,
Nick Maclaren.

 

[Grant Edwards]

[Grant]

Am I remebering incorrectly?

Mostly but not entirely.

Didn't the old fixed-width integers operate modulo-wordsize?

Not in Python.

I distinctly remember having to rewrite a bunch of checksum code when
fixed-width integers went away.

Best guess is that you're working on a 32-bit box, and remember this
Python <= 2.2 behavior specific to the left shift operator:
0

That's probably it.

I've also spent some time on/off fighting with 32-bit constants
that have the high-order bit set. You've got to jump through
hoops when you need to pass a value like 0xC0000000 to an
extension that demands a 32-bit value.

+ - * / on short ints always complained about overflow before
int-long unification,

I was definitely mis-remembering things. I had to have been
the left shift that caused the problems.