Integer/Float oddity

[horati0]

not really an oddity, more a lack of understanding on my part regarding
ruby's internal type conversion...

(100*(1.15-1))
=> 15.0

(100*(1.15-1)).to_int
=> 14

(100*(1.15-1)).to_int.round
=> 15

15.0.to_int
=> 15

what am i missing here? i never would have thought to use .round to
get the 'correct' result unless someone told me. by correct i mean if
i was doing this with a paper and pencil, i would expect to get 15, not
14.

any insight would be appreciated, especially a way (if possible) to
override this behaviour so i dont run into this in the future.

thanks,
horati0

 

[Patrick Hurley]

not really an oddity, more a lack of understanding on my part regarding
ruby's internal type conversion...

(100*(1.15-1))
=> 15.0

(100*(1.15-1)).to_int
=> 14

(100*(1.15-1)).to_int.round
=> 15

15.0.to_int
=> 15

what am i missing here? i never would have thought to use .round to
get the 'correct' result unless someone told me. by correct i mean if
i was doing this with a paper and pencil, i would expect to get 15, not
14.

any insight would be appreciated, especially a way (if possible) to
override this behaviour so i dont run into this in the future.

thanks,
horati0

Two quick notes - I think you meant

(100*(1.15-1)).round.to_int
=> 15

See if this clears anything up:
printf("%.15f", (100*(1.15-1)))
14.999999999999991=> nil

The result is not really 15, but just really really close, due to the
imprecision of floats - always round moving to ints - to_i is a
"floor" function. Similarly be very careful when comparing floats.

Patrick

 

[Robert Klemme]

not really an oddity, more a lack of understanding on my part regarding
ruby's internal type conversion...

(100*(1.15-1))
=> 15.0

(100*(1.15-1)).to_int
=> 14

(100*(1.15-1)).to_int.round
=> 15

15.0.to_int
=> 15

what am i missing here? i never would have thought to use .round to
get the 'correct' result unless someone told me. by correct i mean if
i was doing this with a paper and pencil, i would expect to get 15, not
14.

any insight would be appreciated, especially a way (if possible) to
override this behaviour so i dont run into this in the future.

thanks,
horati0

This is a typical limitation of numeric calculations. The reason for what
you see is that 1.15-1 is smaller than 0.15:
=> -8.32667268468867e-17

consequently
=> -8.88178419700125e-15

this is negative => (100 * (1.15-1)) is smaller than 15 and thus it's
correctly converted to 14 by to_int.

Kind regards

robert

 

[Glenn Parker]

not really an oddity, more a lack of understanding on my part regarding
ruby's internal type conversion...

(100*(1.15-1))
=> 15.0

(100*(1.15-1)).to_int
=> 14

(100*(1.15-1)).to_int.round
=> 15

FWIW, I get 14. Perhaps you meant (100*(1.15-1)).round.to_int ?

Float#to_int simply truncates, while Float#round goes to the nearest
integer.

Try this: "%.20f" % (100*(1.15-1))

 

[michael.b.masi]

FWIW, I get 14. Perhaps you meant (100*(1.15-1)).round.to_int ?

oops, yep thats what i meant. thanks for all the replies to my query;
i figured it had something to do with the internal representation of fp
numbers.

this whole thing started as a conversation i was having with my dad
about the relative merits of various programming languages. he is a
die-hard xbasic user, and pointed me to the following post on an xb
list:

---------------------------------------------------------------------

[snip]

Message: 7
Date: Wed, 30 Mar 2005 12:59:42 -0000
From: "Bruno Schaefer" <bup.schaefer@...>
Subject: XBasic is one of the best!


I'm back now to work with XBasic, because there are strange things
with other languages.

If you try to calculate the following simple expression

INT(100*(1.15-1))

XBasic produces the correct result: 15.

This is a calculation, which is solved by children easy.

But try this with other languages
(e.g. newLISP, Visual Basic, Liberty Basic, IBasic, Yabasic, Python)
you will get the result

14 !!!!?

What a nonsense ! Does somebody understand this ?

This is one the reason for me to use XBasic!

---------------------------------------------------------------------
i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

horati0

 

[Glenn Parker]

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

Probably nothing about the internal representation is different. The
difference would have to be in xbasic's "INT()" operator, which
apparently performs rounding.

Dealing with floating point numbers can often produce counter-intuitive
results, mostly because simple analogies based on fixed-point math fail.
For example, the concept of "equality" in floating point math often
clashes with common sense. You might assume that the following
expression is true:

(100*(1.15-1)) - 15 == 0

The computer would disagree. What you really want to test is:

abs( (100*(1.15-1)) - 15 ) < TOLERANCE

Where the choice of TOLERANCE depends very much on the application.

 

[Steven Jenkins]

[snip]
i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

Nothing. It's just that Xbasic (apparently) defines INT() to mean
"closest integer" and some other languages define it to mean "largest
integer not exceeding". Both give results consistent with their
definitions. Either one gives you what you need to round *or* truncate.

round(x) is equivalent to truncate(x + .5)

Steve

 

[Mark Hubbart]

i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

horati0

While the behavior of xbasic in this circumstance would seem to be
"correct", and is probably useful in simple calculations, it would
make it unsuitable for serious calculations. The "correct" answer is
actually not. This is due to the way floating point operations are
calculated by computers.

Without going into too much depth, lets just say that floats are
stored as fractions in the computer's memory. Specifically, as
fractions with denominators that are powers of 2; (x/2^n). Just as
some fractions don't translate precisely to decimal numbers (1/2
becomes 0.5, but 1/7 becomes 0.142857142857142...), many decimal
numbers don't translate precisely to binary numbers (0.5 becomes
0.1b2, but 0.3 becomes 0.00100110011b2...). This causes rounding
errors.

To avoid this, use rational numbers, which are precise fractions:
require 'mathn'
==> true
num = 115/100
==>23/20
num -= 1
==>3/20
num * 100
==>15

cheers,
Mark

 

[Mark Hubbart]

[snip]
i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

Nothing. It's just that Xbasic (apparently) defines INT() to mean
"closest integer" and some other languages define it to mean "largest
integer not exceeding". Both give results consistent with their
definitions. Either one gives you what you need to round *or* truncate.

round(x) is equivalent to truncate(x + .5)

Still, one is considered the standard (which can be translated as
"correct"), and the other isn't. The standardized behavior is to
truncate, if you count on that and it rounds, your computations are
not going to come out the same. IIRC, the IEEE standard for floating
point numbers specifies how they should be converted to integers.

cheers,
Mark

 

[Steven Jenkins]

Still, one is considered the standard (which can be translated as
"correct"), and the other isn't. The standardized behavior is to
truncate, if you count on that and it rounds, your computations are
not going to come out the same. IIRC, the IEEE standard for floating
point numbers specifies how they should be converted to integers.

I don't think it does. IEEE 754 specifies how rounding is applied in
floating-point calculations, but does not define an API.

No standard requires *every* language to have an operator called INT()
that truncates. (C doesn't, for example.) Absent that, Xbasic can define
INT() however it wants. You might wish they'd chosen differently, but
that doesn't make it wrong.

Also note that ISO C defines various math library functions (e.g.,
floor()), but does not mandate IEEE 754 compliance.

Steve

 

[Lutz]

FWIW, I get 14. Perhaps you meant (100*(1.15-1)).round.to_int ?

oops, yep thats what i meant. thanks for all the replies to my query;
i figured it had something to do with the internal representation of fp
numbers.

this whole thing started as a conversation i was having with my dad
about the relative merits of various programming languages. he is a
die-hard xbasic user, and pointed me to the following post on an xb
list:

---------------------------------------------------------------------

[snip]

Message: 7
Date: Wed, 30 Mar 2005 12:59:42 -0000
From: "Bruno Schaefer" <bup.schaefer@...>
Subject: XBasic is one of the best!


I'm back now to work with XBasic, because there are strange things
with other languages.

If you try to calculate the following simple expression

INT(100*(1.15-1))

XBasic produces the correct result: 15.

This is a calculation, which is solved by children easy.

But try this with other languages
(e.g. newLISP, Visual Basic, Liberty Basic, IBasic, Yabasic, Python)
you will get the result

14 !!!!?

What a nonsense ! Does somebody understand this ?

This is one the reason for me to use XBasic!

---------------------------------------------------------------------
i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

horati0

 

[Lutz]

Actually newLISP produces the correct answer:

(mul 100 (sub 1.15 1)) => 15

You probably used * and - for integer arithmetik only, instead of mul
and sub for floats.

Lutz

www.newlisp.org

when contacting me via email put the word newlisp in the subject line.