computers + floating-point = a chore

[Yossef Mendelssohn]

I recall seeing a few posts about the "fun" involving floating-point
representation of numbers, but this is something I found a little
surprising:

Cassady:~ yossef$ irb --prompt simple=> 0

Odd, isn't it? What's going on here?
=> 19.76

Ah, maybe this is it.
=> 0

Nope!
Well, a bit of experimentation led me to this, which demonstrates two
threshold points.
=> 0

It appears 19.75999999999999 (that's 12 9s, or 14 decimal places
total) is just close enough to *display* rounded up, but not actually
*become* that number. Adding two more 9s makes it close enough to
actually become that number.

No question or call for help, really. I just wanted to point out
something that intrigued me.

 

[John Joyce]

floating point math is always like this.
Any language or framework that purports to have no trouble with it is
actually using objects (or structs) consisting of integers for each
of its aspects. All of the aspects are assembled.

 

[Yossef Mendelssohn]

floating point math is always like this.
Any language or framework that purports to have no trouble with it is
actually using objects (or structs) consisting of integers for each
of its aspects. All of the aspects are assembled.

Yes, I know the troubles with floating point. I think what I found
most interesting was a number that claimed to be 19.76 when it
actually wasn't, simply because it's more convenient (?) to display it
as such.

 

[Gary Wright]

Yes, I know the troubles with floating point. I think what I found
most interesting was a number that claimed to be 19.76 when it
actually wasn't, simply because it's more convenient (?) to display it
as such.

Well, the culprit is actually Float#to_s, which has it's own idea of
how to convert floating point numbers to text. In particular it
defaults
to 15 digits of precession and also works hard to strip trailing zero's:

$ irb
irb(main):001:0> a = 19.76
=> 19.76
irb(main):002:0> a.to_s
=> "19.76"
irb(main):003:0> sprintf "%#.15g", a
=> "19.7600000000000"
irb(main):004:0> sprintf "%#.32g", a
=> "19.760000000000001563194018672220"
irb(main):005:0>


You can see from this session that 15 digits of precision is not
enough to 'show' that a isn't really 19.76.

What was surprising to me is that Float#to_s doesn't just use a standard
sprintf format string:

static VALUE
flo_to_s(flt)
VALUE flt;
{
char buf[32];
double value = RFLOAT(flt)->value;
char *p, *e;

if (isinf(value))
return rb_str_new2(value < 0 ? "-Infinity" : "Infinity");
else if(isnan(value))
return rb_str_new2("NaN");

sprintf(buf, "%#.15g", value); /* ensure to print decimal point */
if (!(e = strchr(buf, 'e'))) {
e = buf + strlen(buf);
}
if (!ISDIGIT(e[-1])) { /* reformat if ended with decimal point
(ex 111111111111111.) */
sprintf(buf, "%#.14e", value);
if (!(e = strchr(buf, 'e'))) {
e = buf + strlen(buf);
}
}
p = e;
while (p[-1]=='0' && ISDIGIT(p[-2]))
p--;
memmove(p, e, strlen(e)+1);
return rb_str_new2(buf);
}


Gary Wright

 

[Yossef Mendelssohn]

Yes, I know the troubles with floating point. I think what I found
most interesting was a number that claimed to be 19.76 when it
actually wasn't, simply because it's more convenient (?) to display it
as such.

Well, the culprit is actually Float#to_s, which has it's own idea of
how to convert floating point numbers to text. In particular it
defaults
to 15 digits of precession and also works hard to strip trailing zero's:

$ irb
irb(main):001:0> a = 19.76
=> 19.76
irb(main):002:0> a.to_s
=> "19.76"
irb(main):003:0> sprintf "%#.15g", a
=> "19.7600000000000"
irb(main):004:0> sprintf "%#.32g", a
=> "19.760000000000001563194018672220"
irb(main):005:0>

You can see from this session that 15 digits of precision is not
enough to 'show' that a isn't really 19.76.

What was surprising to me is that Float#to_s doesn't just use a standard
sprintf format string:

static VALUE
flo_to_s(flt)
VALUE flt;
{
char buf[32];
double value = RFLOAT(flt)->value;
char *p, *e;

if (isinf(value))
return rb_str_new2(value < 0 ? "-Infinity" : "Infinity");
else if(isnan(value))
return rb_str_new2("NaN");

sprintf(buf, "%#.15g", value); /* ensure to print decimal point */
if (!(e = strchr(buf, 'e'))) {
e = buf + strlen(buf);
}
if (!ISDIGIT(e[-1])) { /* reformat if ended with decimal point
(ex 111111111111111.) */
sprintf(buf, "%#.14e", value);
if (!(e = strchr(buf, 'e'))) {
e = buf + strlen(buf);
}
}
p = e;
while (p[-1]=='0' && ISDIGIT(p[-2]))
p--;
memmove(p, e, strlen(e)+1);
return rb_str_new2(buf);

}

Gary Wright

Thanks for this, Gary. I wasn't expecting that Float#inspect would
call #to_s, and obviously didn't even bother to test anything out with
sprintf.

This also explains another Float niggle, which is why it will display
"Infinity" but trying to use Infinity as a constant doesn't work.

Maybe I should spend more time looking at the Ruby source.