Linux rounding error???

[Kelly Mah]

Sorry if I got the wrong group.

I seem to be running into a rounding issue on a Linux AS2.1 box using
gcc 2.95. This program always returns 419 on Linux, but on a Solaris
machine, I get the expected 420 output.

------------------------------------------------
#include <math.h>
#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}

 

[Martin Ambuhl]

Sorry if I got the wrong group.

I seem to be running into a rounding issue on a Linux AS2.1 box using
gcc 2.95. This program always returns 419 on Linux, but on a Solaris
machine, I get the expected 420 output.

There is no point in including headers you don't use.

#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}

This is yet another variation on the never-ending question: "I don't
understand floating-point arithmetic and am too lazy to check the FAQ.
Why don't floating-point numbers have infinit precision?" Try the
following on your computer:

#include <stdio.h>
#include <float.h>

int main(void)
{
double x = 0.0010;
double y = 0.420;
long double xl = 0.0010L;
long double yl = 0.420L;
float xf = 0.0010f;
float yf = 0.420f;

printf("In all cases, bogus precision is printed;\n"
"xxx_DIG + 4 is used rather than xxx_DIG\n\n");
printf("y/x = %.*g/%.*g = %.*g, (int)(y/x)= %d\n", DBL_DIG + 4, y,
DBL_DIG + 4, x, DBL_DIG + 4, y / x, (int) (y / x));
printf("yl/xl = %.*Lg/%.*Lg = %.*Lg, (int)(yl/xl)= %d\n",
LDBL_DIG + 4, yl, LDBL_DIG + 4, xl, LDBL_DIG + 4, yl / xl,
(int) (yl / xl));
printf("yf/xf = %.*g/%.*g = %.*g, (int)(yf/xf)= %d\n", FLT_DIG + 4,
yf, FLT_DIG + 4, xf, FLT_DIG + 4, yf / xf, (int) (yf / xf));
return 0;
}

In all cases, bogus precision is printed;
xxx_DIG + 4 is used rather than xxx_DIG

y/x = 0.4199999999999999845/0.001000000000000000021 = 420, (int)(y/x)= 420
yl/xl = 0.4199999999999999999827/0.0009999999999999999999133 = 420,
(int)(yl/xl)= 420
yf/xf = 0.4199999869/0.001000000047 = 419.9999695, (int)(yf/xf)= 419

 

[infobahn]

Sorry if I got the wrong group.

I seem to be running into a rounding issue on a Linux AS2.1 box using
gcc 2.95. This program always returns 419 on Linux, but on a Solaris
machine, I get the expected 420 output.

------------------------------------------------
#include <math.h>
#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}

Let's assume (for the purposes of illustration) that Platform X has
16 bits of precision below the point. On Platform X, the closest we
can get to 0.001 is 0.001007080078125 or so. In binary, this is
0.0000000001000010. The closest we can get to 0.42 on Platform X is
0.4199981689453125 (in binary, that is 0.0110101110000101).

Dividing one by the other, we get 417.04545... which is considerably
less accurate than your own findings on your platforms. Now, we can
get better precision by using more bits below the point. How much
better can we get? Obviously, it depends on how many bits we use.

And there's your explanation.

 

[CBFalconer]

I seem to be running into a rounding issue on a Linux AS2.1 box
using gcc 2.95. This program always returns 419 on Linux, but on
a Solaris machine, I get the expected 420 output.

------------------------------------------------
#include <math.h>
#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}

Yes. The line "result = y/x" computes 419.99999... (the cast is
pointless, and like most casts should not be present) and the
conversion to int on assignment properly truncates. Or the line
computes 420.000000000000000001, etc. The point is floating point
arithmetic is not exact.

You can get more consistent results by controlling rounding:

result = y/x + 0.5;

 

[Gregory Toomey]

Sorry if I got the wrong group.

I seem to be running into a rounding issue on a Linux AS2.1 box using
gcc 2.95. This program always returns 419 on Linux, but on a Solaris
machine, I get the expected 420 output.

------------------------------------------------
#include <math.h>
#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}

I see division, not rounding.
Learn the difference.

gtoomey