Rounding issue

[Willem]

(e-mail address removed) wrote:
) I don't understand your big concern. Anytime you see numbers, decimal
) points, or anything printed in a view from a computer is a rounded
) representation of what the internal variable contains.

No it isn't. It can also be an exact representation. For example,
if the numbers are integers or some kind of decimal format.

) Its just a view, its a one way snapshot of the contents of the variable.
) That doesen't affect the internal calculations. Indeed for percentage
) calculations, there is absolutely nothing wrong with floating point
) at all.

Yes there is. Percentage calculations are fixed point, and to get exact
results, you need to be doing them in an exact representation such as
integers or rationals.

) When you have to display (have a view) where you think its necessary
) to show a correct image of the results of percentages as fractions of
) dollars (cents) then use a rounding translation that everybody and thier
) uncle uses.

If you use floating point internally, then the display result will not
always be exact.

) Why is it so hard?

Because people think too easy of it. Such as you.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

 

[sln]

(e-mail address removed) wrote:
) I don't understand your big concern. Anytime you see numbers, decimal
) points, or anything printed in a view from a computer is a rounded
) representation of what the internal variable contains.

No it isn't. It can also be an exact representation. For example,
if the numbers are integers or some kind of decimal format.

Makes sense. Its either exact or rounded for display only. Either way,
the internal representation produces the same result as exact, to the
viewer.

) Its just a view, its a one way snapshot of the contents of the variable.
) That doesen't affect the internal calculations. Indeed for percentage
) calculations, there is absolutely nothing wrong with floating point
) at all.

Yes there is. Percentage calculations are fixed point, and to get exact
results, you need to be doing them in an exact representation such as
integers or rationals.

Wrong. Percentage calculations are linear, the percentage being the
slope. The result is NOT an integer in a calculator nor computer.

) When you have to display (have a view) where you think its necessary
) to show a correct image of the results of percentages as fractions of
) dollars (cents) then use a rounding translation that everybody and thier
) uncle uses.

If you use floating point internally, then the display result will not
always be exact.

Wrong. I gave you the formulae to display EXACT representation given the
column you want to round to.

) Why is it so hard?

Because people think too easy of it. Such as you.


SaSW, Willem

Its hard to believe that say the OMB has any problems at all using floating
point numbers where formulae and fractions are everything.

Its also amazing that any computational accuracy takes place at all.

I can categorically tell you that .149999999999999000000000 equals .15
The difference can't be measured, tasted, felt or seen. It only exists
in an 80 bit register so far to the right, its meaningless and won't
affect any other calculation at all !!

my $f3 = .149999999999999000000000;
print getFmtRound($f3,8),"\n";
sub getFmtRound
{
my ($val,$width) = @_;
my $fmt = "%.$width"."f";
return sprintf $fmt,(($val * 10**($width+1))+.5)/(10**($width+1));
}

Output:

0.15000000

-sln

 

[Jürgen Exner]

I want to do financial calculations (nothing special, just simple things
like adding items of an invoice, adding taxes, removing disagio from
creditcard transactions etc.) and I want to be sure of correct results.

Then I _strongly_ suggest to brush up on computer numerics. Writing a
financial application that is sound is anything but trivial. So far you
discovered merely the tip of the iceberg of potential issues.

So if a total with tax is $0.015, then I need it to be rounded to $0.02
and not $0.01.

Just one other tiny glimpse into the peculiarities of financial
applications: If you have 5 of those 0.015$ amounts, should their total
be 0.75 or should their total be 1.00?
And there are numerous other pitfalls....

Therefor, numbers need to be represented as they are and
not the typical internal representation.

With "as they are" I suppose you mean in the decimal system.

No, Perl does not provide BCD natively (Binary Coded Decimals; someone
else already pointed that out). The reason is simple: they not only take
up more space (probably a minor issue on modern hardware) but more
important most hardware does not support BCD arithmetic, therefore all
calculations would have to be handcoded in software, making them very
slow.

I don't want to use cents as
integers. Even cents will get odd numbers after the decimal point if you
have to add e.g. 19% value added taxes. And even worser: The whole code
would get bloated and the original intention of the code would be harder
to understand if the whole source is full of "* 100" and "/ 100". In

I think you still don't recognize the basic issue: what you are asking
for cannot be done. Not because of a limitation in the binary system,
but because of a fundamantal mathematical limitation: if you have a tax
of 1/3 of one dollar, then how do you suggest to represent that value in
your favoured decimal system. You can not as we all know from 3rd grade
math. Therefore you have to deal with those rounding imperfections, no
matter which system (binary, decimal, ...) you are using. Just saying "I
want to use decimal instead of binary" doesn't absolve you from that
requirement.

How to deal with them is being tought in Computer Numerics, and that's
why I suggest several times already to brush up on that.

databases it is easy to define a decimal like (10,2): 10 digits before
the point, 2 after it.

And you will still run into the same fundamental limitations.

It would be great to have this directly in Perl
so that sprintf and other functions can still work with such values and
variables, not only specific modules like Math:ecimal.

Well, you got a few pointers to maybe helpful modules. If they don't
suit you needs then you are always free to write your own BCD module.
But as I said, even BCD won't solve your rounding problem, they "bad"
numbers will just be different.

jue

 

[Martijn Lievaart]

Hello,


I want to do financial calculations (nothing special, just simple things
like adding items of an invoice, adding taxes, removing disagio from
creditcard transactions etc.) and I want to be sure of correct results.

Then you need to define first what the correct result is, which is
actually harder then it looks.

I worked on programs that defined results up to 5 digits after the
decimal point and had exact rounding rules for every step in the
calculations. Yes, these were financial programs and no, you will
probably not encounter them unless you work at an insurance company.

The solution used by 99% of programs I know is to use a suitably large
integer type to represent cents, converting to $, E, whatever on output.

M4

 

[Willem]

(e-mail address removed) wrote:
) Makes sense. Its either exact or rounded for display only. Either way,
) the internal representation produces the same result as exact, to the
) viewer.

Floating point is rounded during almost any calculation you care
to do with it. That's the whole problem.

) Wrong. Percentage calculations are linear, the percentage being the
) slope. The result is NOT an integer in a calculator nor computer.

Yes they are, as long as you keep track of factors of 10 separately.

That is: <integer> times <n%> is <integer times n> divided by 100.
You just separately (implicitly) keep track of that 'divided by 100' bit.
If you want accuracy to 0.1%, just work with promilles instead, etc.

)>If you use floating point internally, then the display result will not
)>always be exact.
)
) Wrong. I gave you the formulae to display EXACT representation given the
) column you want to round to.

The floating point number *itself* will not be exact. It will have
internal rounding errors because it has limited precision.

) Its hard to believe that say the OMB has any problems at all using floating
) point numbers where formulae and fractions are everything.

I find it very easy to believe. Floating point inherently has rounding
errors. You need a lot of careful analysis to make sure that you always
get the correct result. If that is even possible.

Using fractions, you want exact results ? Use rationals. It's that simple.

) my $f3 = .149999999999999000000000;
) <snip>
) 0.15000000

So you're saying that the OP should round twice ?
Once to 8 digits and then to 2 digits ?

Why 8 ? Why not 5, or 10 ?
How do you know that this magic 8 will always give the correct result ?

Answer: you don't. Use an exact representation.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

 

[Alan Curry]

No, Perl does not provide BCD natively (Binary Coded Decimals; someone
else already pointed that out). The reason is simple: they not only take
up more space (probably a minor issue on modern hardware) but more
important most hardware does not support BCD arithmetic, therefore all
calculations would have to be handcoded in software, making them very
slow.

Does anyone else find this argument ("We have to use the CPU's floating-point
because anything else is too slow") a bit odd? Did we not once have CPUs with
no floating point at all?[1] It was all "in software", and at about 1MHz too.
Still, somehow, things got done.

I wouldn't be surprised if arbitrary-precision GMP-based Math::BigRat on a
recent machine outperforms the limited-precision software floating-point that
was used in the old days. But still people say "Too slow. Gotta use the FPU
for speed." Really?

Will there ever be a time when our need for speed is satiated, and we can
start thinking about making it easier to get correct results, instead of
bad results quickly? Can a well-rounded, general-purpose language like perl
ever decide that computers are fast enough now, that accuracy-preserving
"bignum" should be the default mode, and FPU speed freaks should be the ones
importing a special module to meet their needs?

If not, then can we at least hope that in the future, Math::BigRat will
play well with sprintf?

[1] I even remember being surprised when I found out the 80x86 could do
integer multiplication in a single instruction. No shift-add-shift-loop
subroutine?! Then I saw "idiv" and was sure someone was joking.

 

[Jürgen Exner]

Does anyone else find this argument ("We have to use the CPU's floating-point
because anything else is too slow") a bit odd?

Please note I said "very slow", I didn't say "too slow". The first is a
comparison against hardware supported arithmetic and can be measured.
The second is an empirical evaluation that has to be made for the
individual application.

Did we not once have CPUs with
no floating point at all?[1] It was all "in software", and at about 1MHz too.
Still, somehow, things got done.

Of course, the question is more if you are patient enough to wait that
long considering today's applications. Somehow I have a feeling that
viewing a video or running a graphic intensive game on a 386 wouldn't be
much fun ;-) .

I wouldn't be surprised if arbitrary-precision GMP-based Math::BigRat on a
recent machine outperforms the limited-precision software floating-point that
was used in the old days. But still people say "Too slow. Gotta use the FPU
for speed." Really?

Well, you can always do a benchmark test.

Will there ever be a time when our need for speed is satiated, and we can
start thinking about making it easier to get correct results, instead of
bad results quickly? Can a well-rounded, general-purpose language like perl
ever decide that computers are fast enough now, that accuracy-preserving
"bignum" should be the default mode, and FPU speed freaks should be the ones
importing a special module to meet their needs?

I'm not sure if that's really the issue.

There are only very few areas where a number has to be accurate to the
last digit, finance being the most obvious.

Pretty much any other application area really doesn't care that much
about precision. Typically either the input data itself is only
approximate to begin with. It really doesn't matter if the readout on
that 5$ digital thermometer from the hardware store is 25.34 or 25.35
degree, its accuracy is lower than the display increment anyway.

Or there is no benefit for a result that is accurate to the last digit.
For example it really doesn't matter if it takes 2 hours, 4 minutes,
25.1234 seconds to reach your destination or 2 hours, 4 minutes, 25.1235
seconds, in particular as the distance as well as the speed are both
only rough approximations in the first place.

Same goes for graphics: if the color for a particular pixel is off by
one and the immediate neighbour colour is shown instead your eye would
never notice.
Same goes for parts manufacturing: those machining tolerances are orders
of magnitudes larger then any rounding error.
Same for architecture and building plans, construction, engineering,
surveying, ... you name it. The list goes on and on while I feel hard
pressed to name an area beside finance where the accuracy of the last
digit is really significant.

Remember those slide rulers and tables of logarithms? I still remember
how we learned in school how to use them properly, i.e. when and how to
round and which precision to expect and were warned sternly not to hunt
for false precision. And those rulers and tables were used to
sucessfully build rockets, airplanes, and nuclear bombs.
Today even a simple 16bit binary FPU is way(!) more accurate than those
rulers and tables. I really don't think we need more precision but a
better understanding why hunting for more precision is pointless.

BTW: Last but not least teachers would also show us how to rearrange our
calculations, such that using those tools would introduce the least
error. And that is what's missing in our OP's understanding. A computer
is a tool. You need to understand _how_ to use it and how to arrange
your calculations to not screw up your final result. Famous example:
adding a very long sequence of very small numbers to a very large
number.

As I said: revisit Computer Numerics.

jue

 

[Jens Thoms Toerring]

Hello,

I want to do financial calculations (nothing special, just simple things
like adding items of an invoice, adding taxes, removing disagio from
creditcard transactions etc.) and I want to be sure of correct results.

I fear you stepped not only into a mathematical but at the same
time legal minefield;-)

So if a total with tax is $0.015, then I need it to be rounded to $0.02
and not $0.01. Therefor, numbers need to be represented as they are and
not the typical internal representation.

As others have pointed out there's no "as they are". You are always
dealing with a certain representation of a number. (Finite) rational
numbers can be represented with a limited number of bits in a repre-
sentation that uses a nominator and denominator. But once you want
to write them in a system with a decimal point in whatever base you
choose most numbers can't be represented exactly (and that are just
the rational numbers). Which these few "lucky numbers" are depends
on the base.

I don't want to use cents as integers. Even cents will get odd
numbers after the decimal point if you have to add e.g. 19% value
added taxes.

Yes. And thus there are, as far as I know (but I am not an expert
on this at all and just have read a bit here an there, so take this
ith a big grain of salt), certain regulations that tell how you have
to do it. That's because there is no "right" and simple way to deal
with these problems, so instead certain rules of how to do it in a
at least in a consistent way where set up. They range from the ques-
tion "should I apply the tax to each item individually and then add
them up or add them up first and then calculate the tax" to "how
many digits after the decimal point have to be taken into account",
how rounding is to be done consistently (which doesn't mean mathe-
matically correct!).

The "most mathematically correct" way to go is use something like
Math::BigRat, hoping that you don't have to deal with operations
that result in irrational numbers (like taking square roots just
to mention a very simple case) - then all bets are off. If you can
reduce everything to basically addition, subtraction, multiplica-
tion and division (which also covers exponentiation with integer
exponents) you should be fine with that. Anything more complica-
ted is likely to get you to end up with irrational numbers.

And even worser: The whole code would get bloated and the original
intention of the code would be harder to understand if the whole
source is full of "* 100" and "/ 100".

I fear that most "correctly" written programs in that problem domain
("correctly" in the sense that they satisfy the local regulations)
have to look a bit "ugly", at least in parts. The best way is probably
to write a module that takes care of all the "ugly" stuff, so the main
part of the program doesn't.

In databases it is easy to define a decimal like (10,2): 10 digits before
the point, 2 after it. It would be great to have this directly in Perl

In a database you (normally) only store things. So it's easy to e.g.
store the string (or BCD) representation of "0.15" in a database in
this case. The real problems arise when you have to do something
with these numbers (and that holds also for a database when you
have e.g. stored procedures, they also have to do calculations in
some representation). And what do you think is going to be stored
in the database when you ask it to store the value 1.144999999736
(i.e. the result of some floating point computation) in the 10,2
format? It's rather likely going to be "1.14" and not the "1.15"
you might be wishing for.

so that sprintf and other functions can still work with such values and
variables, not only specific modules like Math:ecimal.

Since there are no general mathematically correct solutions for
those problems they can't be easily incorporated into a program-
ming language. As far as I know there are regulations that tell
how to do things "correctly - but this definition of what is
"correct" may be different in different countries (or may even
already differ if you are calculating taxes or interest rates -
but as I said, I am no real expert on this).

What you're asking for is a simple solution to an extremely com-
plicated problem. And a general solution, unfortunately, doesn't
(and can't) exist. As someone else pointed out, you have to first
define what the "correct" behaviour actually is. But since what
is "correct" is not well-defined the very best you can hope for
is that someone else came up with the same definition of "correct"
before and published a module that implements this "correct" way.
Otherwise you have to write one yourself.

Regards, Jens

 

[Doug Miller]

I don't want to use cents as integers.

Then you have to live with rounding errors.

 

[Peter J. Holzer]

That you will need if the numbers you use ints and end up with
are too large to be stored in a simple int.

Be careful here. Perl doesn't have an "int" type. It has scalars, which
are kind of a mixture between string, signed int, unsigned int and
double, with conversions trown in as needed. You can assume that you can
exactly represent any integral value in the range +/- 2**53.

hp

 

[Peter J. Holzer]

(e-mail address removed) wrote:
) I don't understand your big concern. Anytime you see numbers, decimal
) points, or anything printed in a view from a computer is a rounded
) representation of what the internal variable contains.

No it isn't. It can also be an exact representation. For example,
if the numbers are integers or some kind of decimal format.

The latter is misleading. The decimal format is neither necessary nor
sufficient for an exact representation.

Decimal fractions can be represented exactly in base 10

Binary fractions can be represented exactly in base 2.

So, *if* you need to represent values like 115/100 (which happens to be
common in finance, not even the UK uses shilling and pence any more),
they can be represented exactly in decimal, but not in binary.

But if you need other fractions, decimal won't help you. For example, if
you an American and have decided to store some length measurements in
feet. And now you need to store a length of 1 inch. Well that's 1/12 of
a feet, and there is no way to represent this exactly in a decimal
floating (or fixed) point number. Either you have to round it (and then
it doesn't matter much whether you round to decimal or binary) or you
have to rewrite your program to store everything in inches.

) Its just a view, its a one way snapshot of the contents of the variable.
) That doesen't affect the internal calculations. Indeed for percentage
) calculations, there is absolutely nothing wrong with floating point
) at all.

Yes there is. Percentage calculations are fixed point,

How is a percentage calculation "fixed point"? Financial calculations
are usually fixed point, but a percentage calculation per se isn't.

For example, 19 % of 7.55 is 1.4345, but if that is an amount of money,
it will be rounded to either 1.43 or 1.44.

and to get exact results, you need to be doing them in an exact
representation such as integers or rationals.

Integers won't help in general. For example, if 8.99 is 119 %, how much
are 100 %? With rationals, you can represent the result exactly
(899/119), but with an integer, fixed, or floating point representation
you can't (unless you happen to use base 119 which isn't likely).

Of course in finance you don't care about the exact result of the
computation, but about the result rounded to a specific number of digits
(usually 2, sometimes 3 or even 5) and for that a fixed point (or
integer) representation is nice.

hp

 

[Peter J. Holzer]

Please note I said "very slow", I didn't say "too slow". The first is a
comparison against hardware supported arithmetic and can be measured.

However, we are talking about perl here, not machine code. How much of
the time spent executing a perl "multiply" operator is spent
multiplying, and how much is spent manipulating the stack, checking
whether the operands are NVs or UVs or PVs, etc.?

Even if the actual multiplication is 10 or 100 times slower, how much
slower does the "multiply" operator become? And how much slower does the
whole program (which presumably does other things besides multiplying)
become?

[rearranged]

Well, you can always do a benchmark test.

Yes, you would need to do that. It's very hard to guess how much the
effect would be.

Did we not once have CPUs with no floating point at all?[1] It was all
"in software", and at about 1MHz too. Still, somehow, things got
done.

Of course, the question is more if you are patient enough to wait that
long considering today's applications. Somehow I have a feeling that
viewing a video or running a graphic intensive game on a 386 wouldn't
be much fun ;-) .

Few codecs or 3d engines are written in Perl, AFAIK.

I'm not sure if that's really the issue.

There are only very few areas where a number has to be accurate to the
last digit, finance being the most obvious.

[lots of good examples snipped]

I fully agree.

hp