truncate trouble

[Greg Lorriman]

This

('40.12'.to_f*100).to_i

results in this integer :

4011

This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?
Is there another floating type that avoids this problem? I;m not
seeing anything in the docs that might have warned me.

Greg

 

[Phrogz]

This

('40.12'.to_f*100).to_i

results in this integer :

4011

What I find interesting is that the #inspect result of both f1 and f2
below are the same, when they internally are clearly not.

irb(main):001:0> f1 = '40.12'.to_f
=> 40.12
irb(main):002:0> f1*100
=> 4012.0
irb(main):003:0> (f1*100).to_i
=> 4011
irb(main):004:0> (f1*100).round.to_i
=> 4012

irb(main):005:0> 4012.0.to_i
=> 4012

irb(main):006:0> f2 = '40.12000000000001'.to_f
=> 40.12
irb(main):007:0> (f2*100).to_i
=> 4012

 

[Brian Candler]

This

('40.12'.to_f*100).to_i

results in this integer :

4011

This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?

Only standard numeric rounding error. The problem is that decimal 0.1 cannot
be represented exactly in binary floating point - it's a recurring fraction,
like 1/3 in decimal - so you get only an approximation.

Is there another floating type that avoids this problem? I;m not
seeing anything in the docs that might have warned me.

irb(main):002:0> require 'bigdecimal'
=> true
irb(main):003:0> f = BigDecimal.new('40.12')
=> #<BigDecimal:b7cb7548,'0.4012E2',8(8)>
irb(main):004:0> (f*100).to_i
=> 4012

BigDecimal stores decimal numbers as decimal. It's not exactly well
documented though (I only came across it because Ruby on Rails uses it).

Another solution is to multiply all your numbers by 100 at source, i.e.
works in cents rather than dollars or whatever.

Regards,

Brian.

 

[Robert Klemme]

This

('40.12'.to_f*100).to_i

results in this integer :

4011

This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?
Is there another floating type that avoids this problem? I;m not
seeing anything in the docs that might have warned me.

No, that's a normal rounding error that comes with floats:

irb(main):001:0> f='40.12'.to_f
=> 40.12
irb(main):002:0> "%20.10f" % f
=> " 40.1200000000"
irb(main):003:0> "%40.30f" % f
=> " 40.119999999999997442046151263639"
irb(main):004:0> "%40.30f" % (f*100)
=> " 4011.999999999999545252649113535881"

If you look at the last line it becomes clear why you get 4011 as result.

Kind regards

robert