Well, I am making a script that is doing lots of different calculations. Everything works, until I see something very strange: 64 - 63.7035 = 0.29649999999999 It's completely obvious that this is wrong. The answer should be 0.2965... Why does it do such a stupid error, and how can I fix it?

Reply

After some testing, I discovered that with this code, it would work fine:



But my other code, which do the wrong calculation, is way longer. I don't know why... I've checked the variable, which should be 63.7035 and not 63.7035000000001, and it is correct.

By the way: It only happens with that number, I've tried with some other ones, but only the one I showed shows a wrong answer. I believe there's nothing to do with it...

Reply

It's because of the way computers store numbers. They use binary, so sometimes you can't get a completely accurate number. The way that binary works in decimals is similar to the way it works in whole numbers:Except for fractions and decimals, there are a lot of numbers that it is impossible to store exactly, because the denominator is not a power of 2. That's why with some calculations, you'll get the actual answer ± .000...001. It's easy to fix if you just round to several decimal places after you get the result, but unfortunately, there's no way to fix this at the source.

Reply

Ok, thanks for your answer. But it doesn't really affect me since I am rounding the answer to closest whole number anyway, using the ceil() function.

Reply