Free Republic
Browse · Search
General/Chat
Topics · Post Article

To: Dick Vomer
now the weird part

5596.27+1157.85+559.63=7313.74....

I've readjusted the rounding and it still comes out the wrong answer.

Is the "weird part" that it summed to 7313.74 instead of 7313.75?

If so, then the problem might be that your columns are using "single precision" floating point numbers [32-bits], and if you could change the type of the values to "double precision" floating point numbers, then you might get better [i.e. less noticeable] rounding error.

[Note that there will ALWAYS be rounding error when working with floating point numbers - the only question is HOW BAD the rounding error will be.]

Beyond that, there are some known bugs with the mathematics package in Excel 2007:

Excel 2007 Multiplication Bug
Monday September 24, 2007
http://it.slashdot.org/article.pl?sid=07/09/24/2339203

6 posted on 10/23/2007 8:51:14 AM PDT by KayEyeDoubleDee (const Tag &referenceToConstTag)
[ Post Reply | Private Reply | To 1 | View Replies ]


To: KayEyeDoubleDee

Modifying the multiplication equations as below can eliminate the hidden stuff and solve the penny problem:

L14=0.01*INT(100*L12*0.29+0.5)
L15=0.01*INT(100*L12*0.06+0.5)

etc.


7 posted on 10/23/2007 9:27:55 AM PDT by chopperman
[ Post Reply | Private Reply | To 6 | View Replies ]

Free Republic
Browse · Search
General/Chat
Topics · Post Article


FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson