Free Republic
Browse · Search
News/Activism
Topics · Post Article

To: Excuse_Me
Kewl! Back in the late 80's, my guru was called into a company who had just spent 6 figures on an expensive measuring device that was giving them fits. They made bearings or something.

It was supposed to log a bunch of measurements, do Chi-square stuff, and then tell them when it was time to retool.

It was taking and logging the measurements to 6 places, but then they were turning out big batches of rejects - and it wasn't ever popping up any flags to retool.

He determined that while the actual measuring device was accurate and functioning, the cheeseball computer (Probably a 286), that came with it was rounding all those super precise numbers, and then butchering up the math with even more rounding in the statistical analysis process.

He ended up selling them a clone system, and writing them a program that did something like you did to keep the extra places.

236 posted on 12/08/2001 9:34:26 PM PST by Bill Rice
[ Post Reply | Private Reply | To 225 | View Replies ]


To: Bill Rice
Actually looking over the code, I realize that even though I was calculating Pi to 2000 places, I was truncating it to 1000 places during the square root calculation. So I modified the program and ran it again, this time truncating the output to 1000 places, but leaving all calcs at 2000. There was a minor change. the last 3 digits changed from 670 to 692. So, I do think that with that change, the answer is now accurate for actual SQRT(Pi) to 1000 places.
256 posted on 12/09/2001 12:31:57 AM PST by Excuse_Me
[ Post Reply | Private Reply | To 236 | View Replies ]

Free Republic
Browse · Search
News/Activism
Topics · Post Article


FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson