It was supposed to log a bunch of measurements, do Chi-square stuff, and then tell them when it was time to retool.
It was taking and logging the measurements to 6 places, but then they were turning out big batches of rejects - and it wasn't ever popping up any flags to retool.
He determined that while the actual measuring device was accurate and functioning, the cheeseball computer (Probably a 286), that came with it was rounding all those super precise numbers, and then butchering up the math with even more rounding in the statistical analysis process.
He ended up selling them a clone system, and writing them a program that did something like you did to keep the extra places.