I appreciate the tremendous insight I am reading on this post. Apparently we have quite a few Freepers with firsthand knowledge of the programming business.
“So how is it that despite such technical savvy and programming prowess, they are so woefully poor at project estimation?”
I will add this to the other comments. It is possible to come in under budget and in less time than estimated. It is also possible to come in over budget and finish in more time than estimated. However, it is impossible to come in at less than zero cost or to be finished before you start. And it is possible to take infinite time and incur infinite costs. Thus, on average, estimates tend to be less than the actual results. The more specific we can be when planning and closer to existing, real-world projects in scope, the closer the estimates will be also.
Just remember, there's a cost associated with being specific. Software development--much like carpentry--is a field where "measuring twice and cutting once" often applies. In software development however, the "measuring" is the requirements-gathering, design, and probably testing. These aren't the "sexy" parts of software development, but accuracy pays off here.
Unfortunately, these up-front activities are often areas where timelines and budgets get cut the most. Customers don't want to hear that it'll take six months of planning and user meetings before any code is written. Obviously, that depends on your development methodology, but I hope you get my point.
Of course if you accept the article’s facts that software engineer’s do the estimating. In my experience they don’t.
Engineer’s estimates are sort of like the first level of a rumor that morphs. Usually they are told an end date dictated by some executive’s(Type A) promise.