I’m not a developer by any means, but I do a lot of administrative scripting. Can’t tell you how many times I’ve had to re-work somebody else’s scripts that tested fine using a trivial test set and spun up and froze when confronted with actual production data.
Everybody test markets new products.
You’d think test marketing new software would involve trying to overload it with customers.
And you think you’d discover at that point that it wouldn’t take even ONE customer.
I think the only good test is a true “trial run” by a portion of the real end users, using real data. That’s how they test a movie. They screen it before a real audience of potential viewers. They should’ve paid 100 actual citizens to log in at the same time and try to complete the Barrycare process. Of course, load-testing with really high numbers is harder to do and has to be simulated. But, they also should’ve done the testing a year ago, not the last week before the roll-out. That part should’ve been obvious...
I design online trading interactions for a living. Our business initiatives come to life via defined projects with budgets for each project. Approximately 30-40% of our total cost per project goes to QA. Our QA guys go bananas on our code for 2-3-4 months prior to launch, using tens of thousands of test cases and scripts; they write thousands of bugs, and right before release, for months, all we’re doing is bug fixes. If we get lucky, sometimes it’s only a month/six weeks.
They tested this ACA site for TWO WEEKS.
Holy Moses.