Posted on 11/25/2004 8:29:53 PM PST by Mike Fieschko
Pension and benefit payments face disruption after what is being described as the biggest computer crash in government history left as many as 80,000 civil servants staring at blank screens and reverting to writing out giro cheques by hand in the latest blow to a hi-tech Whitehall revolution.A week-long crisis in the giant Department for Work and Pensions created a backlog of unprocessed claims with up to 80% of the ministry's 100,000 desk machines disrupted or knocked out by a blunder during maintenance.
Engineers battling to fix the problem last night claimed 95% were functioning fully again as they prepared to reboot the entire network after offices closed to the public.
Alan Johnson, the work and pensions secretary, has ordered an internal inquiry into the role of Microsoft and the American contractors EDS, who run the ministry's network as part of a £2bn information technology deal.
The disruption is the latest in a line of government technology failures and follows last week's resignation of the head of the Child Support Agency, part of Mr Johnson's empire, after the disastrous introduction of an EDS system contributed to only one in eight parents receiving the correct amount.
The DWP said some new and amended benefit claims this week would be delayed but it sought to play down the impact of the technology problems, pointing out that the department's mainframe computers were not affected.
But internal DWP correspondence seen by the Guardian, backed up by interviews with staff, appears to contradict public assertions that the disruption was minimal and most of the system continued operating normally.
A "major incident report" distributed on Monday warned of "major problems", with hourly updates issued to senior managers by fax or telephone because email on the department's intranet was blocked.
The DWP established a "crisis management centre" to resolve the problems, with Microsoft troubleshooters flown in from mainland Europe to join a high-level team including EDS technicians.
A memo circulated yesterday within jobcentres said 30% of problems could be eased by today with "a full solution potentially taking another 24-48 hours", and the difficulties running into the weekend.
"At this point there is no known solution or ETA," said the memo. "We are hopeful that some interim measures that are being considered may release some users from their current deadlock."
A routine software upgrade on a small number of PCs last weekend is believed to have gone disastrously wrong when an incompatible system was downloaded on to the whole network.
The DWP said last night that progress had been swifter than expected, insisting contingency plans had worked.
"Pens came back out and in many cases hot desking was used," said a senior official. "The problem was quite random so you could potentially be sitting down at one desk and your mate next door couldn't access the system."
Trade union leaders called on ministers to drop plans to cut 40,000 jobs in the DWP and a total of 104,000 civil servants across the government following the computer crisis.
Mark Serwotka, general secretary of the Public and Commercial Services Union, said: "Just when you thought it couldn't get any worse after the experiences of the CSA we have what can only described as near-meltdown with IT across the whole of the DWP.
"Yet again we are seeing thousands of hardworking staff, many of whom face the axe, trying to deliver essential services with one hand tied behind their back.
"The department and the government are hellbent on axing thousands of civil and public servants, saying IT will enable them to do so, but yet again we are seeing IT systems come to a grinding halt and fail.
"For the government and the department to contemplate axing thousands of jobs when the IT clearly isn't delivering is not only irresponsible and foolhardy, but some would argue pure madness."
Texas-based EDS has already seen £12.1m withheld by the government from an ill-fated £450m CSA project. Mr Johnson is said to be considering scrapping the contract.
Labour has spent some £1.5bn on over-budget or scrapped computer projects, many inherited from the Conservatives, prompting National Audit Office reports.
The Swanwick air traffic control centre, due to open in 1996 at a cost of £475m, started six years late and £180m over-budget, while £300m was spent on a scheme, later scrapped, to use plastic cards to pay benefits via post offices.
EDS failed to get an Inland Revenue contract renewed after complaints about its work.
Keith Wylie, a PCS national officer, said: "I cannot remember a crash this big."
He added: "It's a massive failure and unless it's fixed quickly it's going to result in significant delays in benefits being paid out. If it was two years down the line and those 40,000 staff had been lost, there would have been no one to write out the giros."
A routine software upgrade on a small number of PCs last weekend is believed to have gone disastrously wrong when an incompatible system was downloaded on to the whole network.
It sounds as if someone downloaded and installed the wrong software.
Trade union leaders called on ministers to drop plans to cut 40,000 jobs in the DWP and a total of 104,000 civil servants across the government following the computer crisis.
This could possibly explain WHY someone screwed up the system. If the computerization worked, then 104,000 public-sector layabouts would have been out of jobs. It's understandable that the union might not want that to happen. A lot of jobs and a lot of tax money are at stake.
Another outsourced monster IT project going down in flames.
"Although I do not know why your system is not working at this time, I have been authorized to offer you a coupon for a free Slurpee!"
Uh huh.
To err is human; to REALLY fark up requires a computer.
Somebody's gonna hang for this. I wouldn't want to be a manager in the IT dept. there.
That was my first thought as well, but I wondered if they tried to install it on older machines running earlier versions of Windows.
Hmmm...As an IT professional, being THOROUGHLY familiar with what happens on Wall Street when a problem of this nature arises, I can predict the following:
1. A scapegoat will be found (one individual, preferably disposable, wouldn't put it past them to blame and Indian or Pakistani and in particular, the guy who did the upgrade).
2. It will be decided that too much has been spent on what might be a bad system to begin with, so, we'll soldier on with what we have,which will continue to crash until someone can actually find a fix. Internal porgrammers will blame the vendors, the vendors will blame the programmers, error reports will fly back and forth, and somehow, mysteriously, the problem will have corrected itself. In the business, we call this "Error cleared while testing" which is techspeak for "someone f*cked up, but we're not sure who or how, but since no one's done it again, we hope this goes away quietly."
3. The British gov't will rail and cry about why they had to go to overseas vendors for this project, negelecting to note that perhaps the native talent just doesn't exist. This will lead to the import of more foreigners in order to jump start a new tech sector in Britain, since they work cheap and can be easily kept in line by having their visas revoked.
4. When and if any investigation is ever completed, it will be so full of technical nonsense that most computer geeks won't even be able to read it without falling asleep.
5. Someone will cry that the problem was caused by a lack of funding, and will produce at least one memo obliquely hinting at this idea.
6. Whoever does find a fix will be pushed off into the shadows while his/her bosses step into the limelight to gain whatever praise there might be. Said employee will then be laid off at the earliest opportunity because he/she is just too smart for his/her manager's (personal)comfort.
7. If this were a private enterprise, the managers who initially screwed up the project would still find a way to give themselves raises and/or bonuses.
8. 50/50 that the boo-boo is blamed on some nameless, faceless consultant, who, conveniently, has already been fired.
9. The role of "legacy systems" will be brought up again, with the current owners yelling that "we warned you this would happen if you didn't get rid of this", while simultaneously pushing thier own big-budget White ELephants as a "solution".
10. It will be discovere that the ONE GUY who could have done this smoothly was either out sick or on vacation. Said guy will never be allowed to be sick or go on vacation again.
And they wonder why I quit?
EDS. Every Day Sucks.
If we ever have an armed conflict with China, we had better be expecting the same thing to occur in our government offices and every major corp in the US.
EDS - Expect Diminished Service
Most I.S. departments would site this as an excuse to increase their budget, To prevent it's happening again.
Yep. And who opened the gates? Who leaves them open? Our leaders are blind to the dangers of questionably loyal workers and government employees -- on shore and off.
Data is our lifeblood. If we were Romans, it would be like we were managing it with our Hispanic, Gallic and German conscripts.
I don't want to be a cow!
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.