It's bad, but not as bad as it sounded in the first place. Lots of fine techies are working the problem with the terran copy of Spirit to try to duplicate the glitch and then provide workarounds. (This is true of all long-range spacecraft sent by NASA. Even Galileo had a terran-bound duplicate for such purposes.)
The first step is replaying all commands sent prior to the malfunction. Then comes the process of ruling out causes one-by-one. Then comes ruling out causes-in-combination. It's a painstaking process to say the least.
"Safe mode" probably sounds like a really bad thing to most folks, but it's really not. Those who remember the Galileo mission will recall that the craft went into safe mode several times during the course of its work (and even had a total failure of its high gain antenna), yet the mission was still a success.
I still hold great hope for Spirit. And I'll still be holding my breath when Opportunity goes into the final stages for landing tomorrow night.
Pray for W and The Truth
What the NASA engineers are doing now is basic troubleshooting and debugging. Much of the time finding the problem is a lot longer than fixing it. Especially when the equipment is only telling you limited information about its condition.
Judging from what the engineers have said, this is not a software problem but a hardware failure of some kind. The "Workaround" is not as bad as they make it sound. They will isolate the malfunctioning piece of equipment and then send programming to Spirit telling it to either disconnect the broken equipment from the system or to ignore the fault signal and not communicate with that piece.
Once Spirit is reprogrammed it will be an automatic "workaround" and will not effect the speed or operation of other pieces of equipment.
where the "degradation" comes in is that the broken piece is gone forever. if it is the Mass-spectrometer -- no tests can be done with that piece of equipment, etc.
That is what NASA is worried about, depending on what failed, Spirit may come back 99% functional or 9% functional. They won't know until they know what broke, and what that effects.
No matter what though, Spirit will be operational again, the question is "How operational"
Sounds like they used Windoze CE for Embeddeds.
Then around 1994 or so I went to work for a company putting control systems in railyards. A start-up subsidy of a bigger company. A few weeks before I came on board they had gotten there first major system operational for a major US railroad. It worked very well. The customer loved it. But they had almost been kicked out, under extreme prejudice, too!
They used a real-time unix-like OS. QNX. They had hired -- what we now call out-sourcing -- an "expert" to write the software, to port their prototype system into QNX. The consultant delivered the binaries, the source, all with the very development system itself he had used, that he been loaned. The company then took the binaries to the railyard, installing the system on site, a tight schedule, the yard was to go big time VIP "ribbon-cutting" the next week. The software worked, but after a couple of days it started going haywire. The system kept freezing up and rebooting somewhat randomly, after anywhere for three to fifteen minutes of operation.
For days they burnt the midnight oil trying to figure out what the problem was -- replacing system boards, IO modules, etc. The software had been working so they assumed a faulty hardware component.
Back at the home office, the chief engineer tried to rebuild the software system that had been delivered by the consultant, recompiling everything from source. Part of an attempt to add debug checkpoints -- something, anything to help identify the problem.
His rebuild failed. The customer became incensed -- they had until Monday morning to get the system working -- it was then Friday late-day. Or they would be out. And lose their big customer. And be sued for whatever a bankruptcy would leave, etc.
The chief engineer worked desperately over the weekend. His rebuild of the software had failed because of some unresolved externals. That could only mean not all of the software source had been delivered. But the consultant kept saying they had everything they needed. When they were able to reach him He was out more than in.
The chief engineer decided he see if maybe the missing software had been inadvertantly deleted. It happens. Honest mistakes happen all the time. He ran a disk sector-scanner search looking for the missing externals.
He did find some deleted sectors and painstakingly read through each of them. It was Saturady evening. He had found the missing source module -- the header made it clear. But as he read on he made a surpising and troubling discovery.
The missing and deleted source file had some intriguing comments. They talkedg about placing a "bomb" in the code.
As he looked the code over he saw he was looking at a very sophisticated randomly triggering bomb that buried itself in the QNX operating systems kernel -- that mangled some process scheduling tables that even few QNX experts know about. Amazingly the comments were unambiguous as to the malicious intent of the code. Vanity, oh vanity! Pride of waorkmanship, perverted ...
The big company headquarters was contacted. The findings explained, the dire circumstances on-site with the important and vital customer made clear. On early Sunday morning three company lawyers flew in. They met briefly with the chief engineer and the subsidy's director.
The lawyers then drew up some serious paperwork, and drove over to the consultant's house. They woke him up. They read him the riot act, and told him "Here is what you will do". His alternative? A few federal felonies, and a civil suit that would ruin him, when his jail time finished.
That afternoon he flew to the railyard and in a short while had the system working, sans his software bomb. The chief engineer stood over his shoulder the whole time his hand were near a keyboard.
The lawyers had insisted the consultant tell them why he put the "bomb" in "To make sure I get paid" he told them.
They paid him his travel expenses. And they paid on the contract he had. And left it at that.
Me? I would have pressed the case for criminal charges against him. But then I could have un-did the bomb with or without his help. But at the time -- that desperate time -- the company needed the fix made. Without the blackmail bomb removed they could not suvive -- they needed that project's success they needed to be in the good graces of that major railroad. Despite the fact that it was blackmail, they had to deal with the blakmailing consultant.
* * *
I do not think this martian rover is such a case -- that is I hope not. The management controls, are theoretically orders of magnitude better than the Monster Garage-type management that freight railyard control systems get.