Posted on 01/23/2004 1:39:26 PM PST by Keith
The crippled Spirit rover remains in critical condition on the surface of Mars, engineers said today, the victim of ongoing electronic seizures that have caused its central computer to reboot itself more than 60 times over the past two days.
Engineers successfully coaxed the rover to beam back limited engineering data during two brief communications sessions and they were relieved to discover the spacecraft's power system was providing the necessary life support. But Spirit's state of mind was clearly - and unusually - different in both sessions, ruling out any simple explanations for what might have gone wrong.
"We have a serious problem," said project manager Pete Theisinger. "The fact that we've got a vehicle that we believe is stable for an extensive period of time will give us time to work that problem. We can command it to talk to us and even though we get perhaps limited information, we do get good information and that helps us work through the problem.
"I expect that we will get functionality back out of this rover. I think the chances that it will be perfect again, I would think, are not good. The chances that it will not work at all, I think are also low. I think we're somewhere in that broad middle and we need to understand the problem to find out exactly where we are."
Spirit went on the blink Wednesday as it was carrying out a procedure to calibrate drive motors used by its thermal emission spectrometer. Prior to that moment, everything was operating normally. But some event, possibly a hardware failure of some sort, threw the rover's electronic brain for a loop. Since then, the spacecraft has been in a state of limbo, responding in unusual fashion to anxious flight controllers.
"This morning, we sent an early beep to the spacecraft and did not get a response," Theisinger said. "As we were preparing to send a second, the spacecraft talked to us. We got very fractional frames and then moved very quickly to ask it to speak to us for 30 minutes at 120 bits per second. We got 20 minutes of transmission in that occasion, which was a single frame of engineering data repeated.
"Then we repeated that full sequence of events and we got about 15 minutes of engineering data at 120 bits per second where the frames were updated for 15 minutes and then for the second 15 minutes we had nothing but fill data."
He said Spirit "has been in a processor reset loop of some type, mostly since Wednesday, we believe, where the processor wakes up, loads the flight software, uncovers a condition that would cause it to reset. But the processor doesn't do that immediately. It waits for a period of time - at the beginning of the day it waits for 15 minutes twice and then for the rest of the day it waits for an hour - and then it resets and comes back up."
Complicating the work to track down the problem, "the indications we have on two occasions is that the thing that causes the reset is not always perceived to be the same," Theisinger said. "We are confused by that, but that's the facts as we presume them to be right now."
The reset sequence, similar to repeatedly unplugging one's personal computer and forcing it to restart, began Wednesday morning on Mars when a calibration of the spectrometer motors ended prematurely. An anomaly team has been formed to study the telemetry and to decide what readings to request from Spirit to help narrow down the range of possible failures.
"I think we should expect that we will not be restoring functionality to Spirit for a significant period of time," Theisinger said, "I think many days, perhaps a couple of weeks, even in the best of circumstances, from what we see today."
In the meantime, he said, Spirit remains in "critical" condition.
"We do not know to what extent we can restore functionality to the system because we don't know what's broke," Theisinger said. "We don't know what started this chain of events and I think, personally, that it's a sequence of things, and we don't know, therefore, the consequences of that. I think its difficult at this very preliminary stage to assume we did not have some type of hardware event that caused this to start and therefore, we don't know to what extent we can work around that hardware event and to what extent we can get the software to ignore that hardware event if that's what we eventually have to do.
"We've got a long way to go here with the patient in intensive care. But we have been able to establish that we can command it, and we have been able to establish that it can give us information and we have been able to establish that the power system is good and we're thermally OK and those are all very, very important pieces of information.
"We are a long, long way from being done here, but we do have serious problems and our ability to eventually work around them is unknown. Do not expect a big sea change in either knowledge or theory in the next several days. This is a very complex problem."
Amid the troubleshooting, Spirit's twin - the Opportunity rover - remains on track to land early Sunday morning East Coast time on Meridiani Planum, a region on the other side of Mars where deposits of minerals that form in the presence of water have been detected. Theisinger said engineers do not believe Spirit's problem poses any generic risk to Opportunity, but he said the flight control team would be much more cautious in its daily operations to minimize the chances of a similar problem.
"It is likely, depending upon what happens in the next 48 to 72 hours, that we may not continue the Opportunity impact-to-egress with the same pace and dispatch that we did on Spirit," he said. "It depends on if we can get Opportunity to a defined, sustainable state on the ground and we can continue to make progress (with) Spirit. We will likely do that and try and continue to make progress on Spirit to get it back to some level of functionality. That's a decision the project will make in consultation with management as we take the temperature of this thing over the next couple of days."
So far, the only change for planned for Opportunity's descent is a decision to deploy its braking parachute at a slightly higher altitude than Spirit's to provide more of a safety margin.
In other developments, engineers today presented a dramatic animation of Spirit's landing based on actual telemetry from the spacecraft, showing how a sudden gust of wind forced small side-pointing rockets to fire at the last second to prevent the lander from slamming down at more than 50 mph.
The telemetry, collected earlier and subjected to complex analysis, also shows how the rover bounced across the floor of Gusev Crater before finally rolling to a stop.
Michael Malin, principal investigator of a high-resolution camera aboard NASA's Mars Global Surveyor spacecraft, unveiled a dramatic photograph showing Spirit, it's parachute and its heat shield resting on the surface of Mars. The remarkable photograph even shows several of Spirit's bounce marks in the martian soil.
Sounds like they used Windoze CE for Embeddeds.
The rover itself (that is to say, the one on Mars) is unique in that it had to be fashioned, assembled and prepared for launch in a clean room with sterilization techniques from cradle-to-grave so as to assure no terran-based microbial contamination of Mars.
But yes, there is a terran copy. It is the testbed by which all software updates are sent for quality assurance.
Well, they can talk to the spacecraft and get data back. Albeit S-L-O-W-L-Y -- the mission isn't dead by any means, just in super panic mode. I've been in JPL many times when crises like these emerge. What will happen is a very TINY team of experts will be in charge (they were most likely the same group of people whose input was ignored for many of the development/testing). These people, most familiar with the project and the least listened to, will turn out to be the ones that will save the day. But at 120 bits/sec it will take quite some time to get meaningful data back.
Then around 1994 or so I went to work for a company putting control systems in railyards. A start-up subsidy of a bigger company. A few weeks before I came on board they had gotten there first major system operational for a major US railroad. It worked very well. The customer loved it. But they had almost been kicked out, under extreme prejudice, too!
They used a real-time unix-like OS. QNX. They had hired -- what we now call out-sourcing -- an "expert" to write the software, to port their prototype system into QNX. The consultant delivered the binaries, the source, all with the very development system itself he had used, that he been loaned. The company then took the binaries to the railyard, installing the system on site, a tight schedule, the yard was to go big time VIP "ribbon-cutting" the next week. The software worked, but after a couple of days it started going haywire. The system kept freezing up and rebooting somewhat randomly, after anywhere for three to fifteen minutes of operation.
For days they burnt the midnight oil trying to figure out what the problem was -- replacing system boards, IO modules, etc. The software had been working so they assumed a faulty hardware component.
Back at the home office, the chief engineer tried to rebuild the software system that had been delivered by the consultant, recompiling everything from source. Part of an attempt to add debug checkpoints -- something, anything to help identify the problem.
His rebuild failed. The customer became incensed -- they had until Monday morning to get the system working -- it was then Friday late-day. Or they would be out. And lose their big customer. And be sued for whatever a bankruptcy would leave, etc.
The chief engineer worked desperately over the weekend. His rebuild of the software had failed because of some unresolved externals. That could only mean not all of the software source had been delivered. But the consultant kept saying they had everything they needed. When they were able to reach him He was out more than in.
The chief engineer decided he see if maybe the missing software had been inadvertantly deleted. It happens. Honest mistakes happen all the time. He ran a disk sector-scanner search looking for the missing externals.
He did find some deleted sectors and painstakingly read through each of them. It was Saturady evening. He had found the missing source module -- the header made it clear. But as he read on he made a surpising and troubling discovery.
The missing and deleted source file had some intriguing comments. They talkedg about placing a "bomb" in the code.
As he looked the code over he saw he was looking at a very sophisticated randomly triggering bomb that buried itself in the QNX operating systems kernel -- that mangled some process scheduling tables that even few QNX experts know about. Amazingly the comments were unambiguous as to the malicious intent of the code. Vanity, oh vanity! Pride of waorkmanship, perverted ...
The big company headquarters was contacted. The findings explained, the dire circumstances on-site with the important and vital customer made clear. On early Sunday morning three company lawyers flew in. They met briefly with the chief engineer and the subsidy's director.
The lawyers then drew up some serious paperwork, and drove over to the consultant's house. They woke him up. They read him the riot act, and told him "Here is what you will do". His alternative? A few federal felonies, and a civil suit that would ruin him, when his jail time finished.
That afternoon he flew to the railyard and in a short while had the system working, sans his software bomb. The chief engineer stood over his shoulder the whole time his hand were near a keyboard.
The lawyers had insisted the consultant tell them why he put the "bomb" in "To make sure I get paid" he told them.
They paid him his travel expenses. And they paid on the contract he had. And left it at that.
Me? I would have pressed the case for criminal charges against him. But then I could have un-did the bomb with or without his help. But at the time -- that desperate time -- the company needed the fix made. Without the blackmail bomb removed they could not suvive -- they needed that project's success they needed to be in the good graces of that major railroad. Despite the fact that it was blackmail, they had to deal with the blakmailing consultant.
* * *
I do not think this martian rover is such a case -- that is I hope not. The management controls, are theoretically orders of magnitude better than the Monster Garage-type management that freight railyard control systems get.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.