Posted on 01/23/2004 4:35:20 PM PST by Hiwayman
The crippled Spirit rover remains in critical condition on the surface of Mars, engineers said today, the victim of ongoing electronic seizures that have caused its central computer to reboot itself more than 60 times over the past two days.
Engineers successfully coaxed the rover to beam back limited engineering data during two brief communications sessions and they were relieved to discover the spacecraft's power system was providing the necessary life support. But Spirit's state of mind was clearly - and unusually - different in both sessions, ruling out any simple explanations for what might have gone wrong.
"We have a serious problem," said project manager Pete Theisinger. "The fact that we've got a vehicle that we believe is stable for an extensive period of time will give us time to work that problem. We can command it to talk to us and even though we get perhaps limited information, we do get good information and that helps us work through the problem.
"I expect that we will get functionality back out of this rover. I think the chances that it will be perfect again, I would think, are not good. The chances that it will not work at all, I think are also low. I think we're somewhere in that broad middle and we need to understand the problem to find out exactly where we are."
Spirit went on the blink Wednesday as it was carrying out a procedure to calibrate drive motors used by its thermal emission spectrometer. Prior to that moment, everything was operating normally. But some event, possibly a hardware failure of some sort, threw the rover's electronic brain for a loop. Since then, the spacecraft has been in a state of limbo, responding in unusual fashion to anxious flight controllers.
"This morning, we sent an early beep to the spacecraft and did not get a response," Theisinger said. "As we were preparing to send a second, the spacecraft talked to us. We got very fractional frames and then moved very quickly to ask it to speak to us for 30 minutes at 120 bits per second. We got 20 minutes of transmission in that occasion, which was a single frame of engineering data repeated.
"Then we repeated that full sequence of events and we got about 15 minutes of engineering data at 120 bits per second where the frames were updated for 15 minutes and then for the second 15 minutes we had nothing but fill data."
He said Spirit "has been in a processor reset loop of some type, mostly since Wednesday, we believe, where the processor wakes up, loads the flight software, uncovers a condition that would cause it to reset. But the processor doesn't do that immediately. It waits for a period of time - at the beginning of the day it waits for 15 minutes twice and then for the rest of the day it waits for an hour - and then it resets and comes back up."
Complicating the work to track down the problem, "the indications we have on two occasions is that the thing that causes the reset is not always perceived to be the same," Theisinger said. "We are confused by that, but that's the facts as we presume them to be right now."
The reset sequence, similar to repeatedly unplugging one's personal computer and forcing it to restart, began Wednesday morning on Mars when a calibration of the spectrometer motors ended prematurely. An anomaly team has been formed to study the telemetry and to decide what readings to request from Spirit to help narrow down the range of possible failures.
"I think we should expect that we will not be restoring functionality to Spirit for a significant period of time," Theisinger said, "I think many days, perhaps a couple of weeks, even in the best of circumstances, from what we see today."
In the meantime, he said, Spirit remains in "critical" condition.
"We do not know to what extent we can restore functionality to the system because we don't know what's broke," Theisinger said. "We don't know what started this chain of events and I think, personally, that it's a sequence of things, and we don't know, therefore, the consequences of that. I think its difficult at this very preliminary stage to assume we did not have some type of hardware event that caused this to start and therefore, we don't know to what extent we can work around that hardware event and to what extent we can get the software to ignore that hardware event if that's what we eventually have to do.
"We've got a long way to go here with the patient in intensive care. But we have been able to establish that we can command it, and we have been able to establish that it can give us information and we have been able to establish that the power system is good and we're thermally OK and those are all very, very important pieces of information.
"We are a long, long way from being done here, but we do have serious problems and our ability to eventually work around them is unknown. Do not expect a big sea change in either knowledge or theory in the next several days. This is a very complex problem."
Amid the troubleshooting, Spirit's twin - the Opportunity rover - remains on track to land early Sunday morning East Coast time on Meridiani Planum, a region on the other side of Mars where deposits of minerals that form in the presence of water have been detected. Theisinger said engineers do not believe Spirit's problem poses any generic risk to Opportunity, but he said the flight control team would be much more cautious in its daily operations to minimize the chances of a similar problem.
"It is likely, depending upon what happens in the next 48 to 72 hours, that we may not continue the Opportunity impact-to-egress with the same pace and dispatch that we did on Spirit," he said. "It depends on if we can get Opportunity to a defined, sustainable state on the ground and we can continue to make progress (with) Spirit. We will likely do that and try and continue to make progress on Spirit to get it back to some level of functionality. That's a decision the project will make in consultation with management as we take the temperature of this thing over the next couple of days."
So far, the only change for planned for Opportunity's descent is a decision to deploy its braking parachute at a slightly higher altitude than Spirit's to provide more of a safety margin.
In other developments, engineers today presented a dramatic animation of Spirit's landing based on actual telemetry from the spacecraft, showing how a sudden gust of wind forced small side-pointing rockets to fire at the last second to prevent the lander from slamming down at more than 50 mph.
The telemetry, collected earlier and subjected to complex analysis, also shows how the rover bounced across the floor of Gusev Crater before finally rolling to a stop.
Michael Malin, principal investigator of a high-resolution camera aboard NASA's Mars Global Surveyor spacecraft, unveiled a dramatic photograph showing Spirit, it's parachute and its heat shield resting on the surface of Mars. The remarkable photograph even shows several of Spirit's bounce marks in the martian soil.
Exactly. The cost (in lost money, time, morale, etc.) of a failed manned mission would be hugely greater than the cost of a failed unmanned mission.
Yes, there are problems that people "on the spot" could possibly fix. But there are also more problems that can arise *because* you've got life-support issues to worry about. Overall I'd think that a manned mission would have greater risk of failure than an "unattended" unmanned mission. And the cost of failure is so much higher.
That 'horrendously stagnant technology' has boosted productivity to the point where we have high growth, almost no inflation and companies don't need to hire new workers.
IMHO there is no subsititute for humans in space exploration. If you ever designed a complex piece of software you would understand the impossibility of proving its reliability.
BUMP
"
In conventional terrestrial applications, the vast majority of problems are software-related (usually 99+%).
For planetary exploration purposes, the chances of there being a hardware failure is probably boosted significantly.
Redundancy is achieved by various means. Voting was used on the shuttles. Pair-and-spare are used on some proprietary earth applications. A "Cadillac" solution would be to use multiple vendors for separate missions. In effect missions from other nationalities achieve an approximation to this ideal.
A composite of some of these approaches is represented in Arthur C. Clarke's "Rendezvous with Rama," which incidentally appears to be in the process of being made into a movie.
A full redundant system, such as sending two rovers, presupposes all failures are hardware failures due to abnormally harsh environmental conditions impairing the normal operation of the hardware, and that all software problems can be overcome by (a) extensive pre-flight testing and (b) remote field upgrades and reboots.
I am interested in more details about the communications problems. 120 bits/sec is very slow. I wonder if they sidelined their normal (sliding-frame?) more sophisticated communications stack and substituted a simple request-response protocol, which might account for some of performance degradation. They might simply be suggesting in public, or not bothering to contradict the suggestion, that the problem is hardware, because such a description might be too difficult to communicate through non-technical news media outlets.
I now see that the directory tree is for a real time control application put out by Argonne National Labs and called "EPICS", for "Experimental Physics and Industrial Control System." The web site is here.
Collecting information from the web, it appears that the Rovers are using "RAD6000" CPUs. These are are made by BAE Eystems and are based on the IBM RS-6000 RISC-like 32 bit processor architecture. The processors are radiation-hardened. Check this out.
From here the system seems to have a single 20 MHz CPU, 128 MByte DRAM with error recognition and correction and a 3 MByte EEPROM.
VXWorks OS images appear to be traditionally cross-compiled from Unix (Solaris, etc.) systems, so the directory structure of the build tree does not necessarily represent the directory structure of the live filesystem.
The RS-6000 architecture does not appear to be a standard Wind River supported architecture, so VXWorks might have had to be ported to it by BAE Systems in their BSP (Board Support Package)-- another potential source of problems if it was not done carefully...
If so, its prognosis seems questionable.
Oops.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.