Free Republic
Browse · Search
News/Activism
Topics · Post Article

Skip to comments.

Spirit remains in 'critical' condition (Newest update)3:55 EST
Spaceflightnow.com ^ | 1/23/2004 | William Harwood

Posted on 01/23/2004 1:39:26 PM PST by Keith

The crippled Spirit rover remains in critical condition on the surface of Mars, engineers said today, the victim of ongoing electronic seizures that have caused its central computer to reboot itself more than 60 times over the past two days.

Engineers successfully coaxed the rover to beam back limited engineering data during two brief communications sessions and they were relieved to discover the spacecraft's power system was providing the necessary life support. But Spirit's state of mind was clearly - and unusually - different in both sessions, ruling out any simple explanations for what might have gone wrong.

"We have a serious problem," said project manager Pete Theisinger. "The fact that we've got a vehicle that we believe is stable for an extensive period of time will give us time to work that problem. We can command it to talk to us and even though we get perhaps limited information, we do get good information and that helps us work through the problem.

"I expect that we will get functionality back out of this rover. I think the chances that it will be perfect again, I would think, are not good. The chances that it will not work at all, I think are also low. I think we're somewhere in that broad middle and we need to understand the problem to find out exactly where we are."

Spirit went on the blink Wednesday as it was carrying out a procedure to calibrate drive motors used by its thermal emission spectrometer. Prior to that moment, everything was operating normally. But some event, possibly a hardware failure of some sort, threw the rover's electronic brain for a loop. Since then, the spacecraft has been in a state of limbo, responding in unusual fashion to anxious flight controllers.

"This morning, we sent an early beep to the spacecraft and did not get a response," Theisinger said. "As we were preparing to send a second, the spacecraft talked to us. We got very fractional frames and then moved very quickly to ask it to speak to us for 30 minutes at 120 bits per second. We got 20 minutes of transmission in that occasion, which was a single frame of engineering data repeated.

"Then we repeated that full sequence of events and we got about 15 minutes of engineering data at 120 bits per second where the frames were updated for 15 minutes and then for the second 15 minutes we had nothing but fill data."

He said Spirit "has been in a processor reset loop of some type, mostly since Wednesday, we believe, where the processor wakes up, loads the flight software, uncovers a condition that would cause it to reset. But the processor doesn't do that immediately. It waits for a period of time - at the beginning of the day it waits for 15 minutes twice and then for the rest of the day it waits for an hour - and then it resets and comes back up."

Complicating the work to track down the problem, "the indications we have on two occasions is that the thing that causes the reset is not always perceived to be the same," Theisinger said. "We are confused by that, but that's the facts as we presume them to be right now."

The reset sequence, similar to repeatedly unplugging one's personal computer and forcing it to restart, began Wednesday morning on Mars when a calibration of the spectrometer motors ended prematurely. An anomaly team has been formed to study the telemetry and to decide what readings to request from Spirit to help narrow down the range of possible failures.

"I think we should expect that we will not be restoring functionality to Spirit for a significant period of time," Theisinger said, "I think many days, perhaps a couple of weeks, even in the best of circumstances, from what we see today."

In the meantime, he said, Spirit remains in "critical" condition.

"We do not know to what extent we can restore functionality to the system because we don't know what's broke," Theisinger said. "We don't know what started this chain of events and I think, personally, that it's a sequence of things, and we don't know, therefore, the consequences of that. I think its difficult at this very preliminary stage to assume we did not have some type of hardware event that caused this to start and therefore, we don't know to what extent we can work around that hardware event and to what extent we can get the software to ignore that hardware event if that's what we eventually have to do.

"We've got a long way to go here with the patient in intensive care. But we have been able to establish that we can command it, and we have been able to establish that it can give us information and we have been able to establish that the power system is good and we're thermally OK and those are all very, very important pieces of information.

"We are a long, long way from being done here, but we do have serious problems and our ability to eventually work around them is unknown. Do not expect a big sea change in either knowledge or theory in the next several days. This is a very complex problem."

Amid the troubleshooting, Spirit's twin - the Opportunity rover - remains on track to land early Sunday morning East Coast time on Meridiani Planum, a region on the other side of Mars where deposits of minerals that form in the presence of water have been detected. Theisinger said engineers do not believe Spirit's problem poses any generic risk to Opportunity, but he said the flight control team would be much more cautious in its daily operations to minimize the chances of a similar problem.

"It is likely, depending upon what happens in the next 48 to 72 hours, that we may not continue the Opportunity impact-to-egress with the same pace and dispatch that we did on Spirit," he said. "It depends on if we can get Opportunity to a defined, sustainable state on the ground and we can continue to make progress (with) Spirit. We will likely do that and try and continue to make progress on Spirit to get it back to some level of functionality. That's a decision the project will make in consultation with management as we take the temperature of this thing over the next couple of days."

So far, the only change for planned for Opportunity's descent is a decision to deploy its braking parachute at a slightly higher altitude than Spirit's to provide more of a safety margin.

In other developments, engineers today presented a dramatic animation of Spirit's landing based on actual telemetry from the spacecraft, showing how a sudden gust of wind forced small side-pointing rockets to fire at the last second to prevent the lander from slamming down at more than 50 mph.

The telemetry, collected earlier and subjected to complex analysis, also shows how the rover bounced across the floor of Gusev Crater before finally rolling to a stop.

Michael Malin, principal investigator of a high-resolution camera aboard NASA's Mars Global Surveyor spacecraft, unveiled a dramatic photograph showing Spirit, it's parachute and its heat shield resting on the surface of Mars. The remarkable photograph even shows several of Spirit's bounce marks in the martian soil.


TOPICS: News/Current Events
KEYWORDS: jpl; mars; nasa; rover; spirit
Navigation: use the links below to view more comments.
first 1-2021-26 next last
Sounds like this may be a long haul with a craft needing a constant "work around." Perhaps some of our Freeper techies can give us some insight as to how bad this sounds.
1 posted on 01/23/2004 1:39:29 PM PST by Keith
[ Post Reply | Private Reply | View Replies]

To: MrsEmmaPeel
What do you think?
2 posted on 01/23/2004 1:43:02 PM PST by Dog ("America will never seek a permission slip to defend the security of our Country")
[ Post Reply | Private Reply | To 1 | View Replies]

To: Keith
Isn't that spectrometer made by an ESA contractor?
3 posted on 01/23/2004 1:48:01 PM PST by CasearianDaoist
[ Post Reply | Private Reply | To 1 | View Replies]

To: Keith
Michael Malin, principal investigator of a high-resolution camera aboard NASA's Mars Global Surveyor spacecraft, unveiled a dramatic photograph showing Spirit, it's parachute and its heat shield resting on the surface of Mars. The remarkable photograph even shows several of Spirit's bounce marks in the martian soil.


4 posted on 01/23/2004 1:50:25 PM PST by So Cal Rocket
[ Post Reply | Private Reply | To 1 | View Replies]

To: Keith
Sounds like this may be a long haul with a craft needing a constant "work around." Perhaps some of our Freeper techies can give us some insight as to how bad this sounds.

It's bad, but not as bad as it sounded in the first place. Lots of fine techies are working the problem with the terran copy of Spirit to try to duplicate the glitch and then provide workarounds. (This is true of all long-range spacecraft sent by NASA. Even Galileo had a terran-bound duplicate for such purposes.)

The first step is replaying all commands sent prior to the malfunction. Then comes the process of ruling out causes one-by-one. Then comes ruling out causes-in-combination. It's a painstaking process to say the least.

"Safe mode" probably sounds like a really bad thing to most folks, but it's really not. Those who remember the Galileo mission will recall that the craft went into safe mode several times during the course of its work (and even had a total failure of its high gain antenna), yet the mission was still a success.

I still hold great hope for Spirit. And I'll still be holding my breath when Opportunity goes into the final stages for landing tomorrow night.

5 posted on 01/23/2004 1:50:29 PM PST by Prime Choice (Americans are a spiritual people. We're happy to help members of al Qaeda meet God.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Keith

6 posted on 01/23/2004 1:52:51 PM PST by MineralMan (godless atheist)
[ Post Reply | Private Reply | To 1 | View Replies]

To: So Cal Rocket
You know what's really remarkable about that photograph you posted? It's that it took us over 10 years to pinpoint where Viking had landed on Mars. Now we've got it down to a matter of days. Absolutely amazing.
7 posted on 01/23/2004 1:52:56 PM PST by Prime Choice (Americans are a spiritual people. We're happy to help members of al Qaeda meet God.)
[ Post Reply | Private Reply | To 4 | View Replies]

To: Keith
It depends on whether they can flash the software. They should have put a "safe mode" on the computer. Basically, you just run a terminal waiting for input. Who knows if they did that?
8 posted on 01/23/2004 1:53:56 PM PST by AppyPappy (If You're Not A Part Of The Solution, There's Good Money To Be Made In Prolonging The Problem.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Keith
Any computer that's rebooting itself 30 times a day is a very sick computer. Maybe it will heal itself somehow, but it sounds like hardware problems are contributing.
9 posted on 01/23/2004 1:55:45 PM PST by blowfish
[ Post Reply | Private Reply | To 1 | View Replies]

To: Keith
Sounds like they have to debug code they have already debugged a zillion times, and they have to do it soon because this device will go off-line in a couple of months anyway. If somebody does find the bug and they can reload, there ought to be some kind of reward. Maybe a new pocket protector w/ a set of 8 colors of ballpoint pens.
10 posted on 01/23/2004 1:56:46 PM PST by RightWhale (Repeal the Law of the Excluded Middle)
[ Post Reply | Private Reply | To 1 | View Replies]

To: blowfish
so, perhaps we are looking at a hard drive problem?
11 posted on 01/23/2004 1:58:22 PM PST by Keith (IT'S ALL ABOUT THE JUDGES)
[ Post Reply | Private Reply | To 9 | View Replies]

To: Keith
Will this be a setback on Halibertens planned oil pipeline to Mars???

Pray for W and The Truth

12 posted on 01/23/2004 1:59:08 PM PST by bray (The Wicked Witch of NY and Her (9-6) Flying Monkeys are In Flames!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Keith
a very simplistic version of what Spirit is doing is the same thing your home computer doea when Windows finally craps out. Spirit is basically doing the same thing. Rebooting, finding a fault and rebooting again.

What the NASA engineers are doing now is basic troubleshooting and debugging. Much of the time finding the problem is a lot longer than fixing it. Especially when the equipment is only telling you limited information about its condition.

Judging from what the engineers have said, this is not a software problem but a hardware failure of some kind. The "Workaround" is not as bad as they make it sound. They will isolate the malfunctioning piece of equipment and then send programming to Spirit telling it to either disconnect the broken equipment from the system or to ignore the fault signal and not communicate with that piece.

Once Spirit is reprogrammed it will be an automatic "workaround" and will not effect the speed or operation of other pieces of equipment.

where the "degradation" comes in is that the broken piece is gone forever. if it is the Mass-spectrometer -- no tests can be done with that piece of equipment, etc.

That is what NASA is worried about, depending on what failed, Spirit may come back 99% functional or 9% functional. They won't know until they know what broke, and what that effects.

No matter what though, Spirit will be operational again, the question is "How operational"

13 posted on 01/23/2004 2:01:47 PM PST by commish (Freedom Tastes Sweetest to Those Who Have Fought to Preserve It)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Keith

The component status module must be designed to operate independent of external damage/ingress as much as possible. This must be a sort of black box that has a very high fail-safe
design (relative to the main processor, drive units, motion system, chem lab apparatus etc).. thus it must have a polling routine (with multiple redundant comms) that should establish the working status of critical parts. If component status check routine processing and communications go out of wack... the mission is over because you cannot proceed if you are blind to feedback regarding Spirits damage conditions. If a rock were to be blown into the rover.....
14 posted on 01/23/2004 2:03:46 PM PST by birg
[ Post Reply | Private Reply | To 1 | View Replies]

To: Keith
>so, perhaps we are looking at a hard drive problem?
Who knows? Most operating systems are vulnerable to faulty hardware: memory, harddrives, IO hardware etc etc. When a computer is repeatedly crashing with this frequency I would suspect some bad hardware.
15 posted on 01/23/2004 2:04:12 PM PST by blowfish
[ Post Reply | Private Reply | To 11 | View Replies]

To: Keith
Its a software problem. Can't connect directly to the source. Probably have the wrong address. Happens to earth bound Freepers all the time.-Tom


16 posted on 01/23/2004 2:06:44 PM PST by Capt. Tom (Don't confuse the Bushies with the dumb republicans. - Capt. Tom)
[ Post Reply | Private Reply | To 1 | View Replies]

To: blowfish
It could be a software problem, too.

I worked on embedded systems for many years, though not for NASA. And one thing we always included was a watchdog timer.

Essentially, the watchdog timer is hardware that resets the computer every X seconds, or minutes, or hours (depending on the design), and if the software that's running on the computer doesn't occasionally reset the watchdog timer (because, perhaps it's caught in an infinite loop) then the watchdog timer will reset the computer. If everything is working perfectly, the software always resets the watchdog timer before the watchdog timer can reset the computer.

Considering the article specifically mentions times of 15 minutes and an hour, I suspect it's the watchdog timer that's resetting the computer.

Added to this all is that Spirit is running a multi-tasking, multi-threaded, realtime operating system. As such, the problem could be as simple as a thread having too high a priority (higher than the thread for resetting the watchdog timer), which is grabbing almost all the CPU cycles, thus slowing down communications and pre-empting the watchdog reset thread.
17 posted on 01/23/2004 2:28:24 PM PST by Monitor (Gun control isn't about guns; it's about control.)
[ Post Reply | Private Reply | To 15 | View Replies]

To: Prime Choice
I still hold great hope for Spirit. And I'll still be holding my breath when Opportunity goes into the final stages for landing tomorrow night.

I too hold out hope.

Given the politics of today, I grow tired of hearing and seeing power hunger politicians that care only about reelection and private agenda's. This is real, put together by some of America's best, brightest, best educated people that are on a historical exploration of another world.

On a side note, I can never quite understand American's that mock great American explorations by posting endless, humorless quips and juvenile cartoon pictures and images. I guess I just don't see the humor watching a major project, that is now having considerable difficulty, that so many worked so hard for, be the brunt of jokes. I guess I feel the focus of jokes and mocking should be directed at those in DC. Except for anger from the people, they deserve little else. End of rant.

18 posted on 01/23/2004 2:30:00 PM PST by Joe Hadenuf (I failed anger management class, they decided to give me a passing grade anyway)
[ Post Reply | Private Reply | To 5 | View Replies]

To: Keith
Those Martians are pi55ed off at our dumping debris on their planet. The Brit/Euro Beagle was also silenced.
19 posted on 01/23/2004 2:46:58 PM PST by expatpat
[ Post Reply | Private Reply | To 1 | View Replies]

To: Prime Choice
Are you sure about the duplicate stuff? The reason I ask is because a NOVA special followed these engineers around at JPL and the group leader made the point during an impact test of one of the components that this project was unique in that they only had live components, no backups. As I understand it, the backup is the duplicate project, Opportunity, due to land tomorrow morning.
20 posted on 01/23/2004 2:55:35 PM PST by beckett
[ Post Reply | Private Reply | To 5 | View Replies]


Navigation: use the links below to view more comments.
first 1-2021-26 next last

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
News/Activism
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson