Mars Rover Recovering From Memory Problems

New Scientist ^ | 1-28-2004 | David L Chandler

[blam]

Mars rover recovering from memory problems

13:35 28 January 04

NewScientist.com news service

A full revival of the Mars rover Spirit from its electronic ailments now seems highly likely. Engineers now think there is no real hardware or software problems, but something much easier to fix - a simple overload of files in its onboard memory.

If further testing confirms this diagnosis, that will be very good news for Spirit's twin, Opportunity. Any software bug or hardware weakness would probably be present in both rovers and might require weeks of analysis and repair.

But if, as it appears, the problem is a previously unrecognised limit on the number of files that can be stored in the craft's flash memory, then Opportunity's data collection and file management can be planned to prevent the problem.

This would avoid the bleak situation faced by engineers when Spirit fell silent for more than a day and failed to respond to commands. Having initially described the Spirit's troubles as "critical", mission manager Jennifer Trosper says "the patient is now in rehab".

However, Opportunity has developed a problem of its own, according to another mission manager Jim Erickson. The rover is losing power, apparently due to a heating unit that is switching itself on when it should not. What this will mean for the rover's mission and whether it can be fixed are not yet known.

Coaxed communication

Spirit's controllers have been coaxing the rover back into communication since it ended its silence on Thursday with a single bleep. The engineering data returned has allowed them to piece what had happened.

The rover first failed halfway through a test of a moving mirror that directs light to the mini-TES instrument. The high-gain antenna was also being used at the time, and the spacecraft entered a "safe mode" associated with antenna problems.

Later data returns showed the craft had entered a repeating cycle of resetting its computer system, preventing it from carrying out anything but the simplest commands. At last count, it had rebooted itself more than 120 times. This constant resetting prevented it from entering its night sleep cycle, needed to conserve its batteries.

But detailed analysis of the start of each reset cycle eventually led to the apparent answer to the mystery. The problem was clearly associated with the handling of files being written to one of its three types of internal memory: a non-volatile 256 megabyte flash memory.

Testing on Monday and Tuesday suggests that it is not the flash memory itself that is at fault, but the software's file-handling system. Unbeknownst to the engineers, there seems to be a limit on the number of files that can be simultaneously stored in the flash memory, even though the overall memory capacity is not full.

The solution is likely to be simply deleting unneeded files, many of which were accumulated during the eight-month journey to Mars. It will require some skillful programming to get the computer to do this without falling back into its resetting cycle, but Trosper says a full recovery is now expected.

David L Chandler

[DMCA]

1 Astronomical Unit = 8.31675359 light minutes

Mars is, at its furthest point, 2.5 AU from earth leading to a one way trip time of 20.791884 light minutes.

[thesummerwind]

Mars Rover Recovering From Memory Problems

Damn, I can't even remember last night.

Did anyone see Dennis Miller's second show? If so, give me some stuff!

[ambrose]

When all else fails, CTRL ALT DELETE...

[r9etb]

No, the explaination is not reasonable, but troubling. The rovers were designed to last 90 days, (and travel 8 months). The longest test was 9 days! NASA is still infested by "faster, cheaper". I can't even imagine who would be dumb enough to let these guys send them to Mars!

The underlying problem here is that they didn't do a realistic loading test. They obviously demonstrated some basic capabilities, but didn't do enough uploads to create the problems seen.

This is a systems engineering/test planning problem -- probably nobody even thought of doing such a test.

In hindsight we can point fingers, but of all the problems that could have happened (which would have driven the Failure Modes and Effects Analysis that in turn drove the test plan), I can really understand them not having thought of it.

I think this will probably join the annals of "Learned the Hard Way" lessons, and folks will do that test from now on. I know I'm gonna use this as an example for the students in my class.

[r9etb]

Glad to have somebody to agree with.... ;-)

[John H K]

There actually never was any "NASA Glory Era" where everything worked perfectly.

We had launch vehicles blowing up left and right in the early 60s. The Apollo I ground test killed 3 people. We almost lost Apollo 13.

And we've consistently done better than everyone else. The Soviets had an obscene number of Venus and Mars probes fail. The Japanese just had theirs fail. Beagle 2 failed.

Not that better performance shouldn't be demanded and expected but there's a limit to how perfectly reliable all of this is going to be.

[BigWaveBetty]

Stuff on Freepmail.

[BigWaveBetty]

Nice picture, but it gives rise to a few questions: For example; Why did they send a wood burning stove and where are they going to find the logs?

Ha! It does look like a wood burning stove. Those dang scientists prolly stuck some wheels on one, shot it into space and then charged us a billion for it! ;-)

[6ppc]

why couldn't the number of files storable on a given memory be a SW configurable parameter upon reset?

It has to do with the data type used to store the data, for example a byte (8 bits) is limited to storing numbers between -127 and +128, or 0 to 255 with no sign.

[brownsfan]

There actually never was any "NASA Glory Era" where everything worked perfectly.

I agree. Our memories trick us. It *seems* they were better then they were. Although, they were amazing given what they had to work with. And, that is where the expectation/criticism comes into play. NASA has better tools, and better background. So, personally, I just expect more.

[Lessismore]

Flash File System Selection Guide from Intel.

http://www.intel.com/design/flcomp/support/applnots/29766505.pdf

Wind River seems to use TrueFFS from M-Systems. Elsewhere, this has a configurable maximum number of open files. It can provide FAT if the operating system doesn't provide the File System layer above TrueFFS.

[Fitzcarraldo]

Recovering Spirit Sends a New Picture

NASA's Mars Exploration Rover Spirit took and returned this image on January 28, 2004, the first picture from Spirit since problems with communications began a week earlier. The image from the rover's front hazard identification camera shows the robotic arm extended to the rock called Adirondack. As it had been instructed a week earlier, the Moessbauer spectrometer, an instrument for identifying the minerals in rocks and soils, is still placed against the rock. Engineers are working to restore Spirit to working order so that the rover can resume the scientific exploration of its landing area.

[GretchenEE]

marker bump

[GretchenEE]

What is bleeding-edge technology?

Where's your .gif / .jpg / .bmp? I hardly recognized you!

[Nick Danger]

What is bleeding-edge technology?

That's the stuff where, when it works as planned, you say, "Hey! We got lucky!"

OK, here's the plan. We're gonna make this machine, and we're gonna shoot it to Mars, and just before it hits the ground going a hundred miles an hour, these air bags are gonna inflate in the nick of time just like in a car crash, and our little car is going to crash into Mars. And the air bag is going to save it. And then we'll deflate the air bag, the thing will fall a couple feet to the ground, and then it's gonna roll around and take pictures and send them back to us.

When that actually works, all you can think is, "Just damn!"

[GretchenEE]

Thanks for the explanation. And the poetic way to describe it.

Li'l ol' Spirit is a marvel.

It's reassuring to see your face with your post. ;-)